Migrating social data from MongoDB to the AR System database

Starting with version 18.05, Smart IT no longer uses MongoDB for storing your social data. If you are upgrading from any version prior to 18.05, before you upgrade to Smart IT latest version, use the MongoDB migration utility to migrate your social data stored in MongoDB to the AR System database. This migration ensures that your social data is visible in the Smart IT dashboard seamlessly after you upgrade to Smart IT latest version.

The Mongo migration utility is available for download through BMC Communities.

This utility migrates the social data that is not already present in the AR System database. In particular, this utility migrates the following social data:

• User follow data (items in Smart IT that you have opted to follow)
• Social events data (only system events data such as SLA alerts, outages, and so on)
• User profile thumbnail images 
• Asset profile thumbnail images

The following table provides details of social data that is migrated using this utility and what is not migrated:

List of ITSM forms

The Mongo migration utility migrates social data from MongoDB to the following ITSM forms:

Workflow

The following diagram illustrates the process of performing the MongoDB data migration:

To add parameter values to the migration script

Before you run this utility, you must edit the relevant script file and add values to the parameters required to perform the migration. 

1. Open the Mongo migration script file for editing. 
   
   • • (Windows) Smart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\mongo-migration\mongo-migration-script.bat
     • (UNIX) <Smart_IT_Installation_folder>/mongo-migration-script.sh
       
       
2. Specify values for the following parameters:

To run the migration utility

You can run this utility from a command prompt using a batch file (mongo-migration-script.bat) on the Microsoft Windows platform or a script (mongo-migration-script.sh) on the UNIX platform. This utility is available as part of the Smart IT installer and is available as a MongoMigration.zip file located in the Smart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\data-migration folder. The utility is also available for download on BMC Communities.

1. Extract the MongoMigration.zip file to a folder, for example, C:\Utility folder.

2. (Optional) Open the log4j.properties file that is present in C:\Utility\mongo-migration\src\main\resources folder.

   • To run the migration utility in debug mode, set the following parameter to debug:
     log4j.rootLogger=DEBUG, stdout, file
3. Go to a command prompt.
4. Enter either of the following commands:

• • (Windows) mongo-migration-script.bat
  • (UNIX) mongo-migration-script.sh

After the utility completes the run, an appropriate message is displayed at the command prompt.

To obtain the number of records before migration

Use the queries in the following table to calculate the number of records for various collection types before migration.

Before you run the query, specify the date range in the following lines of the query:

{"createdOn":{$lt: new Date("<To Date>T23:59:59Z")}},
{"createdOn":{$gt: new Date("<From Date>T00:00:00Z")}},

To verify the migration

1. Open the SMT:MIG:Social_Mongo_Audit form.
2. Review the values in the following fields and take any required action:
   
   • Added into AR: Indicates the number of records added in the AR System forms.

   • Read from Mongo: Indicates the number of records successfully read from MongoDB.

     Note

     If you have migrated only the Event configuration, the values in the Added into AR field and this field should be the same. However, if you are migrating the Follow object configuration, the number in the Added into AR field can be greater than the number in the Read from Mongo field. This difference is because the Follow object configuration can have a one-to-many relationship. For example, one ticket is followed by many users.

   • Time taken: Indicates the amont of time (in seconds) taken for the migration. If this field is empty, it indicates that the migration is in progress. After the migration is complete, the value in this field indicates the total time taken for the migration.
   • Failed Object IDs: Indicates the list of objects whose data was not migrated. When you specify a rerun for this utility using the RERUN_ID parameter, information from the Failed Object IDs field is used to migrate the data again.
   • Status: 
     • Success: All records got migrated successfully. The successful message is displayed in the Console.
     • Success with Warning: Migration ran successfully with some of the records failed to migrate as these records had invalid data. Document ids for failed records has been captured into failed Object ids field,
     • Failed: Mongo server connectivity failed, 
     • In Process: Migration in progress.
       
       
       
       

3. Verify the count of collection types after migration with the number of records available on the audit form.

To rerun the migration utility

You don't need to delete any record if the migration utility fails for any reason and you can rerun the utility. If the Failed Object IDs field is not empty, check the migration.log file for any failure message and run the migration again in rerun mode and specify the RERUN_IDs. You need to check the SMT:MIG:Social_Mongo_Audit form to get the Audit ID, which is required to rerun the utility.

To clean the MongoDB migration data and running the utility from scratch

You may want to clean the already migrated data first, and then run the utility from scratch. To see the steps to clean the MongoDB migration data, see Clean the MongoDB migration data.

To run the migration utility for large data

If your MongoDB data size is greater than 10 GB and have a large number of records in the activity collection (For example, 2.5 million), it is recommended that you first clean up the data, and then run the utility in parallel for different date range. You have to check the activity collection records for all years. You have to run one instance of migration utility for a year, if a year has more than 600,000 of record. If the record count for multiple years is less than 600,000, then you can combine the data in single migration utility but the years must be in sequence. You can execute the following script to find data within a date range:

db.getCollection('activities').find({"createdOn":{$gte:new ISODate("2015-01-01T23:59:59Z"),$lte:new ISODate("2015-12-31T23:59:59Z")}}).count();

To clean up data

It is recommended that you take a back up of your data before you clean up. Remove Approval, broadcast, and comments feed type, which are not relevant for Smart IT. The Digital Workplace related data are not required if it has already been upgraded or migrated. In that case you can remove the Digital Workplace related data. By using the Robo 3T, run the following queries. For a large data set, the execution of these queries may take time:

//To get the count details of number of APPROVAL records are present in subactivities collection
 db.subactivities.find({"parent.feedType":"APPROVAL"}).count();
//To get the count details of number of broadcasts records are present in subactivities collection
db.subactivities.find({"parent.feedType":"broadcasts"}).count();
//To get the count details of number of comments records are present in subactivities collection
db.getCollection('subactivities').count({"subFeedType" : "comment", "data.textMeta" :  ""}) 

//To remove the APPROVAL records from subactivities collection
db.subactivities.remove({"parent.feedType":"APPROVAL"});
//To remove the broadcasts records from subactivities collection
db.subactivities.remove({"parent.feedType":"broadcasts"});
//To remove the comments records from subactivities collection
db.getCollection('subactivities').remove({"subFeedType" : "comment", "data.textMeta" :  ""}) 

//To get the count details of number of APPROVAL records are present in activities collection
db.activities.find({"feedObj.feedType":"APPROVAL"}).count();
//To get the count details of number of broadcasts records are present in activities collection
db.activities.find({"feedObj.feedType":"broadcasts"}).count();

//To remove the APPROVAL records from activities collection
db.activities.remove({"feedObj.feedType":"APPROVAL"});
//To remove the broadcasts records from activities collection
db.activities.remove({"feedObj.feedType":"broadcasts"});

To run the utility in parallel

You need to run multiple instances of utility manually from different command lines. You can run MongoDB migration utility in parallel for different date range, for example, yearly, monthly etc.

Follow these steps to run the migration utility in parallel:

1. Copy the mongo-migration.zip from the Smart IT installer, and extract it.
2. Create multiple folders of MongoDB migration equal to the number of parallel runs you want. It is recommended that you rename the folders by appending date rang for a better readability, for example, mongo-migration_01Jan16_31Dec16.
3. Change the date range that is provided as the values for the fields FROM_DATE and TO_DATE inside the mongo-migration.bat or mongo-migration.sh file inside all the folders. Other fields will remain same in all the folders.
4. Run the mongo-migration-script.bat or mongo-migration-script.sh from all the folders. 
   This will start the parallel run of the migration utility.
   You can check the status in the SMT:Social_Mongo_Audit form. 

To run the migration utility for a specific feed type

To run the migration utility for a specific feed type, change the value of the parameter ITEMS_TO_MIGRATE. For example, if you want to run the migration for follow, then set the field as given below:

ITEMS_TO_MIGRATE=follow

To get the detail of the year in which the first record was created

Run the following query to get the detail of the year in which the first record was created:

db.getCollection('activities').aggregate([ 
    { "$group": { 
        "_id": null,
        "min": { "$min": "$createdOn" } 
    }}
]);

To check the database size

You can use a MongoDB management tool to check the database size. For example, you can use Robo3T-1.2 to connect to MongoDB database. Right click on the database name and select Database Statistics option. The dataSize field displays the the data value in bytes.

To check activities collection size

Right click on the activities collection and select Statistics option as displayed in the following screenshot. The count field displays the activity collection records.

To troubleshoot migration issues

This utility generates a log file with name migration_dd-mm-yyyy-hh-mm-ss.log (For example, migration_01-12-2017-04-38-48.log) in the same folder. You can check this file if any error is displayed.

The mongo migration utility captures the IDs of failed records in the Failed Object IDs field of the SMT:MIG:Social_Mongo_Audit form.

• After the migration, if the Failed Object IDs field is empty, it indicates that all records have been successfully migrated.
• If your system or the utility crashes during migration, you can resume the migration by running the utility in rerun mode. Use the following command format to run the utility in rerun mode:
  mongo-migration.bat -r <rerun_id>


To get the Group ID for Company to migrate Company specific data

1. In the Remedy IT Service Management, click AR System Administration > AR System Administration Console > Application > Users/Groups/Roles > Groups.  The Group Information form is displayed. 
2. Click Advanced search. 
   
3. At the bottom right corner of the form, click Fields > Long Group Name.
   
4. At the bottom of the page, in the advanced search query, provide your company name in double quote against Long Group Name field. For example, if your company name is Calbro Services, then the following information should be displayed in the advanced search query:
    'Long Group Name'="Calbro Services" 
5. Click Search.
6. The Group ID field is populated, which you can use as Company ID for company specific MongoDB data migration.
   

Where to go from here

Performing the Smart IT upgrade