This documentation supports the 19.02 version of Remedy with Smart IT.

To view the latest version, select the version from the Product version menu.

Migrating social data from MongoDB to the AR System database

Starting with version 18.05, Smart IT no longer uses MongoDB for storing your social data. If you are upgrading from any version prior to 18.05, before you upgrade to Smart IT latest version, use the MongoDB migration utility to migrate your social data stored in MongoDB to the AR System database. This migration ensures that your social data is visible in the Smart IT dashboard seamlessly after you upgrade to Smart IT latest version. 

The mongo migration utility is available for download through BMC Communities.

This utility migrates the social data that is not already present in the AR System database. In particular, this utility migrates the following social data:

  • User follow data (items in Smart IT that you have opted to follow)
  • Social events data (only system events data such as SLA alerts, outages, and so on)
  • User profile thumbnail images 
  • Asset profile thumbnail images

The following table provides details of social data that is migrated using this utility and what is not migrated:

Migrated to AR System databaseNot required to migrate to AR System database
  1. Follow configuration of tickets
  2. SLA Alerts
  3. Outages
  4. Related Task Status Change events
  5. Related Change Events
  6. Unrelated Change Events
  7. Related Create Events
  8. Region change (asset)
  9. Create Ticket Feeds
  10. Assignment system event (for Knowledge)
  11. Approval/Hold/Reject events (for Knowledge)
  1. Worklogs
  2. Worklog attachments
  3. Worklog attachment thumbnails
  4. Status change system event (except Knowledge,request)
  5. Priority system event
  6. Assignment system event (except Knowledge)
  7. Broadcast
  8. Broadcast Attachment
  9. Approval/Hold/Reject events (except Knowledge)
  10. Ownership/user/primary owner events


List of ITSM forms

The mongo migration utility migrates social data from MongoDB to the following ITSM forms:

Form nameContains
SMT:Social_FollowConfigUser follow data.
SMT:Social_EventsSocial events data except status-change for knowledge and request.
RKM:UpdateRequestsStatus-change events for knowledge.
SRM:WorkInfoStatus-change events for service.
CTM:PeopleUser profile thumbnail image for all users.
AST:AttributesAsset profile thumbnail image for all assets.


Workflow

The following diagram illustrates the process of performing the MongoDB data migration:


To add parameter values to the migration script

Before you run this utility, you must edit the relevant script file and add values to the parameters required to perform the migration. 

  1. Open the Mongo migration script file for editing. 
      • (WindowsSmart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\mongo-migration\mongo-migration-script.bat
      • (UNIX<Smart_IT_Installation_folder>/mongo-migration-script.sh

  2. Specify values for the following parameters:
ParameterDescription
ITSM settings
ITSM_HOST_NAMESpecify the name of the primary server in your server group environment.
ITSM_PORTSpecify the port number where the primary server is running.
ITSM_ADMIN_USERSpecify the user name to log on to the server.
ITSM_USER_PASSWORDSpecify the password to connect to the server.
Mongo DB settings

MONGODB_HOST_NAME

Specify the name of the MongoDB server host.
MONGODB_PORT

Specify the port number of the MongoDB server.

Default port number is 0.

MONGODB_DB_NAME

Specify the MongoDB database name that you specified during the Smart IT server installation.

The default database name is Social.  

MONGODB_USERNAME(Optional) Specify the MongoDB user name that you have configured.
MONGODB_PASSWORD(Optional) Specify the MongoDB password that you have configured.
Other settings
ITEMS_TO_MIGRATE

Specify the names of the objects, separated by a comma. The following object types are supported:

  • all - Represents all objects
  • follow - Represents only user follow events
  • event - Represents only system events
  • user - Represents only user thumbnail images
  • asset - Represents only asset thumbnail images
  • knowledge- Represents knowledge events, like status change, assignment change, approvals (Aaccept, Hold, and Reject), minor edit, and create feed
  • createfeed- Represents all the create feeds of all the ticket types except knowledge
  • emailfeed- Represents the email recepients, feed to all the ticket types

If you do not specify a value, all objects are migrated. For optimum performance, specify the type of object that you want to migrate.

WORKLOG_TYPE

Specify the priority of the worklog items to migrate. The following options are supported:

  • all - Represents all worklog items
  • priority - Represents worklog items having High priority, for example, SLA Alerts and Outages.
  • normal - Represents worklog items having normal priority except SLA Alerts and Outages.

If you have a large amount of data to migrate, for optimum performance, you must migrate the high priority items first. For example, SLA Alerts and Outages.

TENANT_IDEnter the unique ID of the tenant on which you want to perform the migration.

FROM_DATE (Required)

TO_DATE

Enter the From date in the yyyy-mm-dd format.

Enter the To date in the yyyy-mm-dd format.

  • If you do not specify a value in FROM_DATE fields, an error message is displayed.
  • If you do not specify a value in TO_DATE fields, all data is migrated till the date of migration.
  • If you specify only a FROM_DATE value, TO_DATE is considered as the current date and only data relevant from the FROM_DATE is migrated.
  • If you specify both FROM_DATE and TO_DATE values, all relevant data within the given dates is migrated.

BMC recommends that you migrate the most recent data first (for example, previous month) and, if needed, next migrate data of the previous three months and so on.

RERUN_IDS

Specify the request ID to run a failed migration job. You need to copy the request ID of the failed migration job from the SHR:Mongo Migration Audit form.

You can specify multiple request IDs separated by a comma.

COMPANY_LISTYou can specify the Group IDs for company separated by a comma. This is used to migrate company specific data. To know the steps to get the Group ID for company, see To get the Group ID for Company to migrate Company specific data.


To run the migration utility

You can run this utility from a command prompt using a batch file (mongo-migration-script.baton the Microsoft Windows platform or a script (mongo-migration-script.sh) on the UNIX platform. This utility is available as part of the Smart IT installer and is available as a MongoMigration.zip file located in the Smart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\data-migration folder. The utility is also available for download on BMC Communities.

  1. Extract the MongoMigration.zip file to a folder, for example, C:\Utility folder.
  2. (Optional) Open the log4j.properties file that is present in C:\Utility\mongo-migration\src\main\resources folder.

    • To run the migration utility in debug mode, set the following parameter to debug:
      log4j.rootLogger=DEBUG, stdout, file
  3. Go to a command prompt.
  4. Enter either of the following commands:
    • (Windowsmongo-migration-script.bat
    • (UNIXmongo-migration-script.sh

After the utility completes the run, an appropriate message is displayed at the command prompt.

Note

To prevent duplicate data from the migration, you cannot run this utility more than once for a date range. An appropriate message is logged in the migration.log file, which is located in the extracted folder.



To verify the migration

  1. Open the SMT:MIG:Social_Mongo_Audit form.
  2. Review the values in the following fields and take any required action:
    • Added into AR: Indicates the number of records added in the AR System forms.
    • Read from Mongo: Indicates the number of records successfully read from MongoDB.

      Note

      If you have migrated only the Event configuration, the values in the Added into AR field and this field should be the same. However, if you are migrating the Follow object configuration, the number in the Added into AR field can be greater than the number in the Read from Mongo field. This difference is because the Follow object configuration can have a one-to-many relationship. For example, one ticket is followed by many users.

    • Time taken: Indicates the amont of time (in seconds) taken for the migration. If this field is empty, it indicates that the migration is in progress. After the migration is complete, the value in this field indicates the total time taken for the migration.
    • Failed Object IDs: Indicates the list of objects whose data was not migrated. When you specify a rerun for this utility using the RERUN_ID parameter, information from the Failed Object IDs field is used to migrate the data again.
    • Status
      • Success: All records got migrated successfully. The successful message is displayed in the Console.
      • Success with Warning: Migration ran successfully with some of the records failed to migrate as these records had invalid data. Document ids for failed records has been captured into failed Object ids field,
      • Failed: Mongo server connectivity failed, 
      • In Process: Migration in progress.



  3. Compare the number of records that are migrated. Perform the following:
    Before performing the migration, run the following query on MongoDB to get the count of system events that are to be migrated:

    > use social; //dbname of mongo
    >     db.subactivities.count({$or: [
        { $and: [ 
                  {"parent.feedType":{$ne: "user"}},
                  {"subFeedType" : {$in :["system","outage"]}},
                  { "data.text.event.eventType": {$in :[ "sla-change","outage-change","related-change","unrelated-change",
                                                "ka-minor-edit", "region-change","task-relation","incident-create",
                                                "workorder-create","change-create","problem-create","knownerror-create",
                                                "knowledge-create","activity-create","release-create","request-create","task-create","asset-relation" ]}}]},
        { $and: [{"parent.feedType":"knowledge"},{"subFeedType" : "system"},
                 {'data.text.event.eventType': {$in :[ "approval-accept-event","approval-reject-event","approval-hold-event","status-change"]}}]},
        { $and: [{"parent.feedType":"request"},{"subFeedType" : "system"},
                 {'data.text.event.eventType': "status-change"}]},          
        { $and: [{"parent.feedType":{$ne: "user"}},{"subFeedType" : "system"},
                 {"data.text.event.eventType":"status-change"}, {"data.text.event.classId" : "Manual"},{"data.text.event.messageId" : "19928"}]}
             ]});  

    After the migration, run the following SQL query to fetch the count of system events that were migrated:

    SELECT * FROM arschema WHERE  name = 'SMT:Social_Events'  //  This fetches the dynamic table name, here for ex: its “T3839” 
       select count(*) from T3839;                                                          // and then do count from that table  	
       >> 10000202
       SELECT * FROM arschema WHERE  name =  'RKM:UpdateRequests'
       >> T2864
       Select count(*) from T2864 where C302301056 like 'Status Marked:%'
          >> 10000      
         SELECT * FROM arschema WHERE  name =  SRM:WorkInfo'
       >> T2864
       Select count(*) from T1994 where C10001950 = '40000'
         >> 10000

    If the migration is successful, the count of records before and after migration should be the same.


To obtain the number of records for other items before migration

Use the following commands to calculate the number of records for other items before migration (as the script in the previous section is only for events). These commands might take some to run depending on your Mongo DB size.

  • To obtain the ‘subscribedBy’ count per ‘activities’ document:

    db.activities.aggregate({$project: {count:{$sum:{$size:"$subscribedBy"}}}})
  • To obtain the total ‘subscribedBy’ items: 

    db.activities.aggregate({$group: { _id: null, totalSubscribedBy: { $sum: { $size: "$subscribedBy"}}}})
  • To identify the count of ‘resourceFollowing’ per user:

    db.users.aggregate({$project: {resourceFollowingCount: {$size:"$resourceFollowing"}}})
  • To get a total number of ‘resourceFollowing’ items for all users:

    db.users.aggregate({$group: { _id: null, totalResourceFollowingAllUsers: { $sum: { $size: "$resourceFollowing"}}}})


To rerun the migration utility

You don't need to delete any record if the migration utility fails for any reason and you can rerun the utility. If the Failed Object IDs field is not empty, check the migration.log file for any failure message and run the migration again in rerun mode and specify the RERUN_IDs. You need to check the SMT:MIG:Social_Mongo_Audit form to get the Audit ID, which is required to rerun the utility.

To clean the MongoDB migration data and running the utility from scratch

You may want to clean the already migrated data first, and then run the utility from scratch. To see the steps to clean the MongoDB migration data, see Clean the MongoDB migration data.


To run the migration utility for large data

If your MongoDB data size is greater than 10 GB and have a large number of records in the activity collection (For example, 2.5 million), it is recommended that you first clean up the data, and then run the utility in parallel for different date range. You have to check the activity collection records for all years. You have to run one instance of migration utility for a year, if a year has more than 600,000 of record. If the record count for multiple years is less than 600,000, then you can combine the data in single migration utility but the years must be in sequence. You can execute the following script to find data within a date range:

db.getCollection('activities').find({"createdOn":{$gte:new ISODate("2015-01-01T23:59:59Z"),$lte:new ISODate("2015-12-31T23:59:59Z")}}).count();


To clean up data

It is recommended that you take a back up of your data before you clean up. Remove Approval, broadcast, and comments feed type, which are not relevant for Smart IT. The Digital Workplace related data are not required if it has already been upgraded or migrated. In that case you can remove the Digital Workplace related data. By using the Robo 3T, run the following queries. For a large data set, the execution of these queries may take time:

//To get the count details of number of APPROVAL records are present in subactivities collection
 db.subactivities.find({"parent.feedType":"APPROVAL"}).count();
//To get the count details of number of broadcasts records are present in subactivities collection
db.subactivities.find({"parent.feedType":"broadcasts"}).count();
//To get the count details of number of comments records are present in subactivities collection
db.getCollection('subactivities').count({"subFeedType" : "comment", "data.textMeta" :  ""}) 

//To remove the APPROVAL records from subactivities collection
db.subactivities.remove({"parent.feedType":"APPROVAL"});
//To remove the broadcasts records from subactivities collection
db.subactivities.remove({"parent.feedType":"broadcasts"});
//To remove the comments records from subactivities collection
db.getCollection('subactivities').remove({"subFeedType" : "comment", "data.textMeta" :  ""}) 

//To get the count details of number of APPROVAL records are present in activities collection
db.activities.find({"feedObj.feedType":"APPROVAL"}).count();
//To get the count details of number of broadcasts records are present in activities collection
db.activities.find({"feedObj.feedType":"broadcasts"}).count();

//To remove the APPROVAL records from activities collection
db.activities.remove({"feedObj.feedType":"APPROVAL"});
//To remove the broadcasts records from activities collection
db.activities.remove({"feedObj.feedType":"broadcasts"});

To run the utility in parallel

You need to run multiple instances of utility manually from different command lines. You can run MongoDB migration utility in parallel for different date range, for example, yearly, monthly etc.

Follow these steps to run the migration utility in parallel:

  1. Copy the mongo-migration.zip from the Smart IT installer, and extract it.
  2. Create multiple folders of MongoDB migration equal to the number of parallel runs you want. It is recommended that you rename the folders by appending date rang for a better readability, for example, mongo-migration_01Jan16_31Dec16.
  3. Change the date range that is provided as the values for the fields FROM_DATE and TO_DATE inside the mongo-migration.bat or mongo-migration.sh file inside all the folders. Other fields will remain same in all the folders.
  4. Run the mongo-migration-script.bat or mongo-migration-script.sh from all the folders. 
    This will start the parallel run of the migration utility.
    You can check the status in the SMT:Social_Mongo_Audit form. 

Required heap size

Each migration utility run requires a minimum 512 MB, and maximum 1024 MB of Heap size. You can configure heap size inside the mongo-migration.bat/sh file. Change the value of JAVA_OPTIONS=-Xms512m -Xmx1024m.


To run the migration utility for a specific feed type

To run the migration utility for a specific feed type, change the value of the parameter ITEMS_TO_MIGRATE. For example, if you want to run the migration for follow, then set the field as given below:

ITEMS_TO_MIGRATE=follow

To get the detail of the year in which the first record was created

Run the following query to get the detail of the year in which the first record was created:

db.getCollection('activities').aggregate([ 
    { "$group": { 
        "_id": null,
        "min": { "$min": "$createdOn" } 
    }}
]);


To check the database size

You can use a MongoDB management tool to check the database size. For example, you can use Robo3T-1.2 to connect to MongoDB database. Right click on the database name and select Database Statistics option. The dataSize field displays the the data value in bytes.

Download Robo3T-1.2

Click the following link to download Robo3T-1.2 : https://robomongo.org/download

To check activities collection size

Right click on the activities collection and select Statistics option as displayed in the following screenshot. The count field displays the activity collection records.



To troubleshoot migration issues

This utility generates a log file with name migration_dd-mm-yyyy-hh-mm-ss.log (For example, migration_01-12-2017-04-38-48.log) in the same folder. You can check this file if any error is displayed.

The mongo migration utility captures the IDs of failed records in the Failed Object IDs field of the SMT:MIG:Social_Mongo_Audit form.

  • After the migration, if the Failed Object IDs field is empty, it indicates that all records have been successfully migrated.
  • If your system or the utility crashes during migration, you can resume the migration by running the utility in rerun mode. Use the following command format to run the utility in rerun mode:
    mongo-migration.bat -r <rerun_id>


To get the Group ID for Company to migrate Company specific data

  1. In the Remedy IT Service Management, click AR System Administration > AR System Administration Console > Application > Users/Groups/Roles > Groups.  The Group Information form is displayed. 
  2. Click Advanced search
  3. At the bottom right corner of the form, click Fields > Long Group Name.
  4. At the bottom of the page, in the advanced search query, provide your company name in double quote against Long Group Name field. For example, if your company name is Calbro Services, then the following information should be displayed in the advanced search query:
     'Long Group Name'="Calbro Services" 
  5. Click Search.
  6. The Group ID field is populated, which you can use as Company ID for company specific MongoDB data migration.


Where to go from here

Performing the Smart IT upgrade


Was this page helpful? Yes No Submitting... Thank you

Comments