This documentation supports the 18.08 version of Remedy with Smart IT.

To view the latest version, select the version from the Product version menu.

Migrating social data from MongoDB to the AR System database

Starting with version 18.05, Smart IT no longer uses MongoDB for storing your social data. If you are upgrading from any version prior to 18.05, before you upgrade to Smart IT latest version, use the mongo migration utility to migrate your social data stored in MongoDB to the AR System database. This migration ensures that your social data is visible in the Smart IT dashboard seamlessly after you upgrade to Smart IT latest version. 

The mongo migration utility is available for download through BMC Communities.

This utility migrates the social data that is not already present in the AR System database. In particular, this utility migrates the following social data:

  • User follow data (items in Smart IT that you have opted to follow)
  • Social events data (only system events data such as SLA alerts, outages, and so on)
  • User profile thumbnail images 
  • Asset profile thumbnail images

The following table provides details of social data that is migrated using this utility and what is not migrated:

Migrated to AR System databaseNot required to migrate to AR System database
  1. Follow configuration of tickets
  2. SLA Alerts
  3. Outages
  4. Related Task Status Change events
  5. Related Change Events
  6. Unrelated Change Events
  7. Related Create Events
  8. Region change (asset)
  9. Create Ticket Feeds
  10. Assignment system event (for Knowledge)
  11. Approval/Hold/Reject events (for Knowledge)
  1. Worklogs
  2. Worklog attachments
  3. Worklog attachment thumbnails
  4. Status change system event (except Knowledge,request)
  5. Priority system event
  6. Assignment system event (except Knowledge)
  7. Broadcast
  8. Broadcast Attachment
  9. Approval/Hold/Reject events (except Knowledge)
  10. Ownership/user/primary owner events

List of ITSM forms

The mongo migration utility migrates social data from MongoDB to the following ITSM forms:

Form nameContains
SMT:Social_FollowConfigUser follow data.
SMT:Social_EventsSocial events data except status-change for knowledge and request.
RKM:UpdateRequestsStatus-change events for knowledge.
SRM:WorkInfoStatus-change events for service.
CTM:PeopleUser profile thumbnail image for all users.
AST:AttributesAsset profile thumbnail image for all assets.


Workflow

The following diagram illustrates the process of performing the MongoDB data migration:

To add parameter values to the migration script

Before you run this utility, you must edit the relevant script file and add values to the parameters required to perform the migration. 

  1. Open the Mongo migration script file for editing.
      • (WindowsSmart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\mongo-migration\mongo-migration-script.bat
      • (UNIX<Smart_IT_Installation_folder>/mongo-migration-script.sh

  2. Specify values for the following parameters:
ParameterDescription
ITSM settings
ITSM_HOST_NAMESpecify the name of the primary server in your server group environment.
ITSM_PORTSpecify the port number where the primary server is running.
ITSM_ADMIN_USERSpecify the user name to log on to the server.
ITSM_USER_PASSWORDSpecify the password to connect to the server.
Mongo DB settings

MONGODB_HOST_NAME

Specify the name of the MongoDB server host.
MONGODB_PORT

Specify the port number of the MongoDB server.

Default port number is 0.

MONGODB_DB_NAME

Specify the MongoDB database name that you specified during the Smart IT server installation.

The default database name is Social.  

MONGODB_USERNAME(Optional) Specify the MongoDB user name that you have configured.
MONGODB_PASSWORD(Optional) Specify the MongoDB password that you have configured.
Other settings
ITEMS_TO_MIGRATE

Specify the names of the objects, separated by a comma. The following object types are supported:

  • all - Represents all objects
  • follow - Represents only user follow events
  • event - Represents only system events
  • user - Represents only user thumbnail images
  • asset - Represents only asset thumbnail images
  • knowledge- Represents knowledge events, like status change, assignment change, approvals (Aaccept, Hold, and Reject), minor edit, and create feed
  • createfeed- Represents all the create feeds of all the ticket types except knowledge
  • emailfeed- Represents the email recepients, feed to all the ticket types

If you do not specify a value, all objects are migrated. For optimum performance, specify the type of object that you want to migrate.

WORKLOG_TYPE

Specify the priority of the worklog items to migrate. The following options are supported:

  • all - Represents all worklog items
  • priority - Represents worklog items having High priority, for example, SLA Alerts and Outages.
  • normal - Represents worklog items having normal priority except SLA Alerts and Outages.

If you have a large amount of data to migrate, for optimum performance, you must migrate the high priority items first. For example, SLA Alerts and Outages.

TENANT_IDEnter the unique ID of the tenant on which you want to perform the migration.

FROM_DATE

TO_DATE

Enter the From date in the yyyy-mm-dd format.

Enter the To date in the yyyy-mm-dd format.

  • If you do not specify a value in FROM_DATE and TO_DATE fields, all data is migrated.
  • If you specify only a FROM_DATE value, TO_DATE is considered as the current date and only data relevant from the FROM_DATE is migrated.
  • If you specify both FROM_DATE and TO_DATE values, all relevant data within the given dates is migrated.

BMC recommends that you migrate the most recent data first (for example, previous month) and, if needed, next migrate data of the previous three months and so on.

RERUN_IDS

Specify the request ID to run a failed migration job. You need to copy the request ID of the failed migration job from the SMT:MIG:Social_Mongo_Audit form.

You can specify multiple request IDs separated by a comma.

To run the migration utility

You can run this utility from a command prompt using a batch file (mongo-migration-script.bat) on the Microsoft Windows platform or a script (mongo-migration-script.sh) on the UNIX platform. This utility is available as part of the Smart IT installer and is available as a MongoMigration.zip file located in the Smart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\data-migration folder. The utility is also available for download on BMC Communities.

  1. Extract the MongoMigration.zip file to a folder, for example, C:\Utility folder.
  2. (Optional) Open the log4j.properties file that is present in C:\Utility\mongo-migration\src\main\resources folder.

    • To run the migration utility in debug mode, set the following parameter to debug:
      log4j.rootLogger=DEBUG, stdout, file
  3. Go to a command prompt.
  4. Enter either of the following commands:
    • (Windows) mongo-migration-script.bat
    • (UNIX) mongo-migration-script.sh

After the utility completes the run, an appropriate message is displayed at the command prompt.

Note

To prevent duplicate data from the migration, you cannot run this utility more than once for a date range. An appropriate message is logged in the migration.log file, which is located in the extracted folder.

To verify the migration

  1. Open the SMT:MIG:Social_Mongo_Audit form.
  2. Review the values in the following fields and take any required action:
    • Added into AR: Indicates the number of records added in the AR System forms.
    • Read from Mongo: Indicates the number of records successfully read from MongoDB.

      Note

      If you have migrated only the Event configuration, the values in the Added into AR field and this field should be the same. However, if you are migrating the Follow object configuration, the number in the Added into AR field can be greater than the number in the Read from Mongo field. This difference is because the Follow object configuration can have a one-to-many relationship. For example, one ticket is followed by many users.

    • Time taken: Indicates the amont of time (in seconds) taken for the migration. If this field is empty, it indicates that the migration is in progress. After the migration is complete, the value in this field indicates the total time taken for the migration.
    • Failed Object IDs: Indicates the list of objects whose data was not migrated. When you specify a rerun for this utility using the RERUN_ID parameter, information from the Failed Object IDs field is used to migrate the data again.
    • Status
      • Success: All records got migrated successfully, 
      • Success With Warning: Migration ran successfully with some of the records failed to migrate as these records had invalid data. Document ids for failed records has been captured into failed Object ids field,
      • Failed: Mongo server connectivity failed, 
      • In Progress: Migration in progress.



  3. Compare the number of records that are migrated. Perform the following:
    Before performing the migration, run the following query on MongoDB to get the count of system events that are to be migrated:

    > use social; //dbname of mongo
    >     db.subactivities.count({$or: [
        { $and: [ 
                  {"parent.feedType":{$ne: "user"}},
                  {"subFeedType" : {$in :["system","outage"]}},
                  { "data.text.event.eventType": {$in :[ "sla-change","outage-change","related-change","unrelated-change",
                                                "ka-minor-edit", "region-change","task-relation","incident-create",
                                                "workorder-create","change-create","problem-create","knownerror-create",
                                                "knowledge-create","activity-create","release-create","request-create","task-create","asset-relation" ]}}]},
        { $and: [{"parent.feedType":"knowledge"},{"subFeedType" : "system"},
                 {'data.text.event.eventType': {$in :[ "approval-accept-event","approval-reject-event","approval-hold-event","status-change"]}}]},
        { $and: [{"parent.feedType":"request"},{"subFeedType" : "system"},
                 {'data.text.event.eventType': "status-change"}]},          
        { $and: [{"parent.feedType":{$ne: "user"}},{"subFeedType" : "system"},
                 {"data.text.event.eventType":"status-change"}, {"data.text.event.classId" : "Manual"},{"data.text.event.messageId" : "19928"}]}
             ]});  

    After the migration, run the following SQL query to fetch the count of system events that were migrated:

    SELECT * FROM arschema WHERE  name = 'SMT:Social_Events'  //  This fetches the dynamic table name, here for ex: its “T3839” 
       select count(*) from T3839;                                                          // and then do count from that table  	
       >> 10000202
       SELECT * FROM arschema WHERE  name =  'RKM:UpdateRequests'
       >> T2864
       Select count(*) from T2864 where C302301056 like 'Status Marked:%'
          >> 10000      
         SELECT * FROM arschema WHERE  name =  SRM:WorkInfo'
       >> T2864
       Select count(*) from T1994 where C10001950 = '40000'
         >> 10000

    If the migration is successful, the count of records before and after migration should be the same.

To obtain the number of records for other items before migration

Use the following commands to calculate the number of records for other items before migration (as the script in the previous section is only for events). These commands might take some to run depending on your Mongo DB size.

  • To obtain the ‘subscribedBy’ count per ‘activities’ document:

    db.activities.aggregate({$project: {count:{$sum:{$size:"$subscribedBy"}}}})
  • To obtain the total ‘subscribedBy’ items: 

    db.activities.aggregate({$group: { _id: null, totalSubscribedBy: { $sum: { $size: "$subscribedBy"}}}})
  • To identify the count of ‘resourceFollowing’ per user:

    db.users.aggregate({$project: {resourceFollowingCount: {$size:"$resourceFollowing"}}})
  • To get a total number of ‘resourceFollowing’ items for all users:

    db.users.aggregate({$group: { _id: null, totalResourceFollowingAllUsers: { $sum: { $size: "$resourceFollowing"}}}})

To rerun the migration utility

You don't need to delete any record if the migration utility fails for any reason and you can rerun the utility. If the Failed Object IDs field is not empty, check the migration.log file for any failure message and run the migration again in rerun mode and specify the RERUN_IDs. You need to check the SMT:MIG:Social_Mongo_Audit form to get the Audit ID, which is required to rerun the utility.

To clean the MongoDB migration data and running the utility from scratch

You may want to clean the already migrated data first, and then run the utility from scratch. To see the steps to clean the MongoDB migration data, see Clean the MongoDB migration data.

To troubleshoot migration issues

The mongo migration utility captures the IDs of failed records in the Failed Object IDs field of the SMT:MIG:Social_Mongo_Audit form.

  • After the migration, if the Failed Object IDs field is empty, it indicates that all records have been successfully migrated.
  • If your system or the utility crashes during migration, you can resume the migration by running the utility in rerun mode. Use the following command format to run the utility in rerun mode:
    mongo-migration.bat -r <rerun_id>

Where to go from here

Performing the Smart IT upgrade

Was this page helpful? Yes No Submitting... Thank you

Comments

  1. Marek Ceizel

    For dwp it is possible to start a script and get the number of records which should be migratet and time how log would it take (roughly). Here I get the number of records using the mongo script, but I have no idea how long can it take (for us it is 822 000 entries). Any tip ?

    Jan 17, 2019 07:35
    1. Dhanya Menon

      Hello Marek,

      Thank you for your comment.

      We are discussing with the SME. We will get back to you soon.

      Thanks,

      Dhanya Menon

      Jan 24, 2019 03:37
    1. Dhanya Menon

      Hello Marek,

      This utility is only run to migrate MongoDB data for Smart IT. 

      For DWP related information, please see the following link (DWP 18.05 documentation):

      Performing the BMC Digital Workplace upgrade#Tomigrateyoursocialdataandupdatethedatabaselocation

      Regards,

      Dhanya Menon

      Jan 28, 2019 12:43