Migrating social data from MongoDB to the AR System database

Starting with version 18.05, Smart IT no longer uses MongoDB for storing your social data. Before you upgrade to Smart IT version 18.05, use the mongo migration utility to migrate your social data stored in MongoDB to the AR System database. This migration ensures that your social data is visible in the Smart IT dashboard seamlessly after you upgrade to Smart IT 18.05.

The mongo migration utility is available as part of the Smart IT installer and is also available for download through BMC Communities.

This utility migrates the social data that is not already present in the AR System database. In particular, this utility migrates the following social data:

User follow data (items in Smart IT that you have opted to follow)
Social events data (only system events data such as SLA alerts, outages, and so on)
User profile thumbnail images
Asset profile thumbnail images

The following table provides details of social data that is migrated using this utility and what is not migrated:

Migrated to AR System database	Not required to migrate to AR System database
Follow configuration of tickets SLA Alerts Outages Related Task Status Change events Related Change Events Unrelated Change Events Related Create Events Region change (asset) Create Ticket Feeds Assignment system event (for Knowledge) Approval/Hold/Reject events (for Knowledge)	Worklogs Worklog attachments Worklog attachment thumbnails Status change system event (except Knowledge,request) Priority system event Assignment system event (except Knowledge) Broadcast Broadcast Attachment Approval/Hold/Reject events (except Knowledge) Ownership/user/primary owner events

List of ITSM forms

The mongo migration utility migrates social data from MongoDB to the following ITSM forms:

Form name	Contains
SMT:Social_FollowConfig	User follow data.
SMT:Social_Events	Social events data except status-change for knowledge and request.
RKM:UpdateRequests	Status-change events for knowledge.
SRM:WorkInfo	Status-change events for service.
CTM:People	User profile thumbnail image for all users.
AST:Attributes	Asset profile thumbnail image for all assets.

Workflow

The following diagram illustrates the process of performing the MongoDB data migration:

To add parameter values to the migration script

Before you run this utility, you must edit the relevant script file and add values to the parameters required to perform the migration.

Open the Mongo migration script file for editing.
- - (Windows) Smart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\data-migration\mongo-migration-script.bat
  - (UNIX) <Smart_IT_Installation_folder>/mongo-migration-script.sh
Specify values for the following parameters:

Parameter	Description
ITSM settings
ITSM_HOST_NAME	Specify the name of the primary server in your server group environment.
ITSM_PORT	Specify the port number where the primary server is running.
ITSM_ADMIN_USER	Specify the user name to log on to the server.
ITSM_USER_PASSWORD	Specify the password to connect to the server.
Mongo DB settings
MONGODB_HOST_NAME	Specify the name of the MongoDB server host.
MONGODB_PORT	Specify the port number of the MongoDB server. Default port number is 0.
MONGODB_DB_NAME	Specify the MongoDB database name that you specified during the Smart IT server installation. The default database name is Social.
MONGODB_USERNAME	(Optional) Specify the MongoDB user name that you have configured.
MONGODB_PASSWORD	(Optional) Specify the MongoDB password that you have configured.
Other settings
ITEMS_TO_MIGRATE	Specify the names of the objects, separated by a comma. The following object types are supported: all - Represents all below objects follow - Represents only user follow events for all types of tickets event - Represents only system events (SLA change and outage change, asset relation, related and unrelated change, task relation, and region change user - Represents only user thumbnail images asset - Represents only asset thumbnail images knowledge- Represents knowledge events, like status change, assignment change, approvals (Aaccept, Hold, and Reject), minor edit, and create feed createfeed- Represents all the create feeds of all the ticket types except knowledge emailfeed- Represents the email recepients, feed to all the ticket types If you do not specify a value, all objects are migrated. For optimum performance, specify the type of object that you want to migrate.
WORKLOG_TYPE	Specify the priority of the worklog items to migrate. The following options are supported: all - Represents all worklog items priority - Represents worklog items having High priority, for example, SLA Alerts and Outages. normal - Represents worklog items having normal priority except SLA Alerts and Outages. If you have a large amount of data to migrate, for optimum performance, you must migrate the high priority items first. For example, SLA Alerts and Outages.
TENANT_ID	Enter the unique ID of the tenant on which you want to perform the migration.
FROM_DATE TO_DATE	Enter the From date in the yyyy-mm-dd format. Enter the To date in the yyyy-mm-dd format. If you do not specify a value in FROM_DATE and TO_DATE fields, all data is migrated. If you specify only a FROM_DATE value, TO_DATE is considered as the current date and only data relevant from the FROM_DATE is migrated. If you specify both FROM_DATE and TO_DATE values, all relevant data within the given dates is migrated. BMC recommends that you migrate the most recent data first (for example, previous month) and, if needed, next migrate data of the previous three months and so on.
RERUN_IDS	Specify the request ID to run a failed migration job. You need to copy the request ID of the failed migration job from the SHR:Mongo Migration Audit form. You can specify multiple request IDs separated by a comma.
CHUNK_SIZE	For BMC internal purposes only.
NO_THREADS	For BMC internal purposes only.
FAILED_RUN_ID	Failed Run Id. This is the request id from the Audit Form SHR:Mongo Migration Audit. It is used to re run failed batch.

To run the migration utility

You can run this utility from a command prompt using a batch file (mongo-migration-script.bat) on the Microsoft Windows platform or a script (mongo-migration-script.sh) on the UNIX platform. This utility is available as part of the Smart IT installer and is available as a MongoMigration.zip file located in the Smart_IT_18.05.00_windows\windows\Smart_IT\Disk1\files\04204cabb68h\data-migration folder. The utility is also available for download on BMC Communities.

Extract the MongoMigration.zip file to a folder, for example, C:\Utility folder.
Go to a command prompt.
Enter either of the following commands:

- (Windows) mongo-migration-script.bat
- (UNIX) mongo-migration-script.sh

After the utility completes the run, an appropriate message is displayed at the command prompt.

Note

- To prevent duplicate data from the migration, you cannot run this utility more than once for a date range. An appropriate message is logged in the migration.log file, which is located in the extracted folder.

To verify the migration

Open the SMT:MIG:Social_Mongo_Audit form.
Review the values in the following fields and take any required action:
- Added into AR - Indicates the number of records added in the AR System forms.
- Read from Mongo - Indicates the number of records successfully read from MongoDB.
  Note
  
  If you have migrated only the Event configuration, the values in the Added into AR field and this field should be the same. However, if you are migrating the Follow object configuration, the number in the Added into AR field can be greater than the number in the Read from Mongo field. This difference is because the Follow object configuration can have a one-to-many relationship. For example, one ticket is followed by many users.
- Time taken - Indicates the amont of time (in seconds) taken for the migration. If this field is empty, it indicates that the migration is in progress. After the migration is complete, the value in this field indicates the total time taken for the migration.
- Failed Object IDs - Indicates the list of objects whose data was not migrated. When you specify a rerun for this utility using the RERUN_ID parameter, information from the Failed Object IDs field is used to migrate the data again.

Compare the number of records that are migrated. Perform the following:
Before performing the migration, run the following query on MongoDB to get the count of system events that are to be migrated:

> use social; //dbname of mongo
>     db.subactivities.count({$or: [
    { $and: [ 
              {"parent.feedType":{$ne: "user"}},
              {"subFeedType" : {$in :["system","outage"]}},
              { "data.text.event.eventType": {$in :[ "sla-change","outage-change","related-change","unrelated-change",
                                            "ka-minor-edit", "region-change","task-relation","incident-create",
                                            "workorder-create","change-create","problem-create","knownerror-create",
                                            "knowledge-create","activity-create","release-create","request-create","task-create","asset-relation" ]}}]},
    { $and: [{"parent.feedType":"knowledge"},{"subFeedType" : "system"},
             {'data.text.event.eventType': {$in :[ "approval-accept-event","approval-reject-event","approval-hold-event","status-change"]}}]},
    { $and: [{"parent.feedType":"request"},{"subFeedType" : "system"},
             {'data.text.event.eventType': "status-change"}]},          
    { $and: [{"parent.feedType":{$ne: "user"}},{"subFeedType" : "system"},
             {"data.text.event.eventType":"status-change"}, {"data.text.event.classId" : "Manual"},{"data.text.event.messageId" : "19928"}]}
         ]});

After the migration, run the following SQL query to fetch the count of system events that were migrated:

SELECT * FROM arschema WHERE  name = 'SMT:Social_Events'  //  This fetches the dynamic table name, here for ex: its “T3839” 
   select count(*) from T3839;                                                          // and then do count from that table  	
   >> 10000202
   SELECT * FROM arschema WHERE  name =  'RKM:UpdateRequests'
   >> T2864
   Select count(*) from T2864 where C302301056 like 'Status Marked:%'
      >> 10000      
     SELECT * FROM arschema WHERE  name =  SRM:WorkInfo'
   >> T2864
   Select count(*) from T1994 where C10001950 = '40000'
     >> 10000

If the migration is successful, the count of records before and after migration should be the same.

The audit form looks like the following image after running the data migration tool:

To obtain the number of records for other items before migration

Use the following commands to calculate the number of records for other items before migration (as the script in the previous section is only for events). These commands might take some to run depending on your Mongo DB size.

To obtain the ‘subscribedBy’ count per ‘activities’ document:

db.activities.aggregate({$project: {count:{$sum:{$size:"$subscribedBy"}}}})

To obtain the total ‘subscribedBy’ items:

db.activities.aggregate({$group: { _id: null, totalSubscribedBy: { $sum: { $size: "$subscribedBy"}}}})

To identify the count of ‘resourceFollowing’ per user:

db.users.aggregate({$project: {resourceFollowingCount: {$size:"$resourceFollowing"}}})

To get a total number of ‘resourceFollowing’ items for all users:

db.users.aggregate({$group: { _id: null, totalResourceFollowingAllUsers: { $sum: { $size: "$resourceFollowing"}}}})

To rerun the migration utility

You don't need to delete any record if the migration utility fails for any reason and you can rerun the utility. If the Failed Object IDs field is not empty, check the migration.log file for any failure message and run the migration again in rerun mode and specify the RERUN_IDs. You need to check the SMT:MIG:Social_Mongo_Audit form to get the Audit ID, which is required to rerun the utility.

To troubleshoot migration issues

The mongo migration utility captures the IDs of failed records in the Failed Object IDs field of the SMT:MIG:Social_Mongo_Audit form.

After the migration, if the Failed Object IDs field is empty, it indicates that all records have been successfully migrated.
If your system or the utility crashes during migration, you can resume the migration by running the utility in rerun mode. Use the following command format to run the utility in rerun mode:
mongo-migration.bat -r <rerun_id>

Where to go from here

Performing the Smart IT upgrade

Was this page helpful? Yes No Submitting... Thank you

Last modified by Nilay Agambagis on Apr 12, 2019

migration migration_activity mongo_db social_data social_data_configurations migration_script mongo_db_migration data_migration

Comments

Alok Dewhare

Hi Team,
Please modify the information as mention in the community as its missing in this page,
https://communities.bmc.com/message/780727#780727
Regards, Alok

Aug 30, 2018 06:49
1. Nilay Agambagis
  
  Hello Alok,
  Thanks for your comment.
  Please note the new section 'To rerun the migration utility' in this topic.
  Regards,
  Nilay Agambagis
  
  Aug 31, 2018 01:59
Marek Ceizel

For dwp it is possible to start a script and get the number of records which should be migratet and time how log would it take (roughly). Here I get the number of records using the mongo script, but I have no idea how long can it take (for us it is 822 000 entries). Any tip ?

Jan 17, 2019 07:53
1. Dhanya Menon
  
  Hello Marek,
  Thank you for your comment.
  We are discussing with the SME. We will get back to you soon.
  Thanks,
  Dhanya Menon
  
  Jan 24, 2019 03:26
1. Dhanya Menon
  
  Hello Marek,
  This utility is only run to migrate MongoDB data for Smart IT.
  For DWP related information, please see the following link:
  Performing the BMC Digital Workplace upgrade#Tomigrateyoursocialdataandupdatethedatabaselocation
  Regards,
  Dhanya Menon
  
  Jan 28, 2019 12:40
  1. Marek Ceizel
    
    Hello Dhanya,
    I'm not interested into DWP Data Migration. That one is done by the installer. I'm interested into SmartIT MongoData Migration. The Mongo script here is showing only System Event. But we got either hundreds thousands records for follow and for that here is no script how to find out and prepare. We want to do in-place upgrade and have no idea how long it will take as this page is missing basic help + description what records are actually the follow, event , knowledge etc . We need all this info to plan what, when and if will be migrated.
    regards Marek
    
    Jan 28, 2019 04:12
    
    Nilay Agambagis
    
    Hello Marek,
    Thanks a lot for your comment.
    To set the required parameters in the mongo-migration-script.bat, please refer to the readme file. The parameter details are described in the readme file.
    Please see the To verify the migration section in this topic on the information on the taken time and comparing the number of records that are migrated.
    Regards,
    Nilay Agambagis
    
    Jan 29, 2019 04:05
    
    Marek Ceizel
    
    Nilay, I really can read what is present on this page. But I speak about what is missing. The mongo script returns only the number of System Events. No every records which will be migrated if I use all. Please read it carefully
    
    Jan 30, 2019 05:53
Marek Ceizel

Answer from supportt. To get the number of records for other ITEMs before migration (as the script above is only for Events):
To count the ‘subscribedBy’ count per ‘activities’ document:- db.activities.aggregate({$project: {count:{$sum:{$size:"$subscribedBy"}}}})
To get the total ‘subscribedBy’ items: (I had this one the biggest) db.activities.aggregate({$group: { _id: null, totalSubscribedBy: { $sum: { $size: "$subscribedBy"}}}})
To identify the count of ‘resourceFollowing’ per user, we can run:- db.users.aggregate({$project: {resourceFollowingCount: {$size:"$resourceFollowing"}}})
We can group these to get a total number of ‘resourceFollowing’ items all users:- db.users.aggregate({$group: { _id: null, totalResourceFollowingAllUsers: { $sum: { $size: "$resourceFollowing"}}}})

Feb 14, 2019 07:36
1. Dhanya Menon
  
  Hello Marek,
  Thank you for sharing this information. We have updated this page.
  Regards,
  Dhanya Menon
  
  Apr 12, 2019 06:33

Log in or register to comment.

Creating database for Windows authentication during a Smart IT upgrade

Performing the Smart IT upgrade