Estimating your data size for migration with the AREntryCounter utility

In a staged upgrade, after you replicate the production database and perform the upgrade steps, you begin the delta data period. This means that you are still actively adding data in various BMC Remedy ITSM forms on your current production system.

Before you can migrate this delta data, you must first estimate the volume of data with the AREntryCounter utlity, which will determine the exact count of the records in your forms. With the data estimates, you can then plan your delta data migration (DDM) runs for specific date ranges.

For products with a large volume of data, you might have to run DDM a few times with a smaller date range.

You run the AREntryCounter utility multiple times during the DDM process, as follows:

•  Before any DDM run, check for additional data by running the utility on the destination server for all the products. Use the DB backup date and time for the run.
• Before your first DDM run on the source server, to determine the data counts and estimate the volume to plan the number of DDM runs that will be required.
• After your first DDM run, on the destination server, to search for additional data.
• After your first and all subsequent runs on the destination server, to validate counts.
• After the final DDM run for the full delta period on both the source and destination servers, to validate counts.

Determining data size

Before your start your data migration, perform the following steps on the source server to determine the size of your data:

1. On the source server, run the AREntryCounter utility on all the packages with only the primary DDM start date (-d) and no end date.
    For more information about working with the utility, see Running the AREntryCounter utility.
2. Review all the output count text files to check the volume of data.

3. Based on the volume of data, plan your DDM runs in batches with the start date and end date for each runs.

   Tracking DDM run details

   Use a spreadsheet to record the batch run details to track each run, re-run, and count validation results.

4. Run the utility with a shorter start date and end date intervals to determine the number of DDM runs that you need to perform.
   For example, you find that based on the count files, certain ITSM Foundation and BMC Atrium CMDB forms have a large volume of delta data. You must split and run multiple DDM batches with shorter date intervals for faster data migration.
5. Based on your data volume and your schedule, you might run DDM for all products in parallel for two or more date ranges in multiple Microsoft Windows machines. These machines must have BMC Remedy Migrator installed.
6. Optionally, if the custom and backend forms have a large amount of data, you can use the SQL method to migrate a large portion of that data.
   You can use DDM for the subsequent runs. For more information about migrating custom and backend forms, see Performing data migration of backend and custom forms.

Searching for additional data

After you determine your data size, you must check if the data created by the upgrade process conflicts with the entry IDs that are created or updated on the source server. In such a scenario, you must first backup the data that was created or updated by the installer and then delete that data on the destination server. After the final DDM run, import that data back. Perform the following steps on the destination server to search for additional data, before you start your data migration:

1. Check for post-upgrade additional data on the destination server.
   1. Record the timestamp when the last upgrade was successfully completed.
      You can determine this information from the installation log files.
   2. Run the AREntrycounter utility with the start date parameter for all the packages.
   3. Review the count.txt files generated by the utility and search for counts greater than "0".
      You must delete any post-upgrade data that is not created by the installer.

   4. Perform the following steps to delete any post-upgrade data:

      1. Disable any delete filters on the form that are in development mode, by using the Developer studio.

2. On the destination server, perform the following steps to check for the data created by the upgrade installer.

   1. Capture the DB backup time value from the Source (Current Production) server.
      The backup time must be the date and time when the DB backup was restored to the staging server to perform the upgrade.

   2. Run the AREntrycounter utility with the start date parameter for all the packages.

   3. Review the count.txt files generated by the utility and search for counts greater than "0.

   4. Export, save, and then delete any data that was created by the upgrade installer.
      This is because such records will generate ID conflicts when you migrate the delta data.

   5. Using the Developer studio, disable any delete filters on each of those forms in the base development mode, and then delete the data that was created by the upgrade installer on those forms.

      DDM generates errors for these issues if you were unable to find and delete them before any DDM runs. A Mismatch EntryID error is generated. To migrate the data without any errors, perform steps 2a through 2e in this procedure.

3. Ensure that you follow these guidelines when using the AR Import tool options:

   1. Use the same unique index value as the one in the DDM xml files for importing.

   2. Do not use request ID.

   3. Auto-map all fields and then remove request ID (field ID 1), if it is mapped.

   4. Select suppress filters.

Validating counts after every DDM run

Perform the following steps, after every DDM run:

1. On the source server, obtain the counts for the specific date range and product selection.
2. After you completed the DDM run for the specific date range and product selection, obtain the counts on the destination server for the same date and product selection as on the source server.
   Perform this step to ensure that the upgrade process created no additional records on the destination server, except for the configuration and other forms (Audit/AuditLogSystem). In case of Audit forms, additional data might have been created if the parent forms had Audit turned ON. If you found additional records or data during the comparison, you can safely ignored.
   

Validating final counts after completing migration

Perform the following steps, after you migrate your data:

1. On the source server and destination servers, run the AREntryCounter utility with the primary DDM date on all the packages with the start date and no end date parameter.
   For more information about working with the utility, see  Running the AREntryCounter utility.

2. On the source and destination servers, generate and review all the output count text files.
   

   Perform this step to ensure that the upgrade process created no additional records on the destination server, except for the configuration and other forms (Audit/AuditLogSystem). In case of Audit forms, additional data might have been created if the parent forms had Audit turned ON. If you found additional records or data during the comparison, you can safely ignore.
   

Running the AREntryCounter utility

The AREntryCounter utility is available in the \DeltaDataMigration\Utilities\migratorutilities directory. You must run the counts for each product. The output file is available in the same folder from which you run the Delta Data Migration Tool. You run the utility commands at the command prompt.

AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\ARCore\<YOUR VERSION>\AR_Core_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u <USERNAME> -p <PASSWORD> -x PORT -o AR_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\Atrium\CMDB\<YOUR VERSION>\AtriumPackage.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p <PASSWORD> -x <PORT> -o AtriumCMDB_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\Atrium\AIE\<YOUR VERSION>\AIE_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p <PASSWORD> -x <PORT> -o AtriumAIE_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\Atrium\ProductCatalog\<YOUR VERSION>\ProductCatalogpackage.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p "PASSWORD" -x PORT -o AtriumPCT_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\ITSM\<YOUR VERSION>\ITSM_AST_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p <PASSWORD> -x PORT -o Asset_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\ITSM\<YOUR VERSION>\ITSM_Change_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p <PASSWORD> -x PORT -o Change_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\ITSM\<YOUR VERSION>\ITSM_SD_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p <PASSWORD> -x PORT -o ServicDesk_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\ITSM\<YOUR VERSION>\ITSM_Fnd_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u <Demo> -p <PASSWORD> -x PORT -o ITSMFoundation_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\SRM\<YOUR VERSION>\SRM_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p <PASSWORD> -x PORT -o SRM_counts.txt 
 
AREntryCounter.bat -k "<MigratorInstallationPath>\DeltaDataMigration\Packages\SLM\<YOUR VERSION>\SLM_Package.xml" -d <DDM DATE> -s <SERVERNAME> -u "Demo" -p <PASSWORD> -x PORT -o SLM_counts.txt

Product forms with additional data

The BMC Remedy ITSM version 9.1.02 SP2 installer creates additional data in the following forms, as explained in the Search For Additional Data section:

Where to go from here