Running the migration tool

As an administrator, run the migration tool to migrate your data from TrueSight Capacity Optimization to BMC Helix Continuous Optimization.

Before you begin

Before you migrate data, make sure that the following requirements are met:

You have administrator privileges for accessing TrueSight Capacity Optimization.
TrueSight Capacity Optimization is upgraded to version 20.02. To migrate data from a version earlier than 20.02, you must first upgrade to 20.02 and then migrate data to BMC Helix Continuous Optimization.
You have reviewed key considerations.
You have prepared your target environment by completing the premigration tasks.

To migrate data

Perform the following tasks:

As an OS user, log in to the host computer on which the TrueSight Capacity Optimization Application Server is installed and the migration toolkit is copied as described in the premigration tasks. The OS user is the user that you specified during the Application Server installation (for example: cpit).
Do I need to stop any services in the source environment?
No. You need not stop any TrueSight Capacity Optimization services and ETL data collection. Migration activity can be performed parallelly without needing to stop any services in the source environment.

Extract the contents of the toolkit file. Make sure that the OS user has the read and write permissions to the extracted migration toolkit folder.
The extracted folder contains the following directory structure and files:

Folder or file	File	Description
bin	NA	Contains the jar file of the migration toolkit.
conf		Contains customizable properties and configuration files.
	application.properties	Contains the parameters of the target environment to which you want to migrate data. You need to specify the values of the following mandatory parameters. helix.host: FQDN of BMC Helix Portal; for example, helixhost.onbmc.com helix.tenant: ID of BMC Helix tenant helix.accessKey: Access key to authenticate the user who is performing the migration to connect to the BMC Helix tenant helix.accessSecretKey: Secret key to authenticate the user to connect to the BMC Helix tenant You can add the following optional parameters for specific scenarios: dry-run: Set this parameter to true if you want to perform a dry run of the migration and avoid migrating any real data to the target environment. Best practice: We recommend that you perform the dry run of the migration utility by using the dry-run=true parameter before you run the actual migration. migrate-datamarts-and-views: Set this parameter to false if you do not want to migrate datamarts and views. By default, data marts and views are migrated. To run the migration tool by using the proxy server, add the parameters for the proxy server. For details, see (Optional) To run the migration tool by using the proxy server.
	cpit.properties	Contains internal configurations for the migration toolkit.
	ds.properties	Contains configurations to connect to the source database.
	log4j.conf	Contains log configurations.
	Important: You need to configure the application.properties file only. Other files are used by advanced users or developers.
run.sh		Script used to launch the migration toolkit.

At the shell prompt, change to the directory where you extracted the toolkit files.
Tip
You can run the ./run.sh command to view the help for the available options.
Run the following command to provide the execute permission for the run.sh script to the OS user.
chmod +x run.sh
Configure the application.properties file.
1. Navigate to the /conf folder and edit the application.properties file.
2. Specify the host URL, tenant ID, and API key values that you obtained during the prerequisites procedure, and save the file.
Run the following command to set CUSTOM_JAVA_HOME to the folder with JDK17 installed in the preparation steps:
export CUSTOM_JAVA_HOME=[directory path of the JDK17]
For example:
export CUSTOM_JAVA_HOME=/data/jdk17/jdk-17
Run the following command to start the migration:
./run.sh -b [environment variable/base directory of Capacity Optimization]
For example:
./run.sh -b /data/bmc/BCO/
The migration tool creates a dump of configurations in the migration-toolkit.tar.gz file and sends this file to BMC Helix Continuous Optimization.
Best practice
We recommend that you run the migration tool only once, as it imports all data in its single run.The data is migrated in two phases:
- In phase 1, the migration tool extracts all product configurations, such as the definition of entities, relationships, tags, analysis, models, external DB connections, and entity catalogs, and migrates this data to BMC Helix Continuous Optimization. This Migrator system task might take up to few minutes to be completed. During the same phase, the tool also creates a set of data migration ETLs in BMC Helix Continuous Optimization
  After this task is complete, review the ORACLE_HOME path in BMC Helix Continuous Optimization > Administration > ETL & SYSTEM TASKS > External DB connections. Specify the ORACLE_HOME path for the Oracle client of the Helix Remote ETL Engine if this path is not the same as the path for the Oracle client of the TrueSight Remote ETL Engine.
  For more information, see Adding-external-database-connections.
- In phase 2, the Remote ETL Engine schedules a set of data migration ETLs created in phase 1 to migrate the historical data from the source environment. During the first run of each migration ETL, all configuration metrics data, with the first batch of recent performance metrics data, is migrated. In subsequent runs of these ETLs, the remaining performance metrics data is migrated in batches. You can configure the duration for the data to be migrated in each batch in the Days to extract property in the data migrator ETL (default value is 7). For example, you can see all configuration metrics data and the performance metrics data for the last 7 days after the first run, and the performance metrics data for the 7 days previous to the last 7 days after the second run.
  Depending on the size of the historical data, it might take from a few hours to several days to completely migrate the historical data.
Monitor the progress and status of the migration and logs.
- Review the CSV file created for each migration ETL to check the progress of historical data migration for every entity in percentage.
  - The CSV files for all migration ETLs are located in <Remote ETL Engine Installation Directory>/etl/migrationEntityIdsOutput/<etl_id>.csv and are overridden with every run of the ETL.
  - Each CSV file includes the following details:
    - The ID of the ETL
    - The ID, name, and type of the migrated entity
    - The status of the migration in percentage. This value is calculated based on the first and last extraction dates in the last counter and the min and max dates of entity metrics.
      
      Each migration ETL includes a list of the last counters specific to the ETL instance. The Status detail table for migration ETLs displays the timestamp and last counter for the migration start date under FIRST_EXTRACTION_DATE.
      
      For details on lastcounter, see Overview-of-the-ETL-workflow.
      
      Click here to preview the example CSV file
- Check if the Migrator system task in BMC Helix Continuous Optimization is completed successfully with no errors. For details about running or scheduling a system task, see Managing-ETL-and-System-tasks.
Validate data in BMC Helix Continuous Optimization.
Validate whether all ETLs, historical data, and Workspace data are migrated in BMC Helix Continuous Optimization and perform a sanity check of the environment.

(Optional) To run the migration tool by using the proxy server

To run the migration tool by using the proxy server, perform the following steps in addition to the steps listed in the To migrate datasection.

Navigate to the /conf folder and edit the application.properties file.

Add the following parameters and specify their values:

Parameter	Description	Example
proxy.host	Name of the proxy server host.	proxy.host=vw-abc-dev01.bmc.com
proxy.port	Port number to communicate with the proxy server.	proxy.port=8090
proxy.username	Proxy server user name.	proxy.username=proxy_user
proxy.password	Proxy server password.	proxy.password=proxy_user_password

Save the file and continue with Step 3 listed in the To migrate datasection.

(Optional) To improve the migration ETL performance by using the concurrent database threads

By default, the migration ETL uses one database thread to extract the performance metrics from the TrueSight Capacity Optimization database. However, you can configure multiple threads to enable the ETL to process concurrent database requests. Each thread within the multithreaded ETL can carry out a database request separately, resulting in faster data migration.

Perform the following steps to configure the multiple threads by modifying the run configuration of the data migrator ETL:

Log in to BMC Helix Continuous Optimization.
Select Administration > ETL & System tasks > ETL tasks.
On the ETL tasks page, click the name of the data migrator ETL.
In the Run configurations table, click Edit this run configuration .
On the Run configuration tab, click the You can manually edit ETL properties from this page link.
In the Add new property field at the bottom of the page, type the name of the property as extract.db.concurrent.calls and click Add.
Locate the new property in the list of properties, type the required number for the threads that will run parallelly, and click Save.
Make sure that this value is not blank.
Best practice
Consider the following points when deciding the number of threads:
- All parallel threads share network and system resources; therefore, consider carefully the available resources of your system and network when determining the number of threads to use. Start with as many threads as your database and system resources reasonably support. For small environments, we recommend that you start from the value of 5 and gradually increase it after monitoring the time taken for the ETL run and the load on the TrueSight Capacity Optimization database.
- If you are migrating from the live TrueSight Capacity Optimization environment, more threads might create a load on the database. However, depending on the size of the TrueSight Capacity Optimization database or how fast you want the migration to be completed, you can consider adding more threads. For large environments, we recommend that you start from the value of 30 and then gradually increase it.
Click Run active configuration to re-run the data migrator ETL.

Post migration tasks

Perform the following post migration tasks:

Some default user groups and roles are available out of the box when you subscribe to BMC Helix Continuous Optimization. You can start using the product with these users. Additionally, you can configure more users and roles in BMC Helix Portal.
(Optional) If you missed to configure the service account key file for GCP API Extractor to be read from the upload compressed folder option as suggested in Preparing-for-migration, you need to reconfigure the service account key file setting after migration. For details, see Google-Cloud-Platform-GCP-API-Extractor.

Troubleshooting

Although all the data marts are migrated, some of the SQL data marts might be in the Error or Warning status in BMC Helix Continuous Optimization after migration. You need to resolve issues in such data marts. For more information, see Resolving-SQL-data-mart-issues-after-migration.
If you have performed the migration by using the migration toolkit provided in version 24.4.01, data for some of the configuration metrics created in TrueSight Capacity Optimization might be missing. To resolve this issue, modify the run configuration of the data migrator ETL to reimport the configuration metrics. For details, see Troubleshooting-issues-with-the-migration-of-configuration-metrics.
If you have performed the migration by using the migration toolkit provided in versions earlier than 23.3.02, the minimum and maximum values for metrics of migrated entities might be missing. To resolve this issue, provide a list of these entities in a custom property of the data migrator ETL and rerun the ETL. For details, see Troubleshooting-issues-with-the-migration-of-min-and-max-values-of-entities.