Migrating PostgreSQL data

1. Stop all TrueSight Capacity Optimization services and ETL data collection. For details on stopping the services, see System-level administration overview.
2. Based on the database and repository size, ensure that the sufficient space is available on the dump location.
3. Create a database dump.
   
   1. Copy the migration toolkit to a directory on the TrueSight Capacity Optimization Database Server that runs the PostgreSQL database.
   2. Log in to the Database Server.
   3. Ensure that the database utilities are present inside the user PATH variable.
      1. If database binaries are installed in the /usr/pgsql-11/bin/ folder, run the following command:
         export PATH=/usr/pgsql-11/bin/:$PATH
      2. Verify the database version:
         pg_dump --version
   4. Extract the contents of the toolkit file.
   5. At the shell prompt, change to the directory where you extracted the toolkit files and navigate to the database-specific folder. For example, cd database/PostgreSQL.

   6. Run the following command:

      bash dump/db_dump.sh -h <database FQDN hostname> -u <CO DB user name. For example, BCO_OWN> -p <CO DB user password> -s <DB name. For example, BCO_OWN>
      -P <DB PORT - optional - default: 5432> -l <path to store dump>

      The db_dump.sh utility displays the following message with the parameters. Note down these parameters as they are used to configure the Data Gap Recovery ETL. For details, see Configuring the Data Gap Recovery ETL.

      Information===============================================
      Data recovery parameters for recovery ETL
          sysobjid:  10751
          wkldobjid: 9458
          appobjid: 5689
          start time: 2019-12-1803:38:00
      ===============================================You can also see this message in the db_dump.log file that is generated in the same folder.

4. Create a dump for repository content and encryption keys.
   
   1. Copy the migration toolkit archive to the host computer on which the Application Server is running and log in to it.
   2. At the shell prompt, change to the directory where you extracted the toolkit files and navigate to the repository folder.

   3. Run the following command:

      bash repository/repository_and_keys_backup.sh -c <app server installation directory> -r <CO repository directory> -l <path to store dump>

      When you run the db_dump.sh and repository_and_keys_backup.sh commands, the following dump files are generated at the specified path:

      • db_dump.tar.gz
        
      • repository_backup.tar.gz
      • certificates_backup.tar.gz
5. Restart all TrueSight Capacity Optimization services and ETL data collection.

Note: After you start with the migration activity, avoid creating any new artifacts. Failing to do so, after the migration, those artifacts will be lost in BMC Helix Capacity Optimization.

1. Log in to the Database Server.
2. Ensure that the database utilities are present inside the user PATH variable.
   1. If database binaries are installed in the /usr/pgsql-11/bin/ folder, run the following command:
      export PATH=/usr/pgsql-11/bin/:$PATH
   2. Verify the database version:
      pg_dump --version
3. Create a historical database dump.
   1. Extract the contents of the toolkit file.
   2. At the shell prompt, change to the directory where you extracted the toolkit files and navigate to the database-specific folder. For example, cd database/PostgreSQL.

   3. Run the following command:

      bash dump/history_db_dump.sh -h <db host> -u <CO DB user name. For example, BCO_OWN> -p <BCO_OWN pwd> -s <DB name. For example, BCO_OWN> -l <path to store historical dump> -P <DB PORT - optional> -a <max aging in days - optional>

      The history_db_dump.sh command generates the history_db_dump.tar.gz file at the specified path.

To request a database restore in BMC Helix Capacity Optimization, submit a request from your support portal. For details, see Database backups and restores.

Transfer the following dump files to the BMC team:

• db_dump.tar.gz
  
• repository_backup.tar.gz
• certificates_backup.tar.gz
• history_db_dump.tar.gz

The BMC team will perform the database restore and activation process.

After you get the confirmation from the BMC team, proceed to the next step.

Some default user groups and roles  are available out-of-the-box when you subscribe to BMC Helix Capacity Optimization . You can start using the product with these users. Additionally, you can configure your identity provider to set up more users and user groups, and assign specific permissions to them. BMC Helix Capacity Optimization  leverages Helix Single Sign-On to provide single sign-on authentication for users. For details, see Managing users and access control.

Download and install the Remote ETL Engine and related components that are needed to collect data from your environment. For details, see Installing-remote-components-to-collect-on-premises-data.

After you get the confirmation from the BMC team about successful data restore, validate the restored data. Check all ETLs, data marts, and views customization and perform a sanity check of the environment.

Delta data can be any data changes that occur in the TrueSight Capacity Optimization environment during the data migration and BMC Helix Capacity Optimization activation period. If required, you can recover this data gap between the TrueSight Capacity Optimization and BMC Helix Capacity Optimization instances. You can take one of the following actions:

• Ignore the delta data if you do not need it
• Recover delta data for data sources that support historical recovery.
  • For this, ensure that the last counters property is set in the ETLs. When you rerun such ETLs on the BMC Helix Remote ETL Engines, they start to collect the data from the last data import before the dump was taken, ensuring that data gap is recovered.
  • Your data sources must have the sufficient data retention for this approach to work.
• Fully recover delta data using the Data Gap Recovery ETL provided with the migration toolkit.
  • This custom ETL is written in the Java source code and provided as an EDK project in the migration toolkit. It is located in the <migration_toolkit_archive>/etl/dataGapRecoveryETL folder. If required, you can customize the code.
    Using this ETL, you can:
    • Extract all time series samples from the TrueSight Capacity Optimization database at an hourly resolution for all systems/business drivers till the value of Series per batch that you configure in the ETL.
    • Extract samples in batches, so that the TrueSight Capacity Optimization database is not overloaded.
    • Limit data extraction per each run, providing a way to control data import in BMC Helix Capacity Optimization.

  • Activate and configure the Data Gap Recovery ETL:

    1. Activate this ETL on the Remote ETL Engine in BMC Helix Capacity Optimization using the ETL Development kit. For details, see Creating-and-managing-custom-ETLs.
       After activation, this ETL module is displayed in the list of ETL modules that are available to the administrators when creating a new ETL instance.
    2. Configure the ETL:
       1. Log in to the BMC Helix Capacity Optimization console.
       2. Navigate to Administration > ETL & System Tasks, and select ETL tasks.

    1. 1. On the ETL tasks page, click Add > Add ETL.
       2. On the Run configuration tab, select the Data Gap Recovery ETL module that is displayed in the ETL Module list. 
          
          
       3. On the Entity catalog tab, select Private Entity Catalog.
       4. On the Custom properties tab, specify the parameters that were generated after you run the db_dump.sh utility in Step 2. Also specify Series per batch and Max samples per run.
          
          
       5. On the Connection parameters tab, specify the TrueSight Capacity Optimization database details.
       6. Click Save. The ETL tasks page shows the details of the newly configured ETL.
       7. Click the ETL and click Run active configuration. A confirmation message about the ETL run job submission is displayed.

    Note: BMC Helix Capacity Optimization is configured with an hard data import limit per day. Data recovery has to be properly scheduled to avoid reaching such limit or limit has to be reconfigured based on needs.