Skip to Content

Replicating configurations and data from a source environment to a target environment

Replicate configurations and data from one environment to another by using the database refresh pipeline.

Related topics

Performing the BMC Helix Service Management installation

Warning

Important

You can use the database refresh pipeline for PostgreSQL, Microsoft SQL Server, and Oracle databases.

The following scenarios describe the use case for replicating environment configurations and data:

Information
Scenario 1

Apex Global wants to implement BMC Helix Service Management to address its business needs. The company plans to set up BMC Helix Service Management in both its non-production and production environments.

A system administrator at Apex Global installs BMC Helix Service Management  in both the non-production and production environments. The administrator then configures the nonproduction environment with the required configurations. To replicate the configurations in the production environment, the system administrator uses the database refresh pipeline.

Information
Scenario 2

A new feature has been added to BMC Helix Service Management in a recent hotfix. Apex Global is interested in using this feature and wants to test it before implementing it in their production environment.

A system administrator at Apex Global replicates the production environment configurations in the QA environment by using the database refresh pipeline. The system administrator tests the feature in the QA environment and then implements it in the production environment.

Information
Scenario 3

A new version of BMC Helix Service Management has been released. Apex Global is interested in upgrading BMC Helix Service Management to the latest version and wants to test the upgrade before upgrading their production environment.

A system administrator at Apex Global replicates the production environment configurations in the QA environment by using the database refresh pipeline. The system administrator tests the upgrade in the QA environment and then upgrades the production environment.

Use the database refresh pipeline to replicate the following Foundation data, application data, and application configurations from a source environment to a target environment:

  • BMC Helix Innovation Suite configurations
  • BMC Helix ITSM: Smart Reporting configurations
  • BMC Helix Single Sign-On OAuth configurations
  • BMC Digital Workplace Catalog connectors configurations
  • BMC Digital Workplace configurations
  • Cognitive and BMC Helix Chatbot configurations
  • BMC Helix ITSM users
  • Web services filters
Warning

Important

  • You can run the database refresh pipeline on BMC Deployment Engine, that is the Jenkins server, on the source environment or target environment.
  • The kubeconfig file on the specified Jenkins server is used during pipeline execution.

The following image shows the replication process:

Backup and restore process.png

Before you begin

  • Make sure that the source and target environments are using the same BMC Helix Service Management version.
  • Make sure that you can access the source and target databases from the Jenkins server from where you plan to run the database refresh pipeline. Without this access, the pipeline will fail.
    Verify that you have access to both source and target environments. 

  • Make sure that the PostgreSQL client version on your BMC Deployment Engine where you plan to run the pipeline and the target database PostgreSQL client version are the compatible.

  • Make sure that your BMC Helix Service Management target environment is licensed and that the number of licenses on the target environment and source environment are the same.
  • Make sure that you copy the source and target configuration repositories.
  • (For non-containerized BMC Deployment Engine) Make sure that you add the database packages.
    Install the following packages:

    • To ensure that the autorecon utility runs successfully, install the gtk3 package by running the following command:

      yum install -y gtk3

    • If you are using a Microsoft SQL Server database, install the following packages so that the HELIX_DB_REFRESH  runs successfully:

      sudo yum install unixODBC
      sudo yum install python3-pyodbc

    • If you are using a PostgreSQL database, install the following packages so that the HELIX_DB_REFRESH runs successfully:

      sudo yum install postgresql-client
      sudo yum install python3-psycopg2

To copy the source and target configuration repositories

For non-containerized BMC Deployment Engine, perform the following steps:

  1. In the source environment, copy the CUSTOMER_CONFIGS repository located in the directory that contains all the Git repositories in BMC Deployment Engine. Rename it to SOURCE_CUSTOMER_CONFIGS.
  2. In the target environment, copy the CUSTOMER_CONFIGS repository located in the directory that contains all the Git repositories in BMC Deployment Engine. Rename it to TARGET_CUSTOMER_CONFIGS.
  3. Copy the SOURCE_CUSTOMER_CONFIGS and TARGET_CUSTOMER_CONFIGS repositories to the environment where you want to run the database refresh pipeline.
    For example, to run the database refresh pipeline in the target environment, make sure that the following repositories are present in the git repository of the target environment:
    • SOURCE_CUSTOMER_CONFIGS
    • TARGET_CUSTOMER_CONFIGS
    • DB_REFRESH_REPO

For containerized BMC Deployment Engine, perform the following steps:

  1. To access the source and target environment configuration files, use the following URL:
    https://<Gitea-URL>/ciadmin/onprem-remedyserver-config/src/branch/master/customer/<CUSTOMER-SERVICE>/<CUSTOMER-SERVICE>-<ENVIRONMENT>.sh

  2. ​​​​​​To access the Smart Reporting input file, use the following URL:
    https://<Gitea-URL>/ciadmin/onprem-remedyserver-config/src/branch/master/customer/<CUSTOMER-SERVICE>/sr-input-file.sh

  3. Download the files on the source environment.

  4. Upload the files to target environment by using the Add File option in the target environment.
    Do not use the existing file name while uploading the files.​​​​​

The target environment must include two environment files for BMC Helix Innovation Suite and two sr-input-file.sh files for Smart Reporting, which are available if you are using Smart Reporting.

(For non-containerized BMC Deployment Engine) To add database packages

Add the the following database packages to BMC Deployment Engine based on your database and operating system:

DatabasePackagesRocky 8RHEL 8Rocky 9RHEL 9
Microsoft SQL Server
  • pyodbc
  • unixODBC
  • msodbcsql17
  1. Download the following package:
     https://packages.microsoft.com/config/rhel/8/prod.repo
  2. Run the following commands:
    yum install -y unixODBC
    yum install -y python3-product
    yum install -y msodbcsql17
  1. Download the following package:
    https://packages.microsoft.com/config/rhel/8/prod.repo
  2. Run the following commands:
    yum install -y unixODBC
    yum install -y python3-pyodbc
    yum install -y msodbcsql17
  1. Download the following package:
    https://packages.microsoft.com/config/rhel/9/prod.repo
  2. Run the following commands:
    yum install -y unixODBC
    yum install -y python3-pyodbc
    yum install -y msodbcsql17
  1. Download the following package:
    https://packages.microsoft.com/config/rhel/9/prod.repo
  2. Run the following commands:
    yum install -y unixODBC
    yum install -y python3-pyodbc
    yum install -y msodbcsql17
Oracle
  • oracledb
  • Oracle Instant Client
  1. As a git user, install the package by using the following command:
    python -m pip install oracledb
  2. Verify that the package is installed by using the following commands:
    python -m pip list | grep -i oracledb

    python -c "import oracledb; print(oracledb.version)"

 

  1. As a git user, install the package by using the following command:
    python -m pip install oracledb
  2. Verify that the package is installed by using the following commands:
    python -m pip list | grep -i oracledb

    python -c "import oracledb; print(oracledb.version)"
  1. As a git user, install the package by using the following command:
    python -m pip install oracledb
  2. Verify that the package is installed by using the following commands:
    python -m pip list | grep -i oracledb

    python -c "import oracledb; print(oracledb.version)"
  1. As a git user, install the package by using the following command:
    python -m pip install oracledb
  2. Verify that the package is installed by using the following commands:
    python -m pip list | grep -i oracledb

    python -c "import oracledb; print(oracledb.version)"
PostgreSQL
  • psycopg2
  • postgresql
  1. Download the following package:
    https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
  2. Run the following commands:
    yum  module -y enable postgresql:$postgresVersion
    yum  install -y postgresql
    yum -y install python3-psycopg2
  1. Download the following package:
    https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
  2. Run the following commands:
    yum  module -y enable postgresql:$postgresVersion
    yum  install -y postgresql
  1. Download the following package:
    https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
  2. Run the following commands:
    yum  module -y enable postgresql:$postgresVersion
    yum  install -y postgresql
    sudo dnf install  -y postgresql$postgresVersion-contrib postgresql$postgresVersion-devel
  1. Download the following package:
    https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
  2. Run the following commands:
    yum  module -y enable postgresql:$postgresVersion
    yum  install -y postgresql
    sudo dnf install  -y postgresql$postgresVersion-contrib postgresql$postgresVersion-devel

 

Database refresh pipeline modes and parameters

Use the database refresh pipeline, HELIX_DB_REFRESH, to replicate configurations from one environment to another. 

The database refresh pipeline, HELIX_DB_REFRESH, provides the following modes:

  • BACKUP_CONFIGURATION—In this mode, the HELIX_DB_REFRESH pipeline backs up the configurations of the target environment.
  • RESTORE_CONFIGURATION—In this mode, the HELIX_DB_REFRESH pipeline restores configurations from one environment to another.
Warning

Important

  • You must run the HELIX_DB_REFRESH pipeline separately to backup and restore configurations. First, run the pipeline in BACKUP_CONFIGURATION mode and then in the RESTORE_CONFIGURATION mode.You cannot perform backup and restore in a single pipeline run.
  • Before running the HELIX_DB_REFRESH pipeline, make sure that all the platform and application pods are in a healthy state.

The database refresh pipeline, HELIX_DB_REFRESH, contains the following input parameters:

(For non-containerized BMC Deployment Engine)

Parameter

Description

RERUN

Select this check box to rerun the pipeline from where it failed in the last run.

BACKUP_CONFIGURATION

Select this check box to back up configurations in the target environment.

RESTORE_CONFIGURATION

Select this check box to restore source database backup configurations to the target environment.

LOCATION

Specify the temporary working directory location of the pipeline.

Example: /home/git/workspace

REPO_PATH

Specify the repository path name where the SOURCE_CUSOTMER_CONFIGS, TARGET_CUSTOMER_CONFIGS, and DB_REFRESH_REPO repositories are located.

Example: ssh://<Jenkins server host name>/home/git/git_repo

SOURCE_CUSTOMER_NAME

Specify the CUSTOMER_NAME parameter value that you provided in the HELIX_ONPREM_DEPLOYMENT Jenkins pipeline during the installation of BMC Helix Service Management on the source environment.

SOURCE_CUSTOMER_SERVICE

Specify the CUSTOMER_SERVICE parameter value that you provided in the HELIX_ONPREM_DEPLOYMENT Jenkins pipeline during the installation of BMC Helix Service Management on the source environment.

SOURCE_ENVIRONMENT

Specify the ENVIRONMENT parameter value that you provided in the HELIX_ONPREM_DEPLOYMENT Jenkins pipeline during the installation of BMC Helix Service Managementthe source environment.

The HELIX_DB_REFRESH pipeline uses the SOURCE_CUSTOMER_NAME, SOURCE_CUSTOMER_SERVICE, and SOURCE_ENVIRONMENT parameters to identify the Deployment Input Configuration file location in the source environment.

TARGET_CUSTOMER_NAME

Specify the CUSTOMER_NAME parameter value that you provided in the HELIX_ONPREM_DEPLOYMENT Jenkins pipeline during the installation of BMC Helix Service Management on the target environment.

TARGET_CUSTOMER_SERVICE

Specify the CUSTOMER_SERVICE parameter value that you provided in the HELIX_ONPREM_DEPLOYMENT Jenkins pipeline during the installation of BMC Helix Service Management on the target environment.

TARGET_ENVIRONMENT

Specify the ENVIRONMENT parameter value that you provided in the HELIX_ONPREM_DEPLOYMENT Jenkins pipeline during the installation of BMC Helix Service Management on the target environment.

The HELIX_DB_REFRESH pipeline uses the TARGET_CUSTOMER_NAME, TARGET_CUSTOMER_SERVICE, and TARGET_ENVIRONMENT parameters to identify Deployment Input Configuration file location in the target environment.

TARGET_EXT_IP

Specify the PLATFORM_ADMIN_PLATFORM_EXTERNAL_IPS parameter value that you provided in the HELIX_ONPREM_DEPLOYMENT pipeline during the installation of BMC Helix Service Management on the target environment.
See Performing the BMC Helix Service Management installation.

IS_DB

Select this check box to back up and restore BMC Helix Service Management configurations.

SMART_REPORTING

Select this check box to back up and restore BMC Helix ITSM: Smart Reporting configurations.

(For containerized BMC Deployment Engine)

Parameter

Description

RERUN

Select this check box to rerun the pipeline from where it failed in the last run.

BACKUP_CONFIGURATION

Select this check box to back up configurations in the target environment.

RESTORE_CONFIGURATION

Select this check box to restore source database backup configurations to the target environment.

LOCATION

Specify the temporary working directory location of the pipeline.

Example: /home/git/workspace

Important: Make sure that this location is a writable location.

REPO_PATH

Specify the repository path name where the DB_REFRESH_REPO repository is located.

Example: ssh://<Jenkins server host name>/home/git/git_repo

TARGET_SERVICESpecify the external IP or service name platform-admin-ext.<Innovation Suite namepsace>
SOURCE_ENVIRONMENT_FILE

Specify the source environment file location. 

You can copy the path from Gitea.

SOURCE_SR_INPUT_FILE

Specify the Smart Reporting source environment file location. 

You can copy the path from Gitea.

TARGET_ENVIRONMENT_FILE

Specify the target environment file location. 

You can copy the path from Gitea.

TARGET_SR_INPUT_FILE

Specify the Smart Reporting target environment file location. 

You can copy the path from Gitea.

IS_DBSelect this check box to back up and restore BMC Helix Service Management configurations.

SMART_REPORTING

Select this check box to back up and restore BMC Helix ITSM: Smart Reporting configurations.

In the HELIX_DB_REFRESH pipeline, the containerized BMC Deployment Engine parameters are displayed by default. If you are using a non-containerized BMC Deployment Engine and want to revert to the old parameters, perform the following steps:

  1. In the HELIX_DB_REFRESH pipeline, click Configure.
  2. Modify the value of the script path from the old value to the new value.

    Old value: jenkinsfile/helix_db_refresh.jenkinsfile

    New Value: jenkinsfile\helix_db_refresh_non_containerized.jenkinsfile

  3. Save the changes.
  4. Dry run the pipeline.

The non-containerized BMC Deployment Engine parameters are displayed.

Task 1: To back up the source and target environment

Take a complete backup of your source and target databases.
Use your preferred method to back up the databases.

Task 2:  To back up configurations of the target environment

Warning

Important

If you modify any password parameter in the HELIX_ONPREM_DEPLOYMENT pipeline, make sure that you run the HELIX_GENERATE_CONFIG pipeline before you run the HELIX_DB_REFRESH pipeline.

For non-containerized BMC Deployment Engine, perform the following steps:

  1. Log in to the Jenkins server by using the following URL:
    http://<Jenkins server host name>:8080
  2. On the Jenkins server, select the HELIX_DB_REFRESH pipeline.
  3. Select the BACKUP_CONFIGURATION check box and clear the RESTORE_CONFIGURATION check box.
  4. Specify the values for the LOCATION and REPO_PATH parameters.
  5. Specify the values for the SOURCE_CUSTOMER_NAME, SOURCE_CUSTOMER_SERVICE, and SOURCE_ENVIRONMENT parameters.
  6. Specify the values for the TARGET_CUSTOMER_NAME, TARGET_CUSTOMER_SERVICE, and TARGET_ENVIRONMENT parameters.
  7. Specify the value for the TARGET_EXT_IP parameter.
  8. Select the IS_DB check box to back up BMC Helix Service Management configurations.
  9. Select the SMART_REPORTING check box to back up BMC Helix ITSM: Smart Reporting configurations.
  10. Click Rebuild.

For containerized BMC Deployment Engine, perform the following steps:

  1. Log in to the Jenkins server by using the following URL:
    http://<Jenkins server host name>:8080
  2. On the Jenkins server, select the HELIX_DB_REFRESH pipeline.
  3. Select the BACKUP_CONFIGURATION check box and clear the RESTORE_CONFIGURATION check box.
  4. Specify the values for the LOCATION and REPO_PATH parameters.
  5. Specify the values for the SOURCE_ENVIRONMENT_FILE and SOURCE_SR_INPUT_FILE parameters.
  6. Specify the values for the TARGET_ENVIRONMENT_FILE and TARGET_SR_INPUT_FILE parameters.
  7. Select the IS_DB check box to back up BMC Helix Service Management configurations.
  8. Select the SMART_REPORTING check box to back up BMC Helix ITSM: Smart Reporting configurations.
  9. Click Rebuild.

The HELIX_DB_REFRESH pipeline runs in the BACKUP_CONFIGURATION mode. It creates a backup of the configurations and scales down all the application pods.

The pipeline creates the backup files and folders at the custom workspace location <LOCATION>\HELIX_DB_REFRESH\<Source_DB_Name>.

Task 3: To restore the source database backup to the target environment

After the database refresh pipeline is completed successfully in the BACKUP_CONFIGURATION mode, restore the source database backup to the target database.

Use your preferred method to restore the database backup.

After you restore the backup, the target database contains the source environment configurations and source environment data.

Task 4: To restore configurations to the target environment

  1. Log in to the Jenkins server on the environment where you want to run the pipeline by using the following URL:
    http://<Jenkins server host name>:8080
  2. On the Jenkins server, select the HELIX_DB_REFRESH pipeline.
  3. In the Build History, select the latest build and click Rebuild.
  4. Clear the BACKUP_CONFIGURATION check box.

  5. Select the RESTORE_CONFIGURATION check box.

    Warning

    Important

    Do not modify other parameter values.

  6. Click Rebuild.

The pipeline runs in the RESTORE_CONFIGURATION mode. It restores the configurations and scales up all the application pods.

After you restore the configurations, the target database contains the target environment configurations and source environment data.

Warning

Important

  • The HELIX_DB_REFRESH pipeline checks the status of all pods every two minutes and has a timeout of 60 minutes. If a pod fails to start during the Check Pod Status stage, restore the pod to a normal state before the timeout.
    When all the pods are up and running, the pipeline proceeds automatically.
  • If the clamav pod fails to start during the Check Pod Status stage, scale down the pod by using the following command:
    kubectl scale deployment clamav --replicas=0 -n <Target environment namespace>After the pipeline execution is completed, scale up the clamav pod by using the following command:
    kubectl scale deployment clamav --replicas=<replica count> -n <Target environment namespace>

Default state of configurations after database refresh

After a database refresh, several configurations are set to a default state. BMC recommends retaining these default settings. If the default configuration does not meet business requirements, the necessary changes must be applied.

For example, the Escalation ASJ:ScheduleJob is disabled by default after a refresh. If this is required for normal operations, you must enable it after the refresh is completed. For more information on configuration settings, see Configuration settings .

TaskDefault state after refresh (target)
Email service / mailbox

Unchanged

Email messages

Deleted

Smart Reporting broadcast service & schedules

Unchanged

Escalation ASJ:ScheduleJob

Disabled

Filter NGIE:EventJobStart / CTM:PPL:StatusRulesErr_032

Disabled

Custom Web Service filters

Restored

ITSM transactional data

Copied

Digital Workplace Catalog transactional data

Copied

Alerts / Notifications

Unchanged

Cognitive, Chatbot, RSSO, DWP, DWPC configurations

Restored

Pentaho connections

Disabled

KEK secret (arkeksecret)

Restored

Smart Reporting data source connections

Restored

Virtual Chat configuration

Restored

To rerun the HELIX_DB_REFRESH pipeline

If you encounter a failure in BACKUP_CONFIGURATION or RESTORE_CONFIGURATION mode while you are running the  HELIX_DB_REFRESH pipeline, use the RERUN option to rerun the pipeline.

  1. On the Jenkins UI, navigate to the HELIX_DB_REFRESH pipeline where the failure occurred.
  2. In the Build History pane of the pipeline where the failure occurred, select the latest build, and click Console Output.
  3. On the Console Output page, go through the logs to identify the reason for the failure.
  4. Debug and fix the issue.
  5. Rerun the HELIX_DB_REFRESH pipeline.
    1. Navigate to the pipeline where the failure occurred.
    2. In the Build History pane of the failed pipeline, select the latest build, and click Rebuild.
    3. On the Rebuild page, select the RERUN check box , and click Rebuild.
      The pipeline starts running from the stage where it stopped due to the failure.

 

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*

BMC Helix Service Management Deployment 26.1.01