Upgrading BMC Helix Service Management to 22.1.06

Upgrade process

The following image describes the process to upgrade your BMC Helix Innovation Suite platform and applications from version 21.3.05 or 21.3.06 to 22.1.06:

Before you begin

• If you are using an Oracle database, make sure that you are using Oracle database 19.16 version.
• Make sure that your BMC Helix Innovation Suite database name does not contain the dot "." character.
  If the database name contains a dot character, perform the following steps:

  1. Scale down all the pods in the BMC Helix Innovation Suite namespace.

     Important

     Scaling down all pods can lead to downtime. We recommend that you perform this activity during scheduled downtime or the database maintenance window to reduce the impact on the users.

  2. Update the database name.

     Important

     Make sure that you the database name starts with a character. You can specify numbers and only the following special characters in the name:

     • Dollar $
     • Hash #
     • Underscore _

     If you do not change the database name, the BMC Helix Innovation Suite upgrade might fail.

  3. Update the Kubernetes statefulset for the platform pods to use the new database name.
  4. Update the Kubernetes deployments to use the new database name.
  5. Scale up all the pods in the BMC Helix Innovation Suite namespace.
  6. License the server by using the new database name.
     See Obtaining-license-keys and Activating-application-licenses.
     
     
• Verify that you have specified the Tenant parameter value while configuring the BMC Helix Single Sign-On realm. 

• Confirm that the platform-admin-ext service has the EXTERNAL-IP value configured by using the following command:

  kubectl get svc -n <BMC Helix Innovation Suite namespace>

  If the command output does not display any IP address for the platform-admin-ext service, the EXTERNAL-IP is not configured. Configure the EXTERNAL-IP by running the following command in the Jenkins server:

  kubectl patch service platform-admin-ext -n <namespace> -p '{"spec":{"externalIPs":["'<IP address of your Kubernetes cluster worker node>'"]}}'

  For example:

  kubectl patch service platform-admin-ext -n <namespace> -p '{"spec":{"externalIPs":["'172.10.101.123'"]}}'

• Install the gtk3 package so that the autorecon utility runs successfully.
  Use the following command:

  yum install -y gtk3

• If required, restrict the upgrade process from changing the Centralized configuration system (CCS) parameters specific to your environment.
  See Changes to configuration settings during an upgrade in AR System documentation.

Back to top

To prepare for upgrade

Perform the following tasks to prepare for an upgrade of BMC Helix Service Management.

Task 1: To download the upgrade artifacts

1. Download the upgrade artifacts; see Downloading-the-installation-files.
2. Confirm access to the container images; see Setting-up-a-Harbor-repository-to-synchronize-container-images.

Back to top

Task 2: To upgrade Helm

Upgrade Helm to the following version based on your orchestration platform version:

Back to top

Task 3: To update the Git repositories

1. Log in to the BMC Deployment Engine as a Git user.
   
2. In your local Git repository, create a backup of the following existing repositories.
   • CUSTOMER_CONFIGS
   • HELM_REPO
   • ITSM_REPO
   • PLAYBOOKS_REPO
   • SMARTAPPS_HELM_REPO
   • SMARTREPORTING_PLAYBOOKS_REPO

   • SMARTREPORTING_HELM_REPO
     For example, create a backup of the ITSM_REPO, PLAYBOOKS_REPO, HELM_REPO, SMARTAPPS_HELM_REPO, CUSTOMER_CONFIGS, SMARTREPORTING_PLAYBOOKS_REPO, and SMARTREPORTING_HELM_REPO repositories located at ssh://<Jenkins server host name>/home/git/Git_Repo.

     Important

     • After creating a backup of the repositories, make sure that you delete the existing repository folders.
     • Make sure that you do not modify or change the location of the LIBRARY_REPO repository.
3. Copy the BMC_Remedy_Deployment_Manager_Configuration_Release_22.1.06.zip file that you downloaded from EPD to your existing local Git repository.
   For example, copy the BMC_Remedy_Deployment_Manager_Configuration_Release_22.1.06.zip file to the local Git repository located at ssh://<Jenkins server host name>/home/git/Git_Repo.

1. In the file system where the Git repository is located, unzip the BMC_Remedy_Deployment_Manager_Configuration_Release_22.1.06.zip file.

   For example, in the Git_Repo repository located at ssh://<Jenkins server host name>/home/git/Git_Repo, unzip the BMC_Remedy_Deployment_Manager_Configuration_Release_22.1.06.zip file.The BMC_Remedy_Deployment_Manager_Configuration_Release_22.1.06.zip file contains the following artifacts:

   • CUSTOMER_CONFIGS
   • HELM_REPO
   • ITSM_REPO
   • PLAYBOOKS_REPO
   • SMARTREPORTING_PLAYBOOKS_REPO
   • SMARTREPORTING_HELM_REPO
   • SMARTAPPS_HELM_REPO
2. (If you are using BMC Helix Platform Common Services 23.2.02) Update the sizing templates for the Compact, Small, Medium, and Large deployment sizes.
   1. Copy the ITSM_REPO.zip file that you downloaded from EPD.
      The BMC_Helix_Innovation_Suite_Performance_Hotfix_Version_22106.zip file located in the BMC Helix Innovation Suite Performance Hotfix Version 22.1.06 folder on EPD contains the ITSM_REPO.zip file.

1. 1. Rename the existing ITSM_REPO by using the following command:

      mv ITSM_REPO ITSM_REPO_old

   2. Extract the ITSM_REPO.zip file by using the following command:

      unzip ITSM_REPO.zip

      After you unzip the file, the Git repository is replaced with the deployment size templates.

   3. Apply owner user permissions to the ITSM_REPO repository by using the following command:

      chown -R git:git ITSM_REPO

      Important

      The platform-sr, platform-int, and platform-user pods are not deployed with the Compact and Small deployment sizes. The HELIX_PLATFORM_DEPLOY pipeline only deploys the platform-fts pod with one replica for Compact deployment and two replicas for Small deployment.

2. (If you are using BMC Helix Platform Common Services 23.4.00 or 24.1.00) Update the sizing templates and disaster recovery enhancements.

   Important

   Skip this step if you are upgrading BMC Helix Platform Common Services version from 23.4.00 to 24.1.00.

   
   

   1. Log in to the Jenkins server with the Git repository owner user, such as git.
   2. Navigate to the Git repositories directory, such as /home/git/Git_Repo
   3. Copy the ITSM_REPO.zip file that you downloaded from EPD.
      The BMC_Helix_Innovation_Suite_Disaster_Recovery_Hotfix_Version_22106.zip file located in the BMC Helix Innovation Suite Disaster Recovery Hotfix Version 22.1.06 folder on EPD contains the ITSM_REPO.zip file.

   1. Rename the existing ITSM_REPO by using the following command:

      mv ITSM_REPO ITSM_REPO_old

   2. Extract the ITSM_REPO.zip file by using the following command:

      unzip ITSM_REPO.zip

      After you unzip the file, the Git repository is replaced with the new deployment size templates.
      

   3. Apply owner user permissions to the ITSM_REPO repository by using the following command:

      chown -R git:git ITSM_REPO

Back to top

Task 4: To add deployment pipelines

1. Verify that the following deployment pipelines already exist in your Jenkins server:

   Pipeline	Repository URL	Branches to build	Script path
   HELIX_SMARTREPORTING_UPGRADE	ssh://git@<jenkins_server>/<GIT_REPO_DIR>/ITSM_REPO/itsm-on-premise-installer.git	*/master	pipeline/jenkinsfile/HELIX_SMARTREPORTING_UPGRADE.jenkinsfile
   HELIX_ONPREM_DEPLOYMENT	ssh://git@<jenkins_server>/GIT_REPO_DIR/ITSM_REPO/itsm-on-premise-installer.git	*/master	pipeline/jenkinsfile/HELIX_ONPREM_DEPLOYMENT.jenkinsfile
   HELIX_RESTART	ssh://git@<jenkins_server>/GIT_REPO_DIR/ITSM_REPO/itsm-on-premise-installer.git	*/master	pipeline/jenkinsfile/HELIX_RESTART.jenkinsfile
   HELIX_FULL_STACK_UPGRADE	ssh://git@<jenkins_server>/GIT_REPO_DIR/ITSM_REPO/itsm-on-premise-installer.git	*/master	pipeline/jenkinsfile/HELIX_FULL_STACK_UPGRADE.jenkinsfile

2. (Deployment pipelines exist) Run the HELIX_FULL_STACK_UPGRADE and HELIX_SMARTREPORTING_UPGRADE pipelines in dry-run mode so that the pipeline parameters that are updated in 22.1.06 version are reflected.
   
   1. On the Jenkins server, select the pipeline.
      For example, on the Jenkins server, select the HELIX_FULL_STACK_UPGRADE pipeline.
   2. In the navigation pane, select the latest build and select Build with Parameters > Build.
   3. Click Rebuild.
      After the pipeline is rebuilt, the updated parameters for the pipeline are reflected.
      To verify that the pipeline is updated with the new parameters, click Build with Parameters and check that the ONPREM check box is present.
3. Perform the following steps to add the pipeline:

1. 
   
   1. Log in to the Jenkins server by using the following URL:
      http://<Jenkins server host name>:8080
   2. On the Jenkins home page, click New Item.
   3. In the Enter an item name field, enter the pipeline name.
      Example: HELIX_FULL_STACK_UPGRADE.
   4. Select Pipeline and click OK.
   5. Click the Pipeline tab.

   6. Enter the following information:

      Field	Description
      Definition	From the Definition list, select Pipeline script from SCM.
      SCM	From the SCM list, select Git.
      Repository URL	Enter the Repository URL as the path of your local Git repository in the format ssh://git@<jenkins_server>/<path to itsm-on-premise-installer.git>. Example: ssh://git@<Jenkins server host name>/home/git/Git_Repo/ITSM_REPO/itsm-on-premise-installer.git.
      Credentials	Enter the Git server credentials.
      Additional Behaviours	Specify the script path. Example: pipeline/jenkinsfile/HELIX_FULL_STACK_UPGRADE.jenkinsfile.

      The following image shows an example:
      

   7. Click Apply and then Save.
   8. From the Jenkins home page, select the pipeline and click Build Now.
      The first build job fails because it needs to run the first time to load all the parameters of the pipeline script. 
   9. After the build job fails, select the pipeline name again from the Jenkins home page.
      The Build Now option changes to Build With Parameters.

Back to top

To perform the upgrade

Perform the following tasks to perform the upgrade of BMC Helix Service Management.

Task 1: To upgrade the Kubernetes platform

If you are using Kubernetes platform earlier than version 1.23, upgrade your Kubernetes platform to version 1.23, 1.24, 1.25, 1.26, 1.27, 1.28, or 1.29.

Back to top

Task 2: To upgrade the Ingress controller

Upgrade the Ingress version based on the Kubernetes platform version that you want to use. Supported Ingress controller versions are as follows:

Back to top

Task 3: To delete the Helm revision secrets of the Kubernetes CronJob

1. Run the following commands to export the kubectl and Helm binary:

   export KUBECTL_BIN=<path of KUBECTL_BIN>
   
   export HELM_BIN=<path of HELM_BIN>

   Copy the path of KUBECTL_BIN and HELM_BIN from the infra.config file. 

2. Download the Cronjob_helm_revision_delete.shscript.

3. Run the Cronjob_helm_revision_delete.sh script to delete the helm revision secrets of the CronJob in the BMC Helix Platform namespace.
   

   ./Cronjob_helm_revision_delete.sh <BMC Helix Platform namespace>

4. Run the Cronjob_helm_revision_delete.sh script to delete the helm revision secrets of the CronJob in the BMC Helix Service Management namespace.
   

   ./Cronjob_helm_revision_delete.sh <BMC Helix IT Service Management namespace>

Task 4: To upgrade BMC Helix Platform Common Services

Perform the steps to upgrade BMC Helix Platform Common Services based on your current BMC Helix Platform Common Services version:

Back to top

Task 5: To upgrade BMC Helix Innovation Suite platform and applications

To upgrade the platform and applications, update the deployment pipelines, specify the required parameters, and run the pipelines.

To update the deployment pipelines

1. Perform the following steps to update the HELIX_ONPREM_DEPLOYMENT pipeline.
   1. On the Jenkins server, select the HELIX_ONPREM_DEPLOYMENT pipeline.
   2. In the Build History, select the latest build and click Rebuild.
   3. In the INFRA-DEPLOY section, clear the SUPPORT_ASSISTANT_TOOL check box.
   4. In the PRODUCT-DEPLOY section, clear the check boxes for all the options.
   5. Click Rebuild.
2. Update the HELIX_GENERATE_CONFIG pipeline so that the updated parameters for the pipeline are reflected. 
   1. On the Jenkins server, select the HELIX_GENERATE_CONFIG pipeline.
   2. In Build History, select the latest build and click Rebuild.
   3. Click Rebuild.
      After the pipeline is rebuilt, the updated parameters for the pipeline are reflected, such as the GIT_REPO_DIR parameter.

To run the HELIX_ONPREM_DEPLOYMENT pipeline

1. Select the HELIX_ONPREM_DEPLOYMENT pipeline, select the latest build, and then click Rebuild.

2. In the CODE section, specify the following parameters:

   Parameter	Description
   GIT_USER_HOME_DIR	Git user home directory. Example: /home/git
   GIT_REPO_DIR	Directory that contains all the Git repositories for the deployment pipelines. Example: ssh://<Jenkins server host name>/home/git/git_repo

3. In the CUSTOMER-INFO section, specify the following parameters:

   Parameter	Description
   OS_RESTRICTED_SCC	Select this check box if you are using an Openshift cluster and have restricted security context constraints enabled.
   IS_NAMESPACE	Specify the namespace where you want to install BMC Helix Innovation Suite.
   CACERTS_FILE	Click Browse and upload your custom cacerts file. Important: If you have used a custom cacerts file in a BMC Helix Service Management version earlier than 21.3.06, upload the custom cacerts file to this parameter. Use this optional parameter in the following cases: You want to use a custom CA certificate or self-signed certificate to use HTTPS communication for Service Management applications. You want BMC Helix Innovation Suite platform to communicate with third-party services that use custom cacerts. Important: Make sure that you use the same custom certificate in BMC Helix Innovation Suite and BMC Helix Platform.  For information about creating a custom cacerts file and ways to apply a custom cacerts file, see Applying-security-certificates-to-your-applications.
   PLATFORM_COMMON_CACERTS_SSL_TRUSTSTORE_PASSWORD	Specify a password for your cacerts file. Leave this field blank if you want to use the default password for the cacerts file.
   CUSTOMER_SIZE	Specify the deployment size, such as compact, small, medium, or large. Enter the value as C, S, M, or L. Important: Make sure that you select the appropriate deployment size. The deployment size is C - compact by default. During upgarde, you can continue to use compact size by specifying the value as C or another size according to your requirement.
   SOURCE_VERSION	Specify the source version of the Helm repositories: If your BMC Helix Innovation Suite platform source version is 21.3.02, select 2021302.1.00.00 If your BMC Helix Innovation Suite platform source version is 21.3.04, select 2021304.1.00.00 If your BMC Helix Innovation Suite platform source version is 21.3.05, select 2021305.1.05.00 If your BMC Helix Innovation Suite platform source version is 21.3.06, select 2021306.1.07.00
   PLATFORM_HELM_VERSION	Specify the target version of the Helm repositories as 2022106.1.03.00
   SMARTREPORTING_SOURCE_VERSION	Specify the SMARTREPORTING_SOURCE_VERSION value as 210503.1.05.00 or 210503.1.09.00 Important: Specify the parameter value if you want to upgrade Smart Reporting.
   SMARTAPPS_HELM_VERSION	Specify the version of the Helm repositories for applications as 2022106.1.03.00
   DEPLOYMENT_MODE	Specify the value as UPGRADE.

4. In the PRODUCTS section, specify the following parameters:

   Parameter	Description
   HELIX_VIRTUALCHAT	Select this check box to install BMC Live Chat.
   HELIX_OPENFIRE	Select this check box to install Openfire.
   HELIX_DWP	Select this check box to install BMC Digital Workplace.
   HELIX_DWPA	Select this check box to install BMC Digital Workplace Catalog.
   HELIX_BWF	Select this check box to install BMC Helix Business Workflows.
   HELIX_MCSM	Select this check box to install BMC Helix Multi-Cloud Broker.
   HELIX_ITSM_INSIGHTS	Select this check box to install BMC Helix ITSM Insights.
   HELIX_TSOMPLUGIN	Select this check box to install TrueSight Operations Management plug-ins.
   HELIX_SMARTAPPS_CSM	Select this check box to install BMC Helix Customer Service Management (CSM). Important: Do not select this check box if you are using a Microsoft SQL Server database. CSM installation fails on a Microsoft SQL Server database.
   HELIX_SMARTAPPS_FAS	Select this check box to install BMC Helix Portfolio Management.
   HELIX_DRIFT_MANAGEMENTPLUGIN	Select this check box to install Drift Management.

5. In the PRODUCT-DEPLOY section, specify the following parameters:

   Parameter	Description
   HELIX_GENERATE_CONFIG	Select this check box.
   HELIX_SMARTREPORTING_DEPLOY	Select this check box to upgrade BMC Helix ITSM: Smart Reporting.
   HELIX_INTEROPS_DEPLOY	Select this check box to activate services such as  BMC Helix ITSM Insights, BMC Helix ITSM: Smart IT, BMC Helix Digital Workplace, BMC Helix Digital Workplace Catalog, and BMC Helix Business Workflows for the BMC Helix Platform users. Important: Select this check box if you have not activated the above services in an earlier version of BMC Helix Service Management.
   HELIX_FULL_STACK_UPGRADE	Select this check box to upgrade the platform and application components.
   HELIX_DR	Do not select this check box.
   SCALE_DOWN	Do not select this check box.
   HELIX_RESTART	Do not select this check box.

   Important

   Make sure that you do not select the HELIX_CONFIGURE_ITSM, HELIX_POST_DEPLOY_CONFIG, HELIX_DR, SCALE_DOWN, and HELIX_RESTART check boxes.

6. In the DATABASE_REGISTRY_STORAGE_PASSWORDS_SECRETS_DETAILS section, specify the following parameters:

   Parameter	Description
   AR_ADMIN_USER	Specify the AR System administrator user name.
   AR_ADMIN_PASSWORD	Specify the password for AR System administrator user. To update the password, click Change Password.
   PLATFORM_ADMIN_PLATFORM_EXTERNAL_IPS	Specify the IP address that you specified as the external IP for the platform-admin-ext service. The external IP must be in JSON list format within square brackets, such as [10.x.x.x].
   ENABLE_PLATFORM_INT_NORMALIZATION	Select this check box to start the Normalization service on the platform-int pod.
   DB_SSL_ENABLED	Do not select this check box.
   DB_JDBC_URL	Specify a JDBC URL to use a database connection through the URL. Example: jdbc:postgresql://server-name:server-port/database-name?sslmode=verify-full This parameter is optional.
   LOGS_ELASTICSEARCH_TLS	Select this check box.
   LOGS_ELASTICSEARCH_PASSWORD	Specify the password that you provided for Kibana in the secrets.txt file while you were installing BMC Helix Platform service.
   MIDTIERCACHEBUILDER_TRIGGER_PRELOAD	Select this check box to enable full data cache mode.
   MIDTIERCACHEBUILDER_SCHEDULE	Specify a cron job schedule for the Mid Tier cache builder job.
   BWF_DEPLOY_SAMPLE_CONTENT_PACK	Select this check box to deploy BMC Helix Business Workflows sample content package.

7. In the RSSO_PARAMETERS section, specify the following parameter:

   Parameter	Description
   TENANT_DOMAIN	Specify the value of the Tenant parameter that you specified while configuring the BMC Helix Single Sign-On realm.

8. In the ITSM_INTEROPS_PARAMETERS section, specify the following parameters:

   Parameter	Description
   HELIX_PLATFORM_DOMAIN	Specify the domain name of the cluster where BMC Helix Platform is installed.
   HELIX_PLATFORM_NAMESPACE	Specify the namespace where BMC Helix Platform is installed.
   HELIX_PLATFORM_CUSTOMER_NAME	Specify the registered tenant in BMC Helix Platform.

9. Click Rebuild.
   The HELIX_ONPREM_DEPLOYMENT pipeline upgrades the platform and applications.
   For information about HELIX_FULL_STACK_UPGRADE pipeline failure during the upgrade, see rollback deployment.

10. After successful completion of the HELIX_ONPREM_DEPLOYMENT pipeline, verify that the platform and application component pods are running by using the following command:

    kubectl get pod -n <namespace>

    The following image shows an example:
    

Back to top

Rollback deployment

During BMC Helix Service Management upgrade, if the HELIX_FULL_STACK_UPGRADE pipeline fails, you can rollback back to the last successful deployed version of the platform and applications by using the rollback option.

When the HELIX_FULL_STACK_UPGRADE pipeline fails, the rollback option is displayed on the pipeline output console and on the pipeline stage view as shown in the following images:

Back to top

Where to go from here

Performing-the-postinstallation-configurations