Upgrading BMC Helix IT Operations Management from version 22.2.01 to 23.2.02

You might experience downtime during the upgrade, which is to be expected but does not affect the upgrade.

Before you begin

Make sure that you have performed all the steps as listed in the  Preparing-for-upgrade  topic.

To uninstall BMC Helix Logging

Perform the following steps only if you had deployed BMC Helix Logging:

1. Back up logs from Kibana into a Microsoft Excel file.

   See, To export logs to a csv file.

   1. On the Kibana home page, click .
   2. In the Analytics area, click Discover .
   3. Use the filter to refine your search, and then click Refresh query .
      
      
      
   4. Save your search.
   5. Select Share > CSV Reports.
      
      
   6. Click Generate CSV.

   • • The system displays a similar message:
       
       

   1. When prompted, click Download report and save the report.
      
2. Perform the following steps to uninstall BMC Helix Logging (Elasticsearch, Fluentd, and Kibana):
   

   1. Export the Helm and Kubectl binary path:

      export HELM_BIN=/usr/local/bin/helm
      export KUBECTL_BIN=/usr/bin/kubectl
   2. Download the Cleanup_Bmchelixlogging.shscript to clean up the bmc-helix-logging namespace.
      The Cleanup_Bmchelixlogging.sh script deletes the PVC, pods, and jobs from the bmc-helix-logging namespace.

1. 1. Run the command to clean up the bmc-helix-logging namespace:

      ./Cleanup_Bmchelixlogging.sh <bmc-helix-logging namespace>

      <bmc-helix-logging namespace> is the name of the bmc-helix-logging namespace that you used in BMC Helix ITOM 22.2.01 version.
      
      

To delete the helm revision secrets of the Kubernetes CronJob

1. Run the following commands to export the kubectl and Helm binary:
   

   export KUBECTL_BIN=<path of KUBECTL_BIN>
   
   export HELM_BIN=<path of HELM_BIN>

   Copy the path of KUBECTL_BIN and HELM_BIN from the infra.config file. See, Configuration-file-settings.
   

2. Download the Cronjob_helm_revision_delete.shscript.
3. Copy the Cronjob_helm_revision_delete.sh script to helix-on-prem-deployment-manager  folder.

4. Run the Cronjob_helm_revision_delete.sh script to delete the helm revision secrets of the CronJob :

   ./Cronjob_helm_revision_delete.sh <namespace>

   <namespace> is the namespace where you had deployed BMC Helix IT Operations Management .

To migrate data from Bitnami PostgreSQL to Patroni PostgreSQL

BMC Helix ITOM version 22.2.01 and earlier supports Bitnami PostgreSQL. Starting with version 23.1.02, Bitnami PostgreSQL is replaced by Patroni PostgreSQL. Because of this change, you must migrate your data from Bitnami PostgreSQL to Patroni PostgreSQL.

1. Make sure that you have the exec permission on the pods. Run the following command to establish a shell session inside the pod and verify the exec permissions:

   kubectl  -n <namespace> exec -it <pod-name> -- bash

   You should be able to establish a shell session inside the pod.

   Important

   If you are not able to establish a shell session inside the pod, it indicates that you do not have exec permission. Contact your cluster administrator, to get the permission.  

   
   

2. In the new_working_directory/ configs/infra.config file, make sure that you set the value of the parameter PG_HOSTNAME as postgres-bmc-pg-ha-pool.
   
   
3. In the new_working_directory/ configs/deployment.config file, make sure that your deployment size is the same as your previous deployment size; for example, small, compact, medium, or large.
   To confirm your deployment size, see the value set for the DEPLOYMENT_SIZE parameter in the configs/ deployment.config file in BMC Helix ITOM version 22.2.01.
   

4. In the new_working_directory/ configs/deployment.config file, set the values of the following properties:

   Property	Expected value
   INSTALL_MODE	upgrade
   _PTPOSTGRESS	yes
   INFRA	yes

5. In the new_working_directory/ configs/deployment.config file, set the value of the following properties to no:
   _KAFKA
   _REDIS
   _RSSO
   _VICTORIAMETRICS
   _ELASTICSEARCH
   _MINIO
   HELIX_DASHBOARD_SERVICES
   ITSMINSIGHT_SERVICES
   AIOPS_SERVICES
   MONITOR
   INTELLI_INT_SERVICES
   INTELLIGENT_AUTOMATION
   BMC_HELIX_LOGGING
   OPTIMIZE
   ARSERVICES
   
   

6. In the new_working_directory/ commons/certs/secrets.txt file, set the passwords. 
   For more information, see Preparing-for-password-encryption.

   Important

   Use only the new secrets.txt file that is present in your new working directory (new_working_directory) to set the passwords.

   Do not copy the older version of the secrets.txt file because it is not compatible with the newer version. However, we recommend that you refer to the older version of the secrets.txt file to understand the syntax and to get the values that you had set previously.

7. Deploy the Patroni PostgreSQL server by running the following command:

   ./deployment-manager.sh

   If the Patroni PostgreSQL server deployment is not successful, contact BMC Support along with the migration log files. The migration log files are located at utilities/migration/logs.
   
   

8. Go to the new_working_directory/ utilities/migration directory by using the following command:

   cd helix-on-prem-deployment-manager/utilities/migration
9. Open the  dbmigration.sh  script present in the new_working_directory/ utilities/migration  directory and, add the following passwords as shown in the following image:
   • Bitnami PostgreSQL password that you specified for the parameter PG_PASSWD in the secrets.txt file in BMC Helix ITOM version 22.2.01.

   • Patroni PostgreSQL password that you specified for the parameter PG_PASSWD in the secrets.txt file in BMC Helix ITOM version 22.4.

     Important

     The passwords can be same or different depending on what you had set.

     With reference to the following example code, do not include the angular brackets while adding the Bitnami PostgreSQL and the Patroni PostgreSQL passwords.

     

10. Save the script and run the following command to start the migration:

    ./dbmigration.sh

    After the migration is complete, you will get the following message:
    Migration done Successfully!!
     The migration log files are located at new_working_directory/ utilities/migration/logs.

 To upgrade BMC Helix IT Operations Management

1. In the new_working_directory/ configs/deployment.config file, make sure that your deployment size is the same as your previous deployment size; for example, small, compact, medium, large or extra large.
   To confirm your deployment size, see the value set for the DEPLOYMENT_SIZE parameter in the new_working_directory/ configs/ deployment.config file in BMC Helix ITOM version 22.2.01.
   
   
2. In the new_working_directory/ configs/deployment.config file, update the following values:
   1. Set the value of INSTALL_MODE to upgrade .
   2. Set the values of the following infrastructure services to yes.
      • _INFRA
      • _PTPOSTGRES
      • _KAFKA
      • _REDIS
      • _RSSO
      • _VICTORIAMETRICS
      • _ELASTICSEARCH
      • _MINIO
   3. For all the BMC Helix ITOM applications for which you are licensed, set the value to yes; for all other services, set the value to no.
      For example, if you are only licensed to use BMC Helix Operations Management and BMC Helix Continuous Optimization:
      • To use BMC Helix Operations Management, set value of MONITOR to yes.
        Optionally,
        • If you want to use AIOps, set AIOPS_SERVICES to yes.
        • If you want to use Log Analytics, set INTELLI_INT_SERVICES to yes.

      • To use BMC Helix Continuous Optimization, set the value of OPTIMIZE to yes. 

        Best practice
        To see the product services for which you are licensed, see the configs/deployment.config file in BMC Helix ITOM version 22.2.01.

3. From the new working directory (new_working_directory) run the deployment manager to upgrade BMC Helix IT Operations Management:

   ./deployment-manager.sh

   After the upgrade is complete, you will get the following message:
   Completed Helix On-prem Installation.

4. If you had deployed BMC Helix IT Operations Management in a multitenant environment, follow the steps to restore the smart-graph-tenant-config  configmap and its content:
   1. Copy the contents from the smart-graph-tenant-config configmap file that you backed up before upgrade.

   2. Run the command to open the smart-graph-tenant-config configmap:

      kubectl -n <namespace> edit cm smart-graph-tenant-config

   3. Paste the contents that you copied from the backed up smart-graph-tenant-config configmap file in the new smart-graph-tenant-config configmap and save it.

   4. Run the following command to restart the smart graph controller pod: 

      kubectl rollout restart deploy/smart-graph-controller -n <namepsace>

5. ( Optional ) To view the logs during the upgrade, run the following command: 

   tail -f logs/deployment.log

Performing post upgrade task

1. After the upgrade is complete and the latest version of the product is deployed, since the Bitnami pods are not needed, scale down the Bitnami PostgreSQL pods to zero by running the following command: 

   kubectl scale deployment -n <NAMESPACE> postgres-postgresql-ha-pgpool --replicas=0
   kubectl scale sts -n <NAMESPACE> postgres-postgresql-ha-postgresql --replicas=0
   

2. Once the data migration is complete and verified successfully, remove the Bitnami PostgreSQL pods by running the following command:

   helm delete -n <namespace> postgres
   kubectl delete pvc -n <namespace> data-postgres-postgresql-ha-postgresql-0 data-postgres-postgresql-ha-postgresql-1