Configuring the on-premises gateway for high availability

Before you begin

• To use the BMC Helix object storage:
  • Access the UI for the on-premises gateway instances that are part of the high-availability deployment and make sure that the same destination is added for all instances.
  • Make sure that the access key and secret key used in the destination have all [*] permissions.
• To use a private object storage, make sure that a standalone S3-compatible storage instance is set up, and you have obtained the credentials that have permissions to create and read objects. Contact your system administrator to obtain the credentials.

Task 1: To prepare the environment for data collection by using a webhook connector

If you are using a webhook connector to collect data from a third-party product in a high-availability deployment, you can use a load balancer (for example, F5) to direct the webhook data traffic to the active on-premises gateway instance. 

Perform the following steps to prepare the environment for data collection by using a webhook connector:

1. Configure the load balancer.
   1. Create a DNS record (for example, VIP_HII) for the virtual IP (VIP).
   2. Create a pool, and add the on-premises gateway instances as members, as shown in the following example:-ltm pool VIP_HII {
          description "Gateway Pool"
          members {
              aus-pun-01.abc.com:https {
                  address 192.168.111.xx
                 session monitor-enabled
                 state down
             }
              aus-pun-02.abc.com:https {
                  address 192.168.112.xx
                 session monitor-enabled
                 state down
             }
                }
          monitor VIP_HII
      }

      The above example creates a pool for VIP_HII, with two members: aus.pun-01.abc.com (IP address: 192.168.111.xx) and aus.pun-01.abc.com (IP address: 192.168.112.xx) 

      1. Create the monitor to check the status of an on-premises gateway instance, as shown in the following example:
         ltm monitor https VIP_HII {
           adaptive disabled
           defaults-from https
           interval 5
           ip-dscp 0   
           recv false
           recv-disable none
           send "GET /hii/api/mediator/v3/standbyStatus HTTP/1.1\r\nHost: VIP_HII\r\nConnection: Close\r\n\r\n""
           time-until-up 0
           timeout 16
         }

         Warning

         Important

         The monitor you are creating must use the standbyStatus API to determine the status of an instance, as shown in the above example.

2. While including the webhook collector URL in the third-party product (for example, Entuity), replace the on-premises gateway host name with the DNS record in the webhook collector URL. For example, an updated webhook collector URL looks like:
   https://VIP_HII/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdfd?token=385261281::Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB
   For more information about configuring the URL for Entuity, see Integrating with Entuity via webhook.

Task 2 (BMC Helix storage): To configure on-premises gateway instances for HA if you are using the storage included with the BMC Helix ITOM deployment

1. Log on to an on-premises gateway instance.
2. Navigate to the /IIGateway_INSTALL_DIR/hii directory.
3. Open the docker-compose.yaml file or podman-compose.yaml file by using a text editor. 
4. Locate the mediator > environment section.
5. Depending on the value of the SET_INTELLIGENT_INTEGRATIONS property that was set during the BMC Helix ITOM deployment, perform the following tasks:
   • If SET_INTELLIGENT_INTEGRATIONS is set to true, perform the following steps:

     • Set the USE_ADE_STORAGE property to true.
       USE_ADE_STORAGE: "true"

     • Set the DATA_PUSH_INTERVAL property to the interval (in milliseconds) in epoch format at which an on-premises gateway instance should push data to the storage bucket (default and minimum 300000 milliseconds).
       DATA_PUSH_INTERVAL: "300000"

   • If SET_INTELLIGENT_INTEGRATIONS is set to false, perform the following steps:
     • Make sure that the USE_ADE_STORAGE property is set to false.
       USE_ADE_STORAGE: "false"

     • Set the MINIO_ACCESS_KEY property to the access key for S3-compatible storage.

     • Set the MINIO_SECRET_KEY property to the secret key for the S3-compatible storage.
     • Set the MINIO_SERVER_URL property to the endpoint URL for the S3-compatible storage.
       # MinIO
       https://vx-push-dev26.abc.com:9000
       # Amazon S3
       https://s3.us-west-1.amazonaws.com
     • Set the MINIO_BUCKET_NAME to the name of the bucket you want to create on S3-compatible storage instance.
       Important:  If you are using the same S3-compatible storage server for multiple HA environments (for example, Test and Production), make sure that each bucket has a unique name. 

     • Set the DATA_PUSH_INTERVAL property to the interval (in milliseconds) in epoch format at which an on-premises gateway instance should push data to the storage bucket (default and minimum 300000 milliseconds).
       DATA_PUSH_INTERVAL: "300000"

       The following example shows sample values:

       MINIO_ACCESS_KEY: "P3pWEoNUEmZB8i0zJAnC"
       MINIO_SECRET_KEY: "SgA3ntRrdM3nzUGpvKjRMQ2FJZNHujfngxJgTb"
       MINIO_SERVER_URL: https://vx-push-dev26.abc.com:9000
       MINIO_BUCKET_NAME: "helix-hii-backup"
       DATA_PUSH_INTERVAL: "300000" 
6. Save and close the file.
7. If your object storage is using custom or CA-signed certificates, import them into the on-premises gateway instance.
8. Restart the container services by using the following commands:

   • For Docker deployments:

     docker-compose down
     docker-compose up -d

   • For Podman deployments:

     podman-compose down
     podman-compose -f podman-compose.yaml up -d

     When you restart the services for the first on-premises gateway instance, it is added as the primary instance.

9. Log on to the subsequent on-premises gateway instances and repeat steps 2 to 8.
   The subsequent instances are added as secondary instances in the storage.

Task 3: To import the custom or CA-signed certificates into the on-premises gateway

If you are using MinIO as the object storage, import the custom or CA-signed certificates into the on-premises gateway instances. 

1. Obtain the custom or CA-signed certificates from the MinIO server and save it in the /<IIGateway_INSTALL_DIR>hii/conf/certs directory.
2. Navigate to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory.
3. Copy the cacerts file present in the swp-mediator container at /opt/java/lib/security to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory by using the following command:

   • For Docker deployments:

     docker cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts

   • For Podman deployments:

     podman cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts

4. Import the certificates into the copied cacerts file by using the keytool utility:

   keytool -importcert -file minio.crt -keystore cacerts -alias minio
5. Navigate to the /IIGateway_INSTALL_DIR/hii directory.
6. Open the docker-compose.yaml file or podman-compose.yaml file by using a text editor. 
7. Search for the mediator section.

8. Add the following line to the volumes section:

   - ./conf/certs/cacerts:/opt/java/lib/security/cacerts
9. Save and close the file.
10. Go to Step 7 to continue configuring the on-premises gateway.

To troubleshoot HA issues

You might encounter the following issue if you are using the MinIO instance that is present in the BMC Helix IT Operations Management on-premises environment.

Issue

The on-premises gateway fails to upload the Leader_backup.json file to the MinIO instance, and the following error message appears in the swp-mediator container logs:

Resolution

1. Open the ingress-nginx-controller ConfigMap for the NGINX Ingress controller.
2. Add the following parameter and save the ConfigMap:
   ignore-invalid-headers: "false"
3. Perform a rolling restart of NGINX Ingress controller pods.
4.  Restart the container services on the on-premises gateway host by using the following commands:

   • For Docker deployments:

     docker-compose down
     docker-compose up -d

   • For Podman deployments:

     podman-compose down
     podman-compose -f podman-compose.yaml up -d