Configuring the on-premises gateway for high availability

Before you begin

1. If you plan to use the BMC Helix object storage, access the UI for the on-premises gateway instances that are part of the high-availability deployment and make sure that the same destination is added for all the instances.
2. If you plan to use a private object storage, make sure that a standalone MinIO or Amazon Simple Storage Service (S3) instance is set up and you have obtained the credentials to access the MinIO or Amazon S3 API endpoint that is set up on the instance. Contact your system administrator to obtain the credentials.​​​​​
   Important

   High-availability deployment of the on-premises gateway has been validated using the Community edition of MinIO. The Enterprise edition is also expected to function correctly.

3. If you plan to use a webhook connector to collect data from a third-party product, you can use a load balancer (for example, F5) to direct the webhook data traffic to the active on-premises gateway instance. The load balancer monitor should use the standbyStatus API to determine the status of an instance.
   Perform the following steps to prepare the environment for data collection by using a webhook connector. These steps are specific to the F5 load balancer and might differ for other load balancer types.
   1. Create a DNS record (for example, VIP_HII) for the virtual IP (VIP).
   2. Configure the load balancer:
      1. Create a pool, and add the on-premises gateway instances as members, as shown in the following example:-ltm pool VIP_HII {
             description "Gateway Pool"
             members {
                 aus-pun-01.abc.com:https {
                     address 192.168.111.xx
                    session monitor-enabled
                    state down
                }
                 aus-pun-02.abc.com:https {
                     address 192.168.112.xx
                    session monitor-enabled
                    state down
                }
                   }
             monitor VIP_HII
         }

         The above example creates a pool for VIP_HII, with two members: aus.pun-01.abc.com (IP address: 192.168.111.xx) and aus.pun-01.abc.com (IP address: 192.168.112.xx) 

      2. Create the monitor to check the status of an on-premises gateway instance, as shown in the following example:
         ltm monitor https VIP_HII {
           adaptive disabled
           defaults-from https
           interval 5
           ip-dscp 0   
           recv false
           recv-disable none
           send "GET /hii/api/mediator/v3/standbyStatus HTTP/1.1\r\nHost: VIP_HII\r\nConnection: Close\r\n\r\n""
           time-until-up 0
           timeout 16
         }
   3. If you plan to use a webhook connector and load balancer, while including the webhook collector URL in the third-party product (for example, Entuity), replace the on-premises gateway host name with the DNS record in the webhook collector URL. For example, an updated webhook collector URL looks like the folllowing:
      https://VIP_HII/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdfd?token=385261281::Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB
      For more information about configuring the URL for Entuity, see Integrating with Entuity via webhook.

To configure on-premises gateway instances for high availability with the BMC Helix object storage

1. Log on to the on-premises gateway instance that you want to add as the primary instance.
2. Navigate to the /IIGateway_INSTALL_DIR/hii directory.
3. Open the docker-compose.yaml file or podman-compose.yaml file with an editor. 
4. Locate the mediator > environment section, and set the following properties:

   •  USE_ADE_STORAGE: Indicates the object storage to be used in the HA deployment. By default, the property is set to false, indicating that private object storage is used. To use the BMC Helix object storage, set the property to true.   

     USE_ADE_STORAGE: "true"

     Important

     When you set the USE_ADE_STORAGE property to true, it takes precedence over the values that you have configured for private object storage properties. Only one type of object storage can be active at a time, as both cannot operate simultaneously.

   • DATA_PUSH_INTERVAL: Indicates the interval (in milliseconds in epoch format) at which an on-premises gateway instance should push data to the BMC Helix object storage (default and minimum 300000 milliseconds).
     The following file shows sample values:

     DATA_PUSH_INTERVAL: "300000" 
   • SOURCE_HOSTNAME: Indicates the name of the server where the on-premises gateway is deployed. Replace $HOSTNAME with the host name. Don't provide a fully-qualified domain name. 
5. Save and close the file.
6. Restart the container services by using the following commands:

   • For Docker deployments:

     docker-compose down
     docker-compose up -d

   • For Podman deployments:

     podman-compose down
     podman-compose -f podman-compose.yaml up -d

     When you restart the service, the instance is added as the primary instance, and a bucket is created on the storage. The bucket contains the configuration backup file, which stores the instance configuration backup with a timestamp. The backup file contains a tag for the on-premises instance.

7. Log on to the subsequent on-premises gateway instances and repeat steps 2 to 6.
   The subsequent instances are added as secondary instances in the storage bucket.

To import the custom or CA-signed certificates into on-premises gateway if you are using MinIO as the object storage

1. Obtain the custom or CA-signed certificates from the MinIO server and save it in the <IIGateway_INSTALL_DIR>hii/conf/certs directory.
2. Navigate to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory.
3. Copy the cacerts file present in the swp-mediator container at /opt/java/lib/security to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory by using the following command:

   • For Docker deployments:

     docker cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts

   • For Podman deployments:

     podman cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts

4. Import the certificates into the copied cacerts file by using the keytool utility:

   keytool -importcert -file minio.crt -keystore cacerts -alias minio
5. Navigate to the /IIGateway_INSTALL_DIR/hii directory.
6. Open the docker-compose.yaml file or podman-compose.yaml file with a text editor. 
7. Search for the mediator section.

8. Add the following line to the volumes section:

   - ./conf/certs/cacerts:/opt/java/lib/security/cacerts
9. Save and close the file.
10. Go to Step 7 to continue configuring the on-premises gateway.

To troubleshoot the high availability issues

You might encounter the following issue if you are using the MinIO instance that is present in the BMC Helix IT Operations Management on-premises environment.

Issue

if you are using the MinIO instance that is present in the BMC Helix IT Operations Management on-premises environment, the on-premises gateway fails to upload the Leader_backup.json file to the MinIO instance, and the following error message appears in the swp-mediator container logs:

Resolution

1. Open the ingress-nginx-controller ConfigMap for the NGINX Ingress controller.
2. Add the following parameter and save the ConfigMap:
   ignore-invalid-headers: "false"
3. Perform a rolling restart of NGINX Ingress controller pods.
4.  Restart the container services on the on-premises gateway host by using the following commands:

   • For Docker deployments:

     docker-compose down
     docker-compose up -d

   • For Podman deployments:

     podman-compose down
     podman-compose -f podman-compose.yaml up -d