Configuring the on-premises gateway for high availability

Before you begin

• To use the BMC Helix object storage, access the UI for the on-premises gateway instances that are part of the high-availability deployment and make sure that the same destination is added for all the instances.
• To use a private object storage, make sure that a standalone MinIO or Amazon Simple Storage Service (S3) instance is set up and you have obtained the credentials to access the MinIO or Amazon S3 API endpoint that is set up on the instance. Contact your system administrator to obtain the credentials.​​​​​
  Important

  High-availability deployment of the on-premises gateway has been validated with the Community Edition of MinIO. The Enterprise Edition is also expected to function correctly.

• To use a webhook connector to collect data from a third-party product, you can use a load balancer (for example, F5) to direct the webhook data traffic to the active on-premises gateway instance. The load balancer monitor should use the standbyStatus API to determine the status of an instance. 
  Perform the following steps to prepare the environment for data collection by using a webhook connector:
  1. Configure the load balancer.
     These steps are specific to the F5 load balancer and might differ for other load balancer types.
     1. Create a DNS record (for example, VIP_HII) for the virtual IP (VIP).
     2. Create a pool, and add the on-premises gateway instances as members, as shown in the following example:-ltm pool VIP_HII {
            description "Gateway Pool"
            members {
                aus-pun-01.abc.com:https {
                    address 192.168.111.xx
                   session monitor-enabled
                   state down
               }
                aus-pun-02.abc.com:https {
                    address 192.168.112.xx
                   session monitor-enabled
                   state down
               }
                  }
            monitor VIP_HII
        }

        The above example creates a pool for VIP_HII, with two members: aus.pun-01.abc.com (IP address: 192.168.111.xx) and aus.pun-01.abc.com (IP address: 192.168.112.xx) 

     3. Create the monitor to check the status of an on-premises gateway instance, as shown in the following example:
        ltm monitor https VIP_HII {
          adaptive disabled
          defaults-from https
          interval 5
          ip-dscp 0   
          recv false
          recv-disable none
          send "GET /hii/api/mediator/v3/standbyStatus HTTP/1.1\r\nHost: VIP_HII\r\nConnection: Close\r\n\r\n""
          time-until-up 0
          timeout 16
        }
  2. While including the webhook collector URL in the third-party product (for example, Entuity), replace the on-premises gateway host name with the DNS record in the webhook collector URL. For example, an updated webhook collector URL looks like:
     https://VIP_HII/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdfd?token=385261281::Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB
     For more information about configuring the URL for Entuity, see Integrating with Entuity via webhook.

To configure on-premises gateway instances for HA with BMC Helix object storage

1. Log on to the on-premises gateway instance that you want to add as the primary instance.
2. Navigate to the /IIGateway_INSTALL_DIR/hii directory.
3. Open the docker-compose.yaml file or podman-compose.yaml file by using a text editor. 
4. Locate the mediator > environment section, and set the following properties:

   •  USE_ADE_STORAGE: Indicates the object storage to be used in the HA deployment. By default, the property is set to false, indicating that private object storage is used. To use the BMC Helix object storage, set the property to true.   

     USE_ADE_STORAGE: "true"

     Important

     When you set the USE_ADE_STORAGE property to true, it takes precedence over the values that you have configured for the private object storage properties. Only one type of object storage can be active at a time, as both cannot operate simultaneously.

   • DATA_PUSH_INTERVAL: Indicates the interval (in milliseconds in epoch format) at which an on-premises gateway instance should push data to the BMC Helix object storage (default and minimum 300000 milliseconds).

     DATA_PUSH_INTERVAL: "300000" 
   • SOURCE_HOSTNAME: Indicates the name of the server where the on-premises gateway is deployed. Replace $HOSTNAME with the host name. Don't provide a fully-qualified domain name. 
5. Save and close the file.
6. Restart the container services by using the following commands:

   • For Docker deployments:

     docker-compose down
     docker-compose up -d

   • For Podman deployments:

     podman-compose down
     podman-compose -f podman-compose.yaml up -d

     When you restart the services, the on-premises gateway instance is added as the primary instance.

7. Log on to the subsequent on-premises gateway instances and repeat steps 2 to 6.
   The subsequent instances are added as secondary instances in the storage.

To import the custom or CA-signed certificates into on-premises gateway

If you are using MinIO as the object storage, import the custom or CA-signed certificates into the on-premises gateway instances. 

1. Obtain the custom or CA-signed certificates from the MinIO server and save it in the /<IIGateway_INSTALL_DIR>hii/conf/certs directory.
2. Navigate to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory.
3. Copy the cacerts file present in the swp-mediator container at /opt/java/lib/security to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory by using the following command:

   • For Docker deployments:

     docker cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts

   • For Podman deployments:

     podman cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts

4. Import the certificates into the copied cacerts file by using the keytool utility:

   keytool -importcert -file minio.crt -keystore cacerts -alias minio
5. Navigate to the /IIGateway_INSTALL_DIR/hii directory.
6. Open the docker-compose.yaml file or podman-compose.yaml file by using a text editor. 
7. Search for the mediator section.

8. Add the following line to the volumes section:

   - ./conf/certs/cacerts:/opt/java/lib/security/cacerts
9. Save and close the file.
10. Go to Step 7 to continue configuring the on-premises gateway.

To troubleshoot HA issues

You might encounter the following issue if you are using the MinIO instance that is present in the BMC Helix IT Operations Management on-premises environment.

Issue

The on-premises gateway fails to upload the Leader_backup.json file to the MinIO instance, and the following error message appears in the swp-mediator container logs:

Resolution

1. Open the ingress-nginx-controller ConfigMap for the NGINX Ingress controller.
2. Add the following parameter and save the ConfigMap:
   ignore-invalid-headers: "false"
3. Perform a rolling restart of NGINX Ingress controller pods.
4.  Restart the container services on the on-premises gateway host by using the following commands:

   • For Docker deployments:

     docker-compose down
     docker-compose up -d

   • For Podman deployments:

     podman-compose down
     podman-compose -f podman-compose.yaml up -d