Configuring the on-premises gateway for high availability

Provide high availability for the BMC Helix Intelligent Integrations on-premises gateway by deploying it in an active-passive high availability configuration.

Before you begin

Make sure that a standalone MinIO or Amazon Simple Storage Service (S3) instance is set up and you have obtained the credentials to access the MinIO or Amazon S3 API endpoint that is set up on the instance. Contact your system administrator to obtain the credentials.
You can use a standalone MinIO instance available in your environment or the one included in the BMC Helix IT Operations Management on-premises deployment.
Important
High availability deployment of the on-premises gateway has been validated using the Community edition of MinIO. The Enterprise edition is also expected to function correctly.
If you plan to use a webhook connector to collect data from a third-party product, you can use a load balancer (for example, F5) to direct the webhook data traffic to the active on-premises gateway node. The load balancer monitor should use the standbyStatus API to determine the status of a node.
Perform the following steps to prepare the environment for data collection by using a webhook connector. These steps are specific to the F5 load balancer and might differ for other load balancer types.
1. Create a DNS record (for example, VIP_HII) for the virtual IP (VIP).
2. Configure the F5 load balancer:
  1. Create a pool, and add the on-premises gateway nodes as members, as shown in the following example:
    ltm pool VIP_HII {
        description "Gateway Pool"
        members {
            aus-pun-01.abc.com:https {
                address 192.168.111.xx
               session monitor-enabled
               state down
           }
            aus-pun-02.abc.com:https {
                address 192.168.112.xx
               session monitor-enabled
               state down
           }
              }
        monitor VIP_HII
    }
  2. Create the monitor to determine the status of an on-premises gateway node, as shown in the following example:
    ltm monitor https VIP_HII {
    adaptive disabled
    defaults-from https
    interval 5
    ip-dscp 0
    recv false
    recv-disable none
    send "GET /swpui/api/mediator/v3/standbyStatus HTTP/1.1\r\nHost: VIP_HII\r\nConnection: Close\r\n\r\n""
    time-until-up 0
    timeout 16
    }
When confiiguring the webhook collector URL in the third-party product (for example, Entuity), replace the on-premises gateway host name with the DNS record in the URL. For example, an updated webhook collector URL might look like the folllowing: https://VIP_HII/hii/api/mediator/v3/push/5a092cf4-e795-43f6-a3af-e2c3d9ee1e9a

For more information about configuring the URL for Entuity, see Integrating with Entuity via webhook.

To configure the on-premises gateway instances for high availability

Log on to an on-premises gateway instance that you want to consider as a primary instance.
Navigate to the /IIGateway_INSTALL_DIR/hii directory.
Open the docker-compose.yaml file or podman-compose.yaml file with an editor.
Locate the mediator > environment section, and set the following properties:
- MINIO_ACCESS_KEY: MinIO or Amazon S3 access key
- MINIO_SECRET_KEY: MinIO or Amazon S3 secret key
- MINIO_SERVER_URL: MinIO or Amazon S3 API endpoint URL
  Example
  # MinIO
  https://vx-push-dev26.abc.com:9000
  # Amazon S3
  https://s3.us-west-1.amazonaws.com
- MINIO_BUCKET_NAME: The name of bucket you want to create on the MinIO or Amazon S3 instance.
  Important
  If you are using the same MinIO or Amazon S3 server for multiple HA environments (for example, Test and Production), make sure that each bucket has a unique name.
- DATA_PUSH_INTERVAL: Interval in milliseconds in epoch format at which an on-premises gateway instance should push data to a MinIO or Amazon S3 bucket in a JSON file (default and minimum 300000 milliseconds).
  The following file shows sample values:
  MINIO_ACCESS_KEY: "P3pWEoNUEmZB8i0zJAnC"
  MINIO_SECRET_KEY: "SgA3ntRrdM3nzUGpvKjRMQ2FJZNHujfngxJgTb"
  MINIO_SERVER_URL: "https://vx-push-dev26.abc.com:9000"
  MINIO_BUCKET_NAME: "helix-hii-backup"
  DATA_PUSH_INTERVAL: "300000"
Save and close the file.
(Applicable if you are using a MinIO instance) If the MinIO instance is using custom or CA-signed certificates, import them into the on-premises gateway.
Restart the container services by using the following commands:
- For Docker deployments:
  docker-compose down
  docker-compose up -d
- For Podman deployments:
  podman-compose down
  podman-compose -f podman-compose.yaml up -d
  When you restart the service, the instance is added as a primary instance and a MinIO or Amazon S3 bucket, helix-hii-backup is created. The bucket contains the configuration backup file, Leader_backup.json, which stores the instance configuration backup with timestamp. The backup file contains a tag for the on-premises instance.
Log on to the remaining on-premises gateway instances and repeat steps 2 to 6.
The subsequent instances are added as secondary instances in the MinIO or Amazon S3 bucket.

To import the custom or CA-signed certificates into on-premises gateway

Obtain the custom or CA-signed certificates from the MinIO server and save it in the <IIGateway_INSTALL_DIR>hii/conf/certs directory.
Navigate to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory.
Copy the cacerts file present in the swp-mediator container at /opt/java/lib/security to the /<IIGateway_INSTALL_DIR>/hii/conf/certs directory by using the following command:
- For Docker deployments:
  docker cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts
- For Podman deployments:
  podman cp swp-mediator:/opt/java/lib/security/cacerts /<IIGateway_INSTALL_DIR>hii/conf/certs/cacerts
Import the certificates into the copied cacerts file by using the keytool utility:
keytool -importcert -file minio.crt -keystore cacerts -alias minio
Navigate to the /IIGateway_INSTALL_DIR/hii directory.
Open the docker-compose.yaml file or podman-compose.yaml file with a text editor.
Search for the mediator section.
Add the following line to the volumes section:
- ./conf/certs/cacerts:/opt/java/lib/security/cacerts
Save and close the file.
Go to Step 7 to continue configuring the on-premises gateway.

To troubleshoot the high availability issues

You might encounter the following issue if you are using the MinIO instance that is present in the BMC Helix IT Operations Management on-premises environment.

Issue

if you are using the MinIO instance that is present in the BMC Helix IT Operations Management on-premises environment, the on-premises gateway fails to upload the Leader_backup.json file to the MinIO instance, and the following error message appears in the swp-mediator container logs:

Error occurred while putting object in bucket: error occurred\nErrorResponse(code = AccessDenied, message = There were headers present in the request which were not signed

Resolution

Open the ingress-nginx-controller ConfigMap for the NGINX Ingress controller.
Add the following parameter and save the ConfigMap:
ignore-invalid-headers: "false"
Perform a rolling restart of NGINX Ingress controller pods.
Restart the container services on the on-premises gateway host by using the following commands:
- For Docker deployments:
  docker-compose down
  docker-compose up -d
- For Podman deployments:
  podman-compose down
  podman-compose -f podman-compose.yaml up -d