Updating expired certificates

You must update the security certificates for BMC Helix Service Management when they expire, are replaced, or are regenerated to ensure secure communication. You must also update the certificates when there are changes to the host name, server, or certificate authority.

The following image describes the tasks to update the expired certificate:

Before you begin

Confirm with your load balancer administrator whether the Helix VIP is configured in passthrough mode.

• If a passthrough connection type is used for the Helix VIP on the load balancer, then:
  • TSL (SSL) terminates on the Ingress Controller, and the new certificate is updated on the Ingress Controller.
  • No action required on the load balancer
• If passthrough mode is not used:
  • SSL terminates on the load balancer.
  • Check whether the load balancer forwards traffic to http or https ports on the Ingress controller and perform one of the following steps:
    • If it forwards traffic to the https port on the Ingress, update the certificates on the Controller
      Update certificates on both the load balancer and the Ingress Controller
    • If it forwards traffic to the http port on the Ingress Controller, update the certificates only on the load balancer

Task 1: To obtain a new certificate

Before the current certificate expires, ensure that you have a new one. To obtain a certificate, perform the following steps:

1. Obtain a new certificate from your Certificate Authority (CA).
2. Ensure that the new certificate includes the same Fully Qualified Domain Names (FQDNs) as the existing certificate. After receiving the certificate, verify that you have completed the following actions:
   1. Signed certificate in PEM format (new.pem)
   2. Private key in text format (new.key)
   3. Individual PEM files for each CA certificate in the chain (for example: signing.pem, intermediate.pem, root.pem)

      ​​​​​

3. Create a new custom_cacert.pem by running the following command:
   
   cat signing.pem intermediate.pem root.pem > custom_cacert.pem.
   
   with the number of CA certs that are in the chain.
4. If TLS terminates on the Nginx Ingress controller, create a new_with_ca.pem by running the following command with the signed certificate and the number of CA certificates in the chain.
   
   cat new.pem signing.pem intermediate.pem root.pem > new_with_ca.pem.

Task 2: To determine whether the certificate authority has changed

Determine whether your CA chain for the new certificate differs from that for the current certificate.  Depending on the result, you may need to take additional steps before applying the new certificate.

1. Run the following command on a Linux system by using the openssl utility to verify that the current certificate matches the Certificate Authority (CA) used for the new certificate:% openssl s_client  -connect <LB_HOST of your Helix environment> -showcerts < /dev/null
   | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -noout -print_certs  | grep issuer

   The issuer lines indicate the Certificate Authority chain that signed your current certificate. 
   Example:
   issuer=C=US, O=Google Trust Services, CN=WR2
   
   issuer=C=US, O=Google Trust Services LLC, CN=GTS Root R1
   
   issuer=C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA

   ​​​

1. Log in to a Kubernetes system with kubectl and OpenSSL access.
2. To update the existing truststores, run the following command and add the new CA certificates. Update the environment.
   
   Warningkubectl must be in the current path.
   
   
   ./changecerts.sh <helix_platform_namespace> </path/to/custom_cacert.pem>
   
   For example:
   The changecerts.sh file is available here.
   
   ./changecerts.sh helixade /tmp/custom_cacert.pem

3. Restart the affected BMC Helix Platform Common Services pods to load the updated truststores:

   # kubectl get pod -n ${helix_platform_namespace} | grep -iv "postgres\|redis\
   |kafka-exporter\|kafka-0\|kafka-1\|kafka-2\|kafka-zookeeper\|victoria"
   | grep Running | awk '{print $1}' | xargs kubectl delete pod -n ${helix_platform_namespace}

4. To apply the new certificate
   1. Determine where TLS(SSL) terminates, for more information, see to determine where TLS (SSL) terminates.
   2. If TLS terminates on the load balancer or on both the load balancer and the Ingress controller. Coordinate with the load balance administrator to replace the certificate.
   3. If TLS terminates only on the load balancer, no further action is required.
   4. If TLS terminates on the Ingress controller or on both the load balancer and the Ingress controller, update the Ingress controller by performing the following
      steps:

1. 
   
   1. 
      
      1. Assuming that the Ingress Controller is deployed in the ingress-nginx namespace, run the following command to find the secret key by looking at the configuration of the DaemonSet.  Use the values appropriate for your environment:
         
         kubectl get ds ingress-nginx-controller -n ingress-nginx -o yaml | grep 'default-ssl-certificate'
         
         You see the output like:
         
         - --default-ssl-certificate=ingress-nginx/my-tls-secret
         
      2. Make a note of the secret name.
      3. Delete the existing secret.
         
         kubectl delete secret -n ingress-nginx my-tls-secret
      4. Using the new_with_ca.Pem file, recreate the secret with a new key and file. 
         
         kubectl create secret tls my-tls-secret --cert=new_with_ca.pem --key=new.key -n ingress-nginx
         
      5. Restart the Ingress controller.
         
         kubectl -n ingress-nginx rollout restart daemonset ingress-nginx-controller
      6. Monitor pod restart and verify certificate replacement.

Task 4: To update the security certificate of BMC Helix Service Management

1. On the BMC Deployment Engine, navigate to the HELIX_ONPREM_DEPLOYMENT pipeline.
2. In the HELIX_ONPREM_DEPLOYMENT pipeline, Build History section, select the last job, and click Rebuild.
3. In the ENVIRONMENT DETAILS section, CUSTOM_CERTIFICATE parameter, click Browse and upload your custom certificate file.

4. In the ENVIRONMENT DETAILS section, select DEPLOYMENT_TYPE as SERVICE.
5. In the PIPELINES section, select both the HELIX_GENERATE_CONFIG and HELIX_RESTART check boxes.
6. Build the HELIX_ONPREM_DEPLOYMENT pipeline by using the Rebuild option.

7. Make sure that the HELIX_ONPREM_DEPLOYMENT pipeline runs successfully.

   Warning

   Important

   • The cacerts file is checked into the Git repository when the HELIX_GENERATE_CONFIG pipeline runs successfully. In case of any failures in the HELIX_ONPREM_DEPLOYMENT or HELIX_GENERATE_CONFIG pipelines, you must upload the cacerts file in the HELIX_ONPREM_DEPLOYMENT pipeline until the HELIX_GENERATE_CONFIG pipeline executes successfully at least once.

   • You do not need to upload the cacerts file for consecutive execution of the HELIX_ONPREM_DEPLOYMENT pipeline and other pipelines if you do not select the HELIX_GENERATE_CONFIG pipeline.