Updating expired certificates

Use the information in this topic to update expired certificates. 

Determine where TLS terminates

Check the following with your Load Balancer administrator: Is the Helix VIP using a passthrough connection type?

• If a passthrough connection type is used for the Helix VIP on the Load Balancer:
  • Then SSL terminates on the Ingress Controller, and the new certificate will be updated there. (i.e. on the Ingress Controller)
  • No action required on the Load Balancer
• Alternatively, if a passthrough connection type is not used:
  • Then SSL terminates on the Load Balancer
  • Is the load balancer sending connections through to the http port on the Ingress Controller or the https port?
    • If https, then the certificate will need to be updated on both the Load Balancer and Ingress Controller
    • If http, then the certificate will need to be updated only on the Load Balancer

Obtain a new certificate

Prior to certificate expiration, obtain a new certificate from your Certificate Authority.

1. Obtain a new certificate from your CA with the same FQDNs as the current certificate.  Ensure you have the following after you receive the new certificate:
   • The new signed certificate is in pem (text) format (new.pem)
   • The key (new.key), also in text format
   • Individual pem files for each CA certificate are in the chain, there may be multiple depending on your CA (signing.pem, intermediate.pem, root.pem)
2. Create a new custom_cacert.pem by running the following command:
   cat signing.pem intermediate.pem root.pem > custom_cacert.pem. with the number of CA certs that are in the chain.
3. If TLS terminates on the nginx controller, create a new_with_ca.pem by running the following command:
   cat new.pem signing.pem intermediate.pem root.pem > new_with_ca.pem. with the signed certificate and number of CA certs that are in the chain.

Determine if the Certificate Authority Changed

Determine if your CA chain for the new certificate is different from the current certificate.  Depending on the result you may need to perform additional steps prior to applying the new certificate.

Run the following command on a Linux system by using the openssl utility to verify that the
current certificate matches the Certificate Authority (CA) used for the new certificate.% openssl s_client  -connect <LB_HOST of your Helix environment> -showcerts < /dev/null
| openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -noout -print_certs  | grep issuer

1. You must update a few ConfigMaps with the new CA chain prior to updating the certificate on the Ingress Controller or Load Balancer.  
   1. Open a machine that has openssl utility and kubectl access to the environment. 
   2. Run the following script to pull down the existing truststores and add the new CA certs, then update the environment. 
      Note: kubectl must be in the current path.
      ./changecerts.sh <helixnamespace> </path/to/custom_cacert.pem>
      For example:./changecerts.sh helixade /tmp/custom_cacert.pem

   3. Restart the affected pods to pickup the new truststore:
      # kubectl get pod -n ${NAMESPACE} | grep -iv "postgres\|redis\
      |kafka-exporter\|kafka-0\|kafka-1\|kafka-2\|kafka-zookeeper\|victoria"
      | grep Running | awk '{print $1}' | xargs kubectl delete pod -n ${NAMESPACE}
      Where, NAMESPACE is the BMC Helix IT Operations Management deployment namespace.

2. Perform the following steps to update the CA chain certs in BMC Discovery.
   1. Open a machine that has kubectl access to the environment. 
   2. Pull down the old cacerts pem file:
      # NAMESPACE=helixade
      
      # kubectl get cm -n "${NAMESPACE}" pemcm -o jsonpath='{.data.cacerts\.pem}' > new_cacerts.pem
   3. Add the new cacerts to this file.
      cat /tmp/custom_cacert.pem >> new_cacerts.pem
      Note: Ensure to append with ">>"
   4. Login to Discovery as the system user
      https://<discovery url>/ui/LocalLogin
   5. Navigate to Administration > Single Sign On.
   6. In the Trusted CA section, you see the current CA chain.
   7. Click Choose File and select the new_cacerts.pem that you created to upload the certificates.
   8. Click Replace CA Bundle.
   9. Click OK when prompted to unpin the current certificate.
      You can see the old entry CA and the new CAs. 
   10. Under Pinned Certificate, click Pin Certificate.
   11. Logout of Discovery.

Applying the new certificate

If, per the above, TLS terminates on the Load Balancer, or on the Load Balancer and the Ingress controller, coordinate the replacement of the certificate with your Load Balancer administrator.  If TLS only terminates on the Load Balancer, then no further action is needed.

If TLS terminates on the Ingress controller or the Load Balancer and Ingress controller, then perform the following steps to apply the new certificate to the Ingress controller:

• For Ingress, the signed certificate and key are stored in a Kubernetes secret
  1. Assuming that the Ingress Controller is deployed in the ingress-nginx namespace, run the following command to find the secret key by looking at the configuration of the DaemonSet.  Use the values appropriate for your environment:
     kubectl get ds ingress-nginx-controller -n ingress-nginx -o yaml | grep 'default-ssl-certificate'
     You see the output like:
     - --default-ssl-certificate=ingress-nginx/my-tls-secret
     
  2. Make a note of the secret name.
  3. Delete the existing secret.
     kubectl delete secret -n ingress-nginx my-tls-secret
  4. Using the new_with_ca.pem file that was created in a previous step, re-create the secret with new new key and file.
     kubectl create secret tls my-tls-secret --cert=new_with_ca.pem --key=new.key -n ingress-nginx
     
  5. Restart the Ingress Controller.
     kubectl -n ingress-nginx rollout restart daemonset ingress-nginx-controller
  6. Monitor the restarting pods and check that the new certificate is replaced after restart.
  7. If you are using BMC Discovery, update the certificate in BMC Discovery.
     1. login to the BMC Discovery appliance as the system user
        https://<discovery url>/ui/LocalLogin
     2. Navigate to Administration > Single Sign On.
     3. Under pinned certificate, click Get Server Certificate, then Replace certificate.
     4. Logout of the discovery appliance.