Troubleshooting upgrade issues with certificate authentication

This topic describes the process for upgrading environments that use Transport Layer Security (TLS) certificates to communicate with BMC Server Automation agents using a certificate with a key size of 1024. 

This topic contains the following sections:

Overview

Beginning in BMC Server Automation version 8.7, if you have enabled certificate authentication level of security you must have at least a 2048-bit key in place; otherwise, you will encounter errors after upgrading. 

If your id.pem certificate was created with a key size of 1024-bits (or less), you must regenerate it with a higher key size (2048 or higher). Beginning with BMC Server Automation versions 8.7 Patch 3, the certificate generation utility (bl_gen_ssl utility) generates a keySize of 4096-bit, by default.

This process is required for the following reasons:

  • If you are using 1024-bit keys and you have certificate authentication configured, then you must perform the procedure outlined in this topic before the upgrade.
  • If there is a 1024-bit id.pem present in your environment, then updates for Configuration Objects (CO) fail, even if you don’t have certificate authentication configured. 

If you are not sure if you have 1024-bit keys in your environment, you can run the following command on your Application Server to determine the current certificate keySize:

openssl rsa -text -noout -in <path to id.pem> -passin pass:<passPhrase> | grep Key

The following example shows the UNIX or Linux Application Server with the default install path: /opt/bmc/bladelogic/NSH/br/.bladelogic/id.pem

openssl rsa -text -noout -in /opt/bmc/bladelogic/NSH/br/.bladelogic/id.pem -passin pass:changeit | grep Key

The key size will be displayed at the top of the output, for example:

Private-Key: (1024 bit)

If you don't have openssl installed in your Windows environment, you can find a copy located in the unified product installer, for example: ..\BBSA89-WIN64\Disk1\files\openssl\windows. You can then run the following command:

C:\BBSA89\BBSA89-WIN64\Disk1\files\openssl\windows>openssl.exe rsa -text -noout
-in C:\Windows\rsc\certs\SYSTEM\id.pem -passin pass:changeit | grep Key
Private-Key: (2048 bit)

Step 1: Disable certificate authorization

Use the following procedure to stop using client-side certificates that secure access between Application Servers and agents or repeaters.

  1. Set up root or Administrator privileges on each managed server hosting an agent or repeater. 
    To perform this procedure, you must have root or Administrator privileges on any servers hosting agents or repeaters where you want to discontinue use of client-side certificates. 
    To grant this privilege, update the exports file by creating the following entry on the server: 

    (Windows
    <host> rw,user=Administrator 

    (UNIX)
    <host> rw,user=root 
    where <host> is the IP address or host name of the Network Shell client.
  2. Remove the SHA1 fingerprint of the Application Server self-signed certificate from managed servers by entering one the following commands, based on your environment: 
    (Windows
    nukecert SYSTEM <agent1...agentN>

    (UNIX
    nukecert bladmin <agent1...agentN> 
    where <agent1...agentN> is a space-delimited list of the names or IP addresses of the servers where you want to stop using the Application Server self-signed certificate.
  3. Configure the secure file on all agents or repeaters where you want to stop using certificates by using Network Shell to run the following secadmin command: 
    secadmin -m rscd -p 5 -T encryption_only -e tls 
    Running this command generates an rscd entry in the secure file like the following:

    rscd:port=4750:protocol=5:tls_mode=encryption_only:encryption=tls
    

    Tip

    You can also run this command using nexec from the Application Server (using nexec <hostname> secadmin ...) or by using a NSH script job.

  4. Revert the setting in the exports file on managed servers back to a more restrictive user mapping. Otherwise, all users accessing those agents are mapped to root or Administrator.
  5. Remove certificates from Application Servers by deleting the SYSTEM directory for Windows Application Servers or the .bladelogic directory for UNIX Application Servers.
    • For Windows Application Servers, the SYSTEM directory can be found at C:\<WINDIR>\rsc\certs\SYSTEM, where <WINDIR> is typically windows.
    • For UNIX Application Servers, the bladmin directory can be found at /opt/bmc/bladelogic/NSH/br/.bladelogic.

Step 2: Generate the new certificate

Download the updated bl_gen_ssl utility from BMC Communities to a temporary directory and then complete the following steps, depending on your Application Server environment. 

You can generate a keySize that is any multiple of 512 (the RSA algorithm requirement) and is equal to or greater than 2048.

On Windows

  1. Log into a Windows Application Server as Administrator.
  2. Create a directory into which you will extract the files (downloaded from BMC Communities) . 
  3. Extract the contents of the zip file you downloaded (either bl_gen_ssl_x64 or bl_gen_ssl_x32, depending on your platform) into the temporary directory.
  4. Using a command line, generate a self-signed Application Server certificate by entering the following: 
    bl_gen_ssl -keylength <length> -appcert 
    where <length> any multiple of 512 (the RSA algorithm requirement) and is equal to or greater than 2048. Possible values are 2048, 4096, 8192, and so on. By default, the utility generates a key length of 4096.

    Note

    If you are generating the id.pem file on a Windows Application Server for the SYSTEM user, use the following command: bgs.bat -keylength 2048 –appcert 

    If you are generating the id.pem file on a Windows repeater, use the following command: bgs.bat -keylength 2048 –repeatcert


  5. You are prompted to provide and then confirm a passphrase. This passphrase is used to encrypt the private key in the id.pem file. 
    The id.pem file is generated by default into C:\Windows\rsc\certs\SYSTEM
  6. Copy the id.pem file to following directory, based on your environment:
    • Linux Application Server: <user’s home directory>
    • Windows Application Server:  C:\Windows\rsc\certs\SYSTEM (if it is not already there)
  7. Update the securecert file to include an encoded copy of the passphrase. To accomplish this, use the command line to enter the following: 
    secadmin -m default -cu SYSTEM -cp <passPhrase> 
    After issuing this command, the contents of the securecert file are updated to appear similar to the following. The encoded passphrase varies.

    [default]
    SYSTEM=FCUVOMLNGLVRZNOO
    

    For the initial installation of BMC Server Automation, you can find the securecert file in the C:\<WINDIR>\rsc directory. If additional instances of BMC Server Automation are installed, you can find securecert in <installDirectoryN>\NSH\conf\securecert. For example, the default location for the second instance of BMC Server Automation could be C:\Program Files\BMC Software\BladeLogic2\NSH.

On UNIX

  1. Log into the UNIX system on the Application Server as root, and then enter the following command: 
    su - bladmin 
    This command logs you in as the bladmin user.
  2. Create a directory into which you will extract the files (downloaded from BMC Communities). The directory must have 700 permission. 
  3. Extract the contents of the tar file you downloaded (either bl_gen_ssl_RHAS40_x64.tar.gz or bl_gen_ssl_RHAS30.tar.gz, depending on your platform) into the temporary directory.
  4. Enter the following command: 
    bl_gen_ssl -keylength <length> -appcert 
    where <length> any multiple of 512 (the RSA algorithm requirement) and is equal to or greater than 2048. Possible values are 2048, 4096, 8192, and so on. By default, the utility generates a key length of 4096.
  5. You are prompted to provide and then confirm a passphrase. This passphrase is used to encrypt the private key in the id.pem file. 
    The id.pem file is generated in the current working directory. On UNIX, the Application Server runs as the bladmin user.
  6. Enter exit to revert back to the root user.
  7. Update the securecert file (contained in the /etc/rsc directory) to contain an encoded copy of the passphrase. To accomplish this, use Network Shell to enter the following: 
    secadmin -m default -cu bladmin -cp <passPhrase>
    After issuing this command, the contents of the securecertfile are updated so they are similar to the following. The encoded passphrase varies.

    [default]
    bladmin=FCUVOMLNGLVRZNOO
    
  8. Ensure that access is restricted to the id.pem file and the .bladelogic directory by running the following commands:

    chmod 700 /opt/bmc/bladelogic/NSH/br/.bladelogic
    chmod 600 /opt/bmc/bladelogic/NSH/br/.bladelogic/id.pem

Step 3: Deploy the new certificate

Perform the following manual steps (depending on environment) to deploy the new certificate.

On Windows

  1. Ensure that the secure file on all managed servers is configured so that tls_mode=encryption_only. If necessary, generate this setting by running the following secadmin command on each agent: 
    secadmin -m rscd -p 5 -T encryption_only -e tls

    Tip

    You can also run this command using nexec from the Application Server (using nexec <hostname> secadmin ...) or by using a NSH script job.

    Before you can provision a managed server with the fingerprint of the Application Server's certificate, you must ensure that the secure file on the agent or repeater is configured correctly. If you prematurely set the rscd entry in a secure file so that tls_mode=encryption_and_auth, the agent or repeater will refuse the incoming connection because it will not have the SHA1 fingerprint of the Application Server's self-signed cert. The secure file must have the rscd entry set as shown below when deploying the certificate fingerprint. The secure file is located in the /etc/rsc directory on a UNIX server and C:\<WINDIR>\rsc on Windows, where <WINDIR> is typically windows
    rscd:port=4750:protocol=5:tls_mode=encryption_only:encryption=tls
    This is the default setting after a fresh installation of an agent, so in most situations there is no need to perform this step.

  2. Set up root or Administrator privileges on each managed server hosting an agent. 
    To provision an agent or repeater with the SHA1 fingerprint of an Application Server's certificate, you must have root or Administrator privileges on the server hosting the agent. To grant this privilege, update the exports file on a Windows server by creating the following entry: 
    * rw,user=Administrator
    On a UNIX server, update the exports file by creating the following entry: 
    * rw,user=root
    To be safe, you should replace the host wildcard ("*") with a more restrictive setting, such as the IP address or host name of the Application Server.
  3. Using a command line on the Application Server, cd to C:\WINDIR\rsc\certs\SYSTEM, the directory containing the id.pem file.
  4. Push the SHA1 fingerprint to managed servers by entering the following command: 
    putcert SYSTEM id.pem <agent1...agentN>
    where <agent1...agentN> is a space-delimited list of the host names or IP addresses (IPv4 or IPv6) of the managed servers hosting agents or repeaters. 
    This command creates or updates a fingerprint file on each targeted agent or repeater. On a Windows machine, the fingerprint file for a Window Application Server is C:\Program Files\BMC Software\BladeLogic\RSCD\certs\SYSTEM; on a UNIX machine, the fingerprint file for a Windows Application Server is /opt/bmc/bladelogic/NSH/certs/SYSTEM
    In environments where multiple Application Servers communicate with agents, you should provision each Application Server with its own self-signed certificate. Performing this procedure for each of those Application Servers generates multiple fingerprints in the SYSTEM file.
  5. Revert the setting in the exports file on managed servers back to a more restrictive user mapping. Otherwise, all users accessing those agents are mapped to root or Administrator.

Note

Ensure that you keep a backup of the id.pem file. This backup file will enable you to restore communication between the Application Server and your RSCD Agents if the id.pem file is inadvertently removed or changed. See also Backup and restore of the BladeLogic environment.

 

On UNIX

  1. Ensure that the secure file on all managed servers is configured so that tls_mode=encryption_only. If necessary, generate this setting by running the following secadmin command on each agent: 
    secadmin -m rscd -p 5 -T encryption_only -e tls

    Tip

    You can also run this command using nexec from the Application Server (using nexec <hostname> secadmin ...) or by using a NSH script job.

    Before you can provision a managed server with the fingerprint of the Application Server's certificate, you must ensure that the secure file on the agent or repeater is configured correctly. If you prematurely set the rscd entry in a secure file so that tls_mode=encryption_and_auth, the agent or repeater will refuse the incoming connection because it will not have the SHA1 fingerprint of the Application Server's self-signed cert. The secure file must have the rscd entry set as shown below when deploying the certificate fingerprint. The secure file is located in the /etc/rsc directory on a UNIX server and C:\<WINDIR>\rsc on Windows, where <WINDIR> is typically windows
    rscd:port=4750:protocol=5:tls_mode=encryption_only:encryption=tls
    This is the default setting after a fresh installation of an agent, so in most situations there is no need to perform this step.

  2. Set up root or Administrator privileges on each managed server hosting an agent from the application server host. 
    To provision an agent or repeater with the SHA1 fingerprint of an Application Server's certificate, you must have root or Administrator privileges on the server hosting the agent. To grant this privilege, update the exports file by creating the following entry: 
    (Windows)<appserver hostname> rw,user=Administrator
    (UNIX)<appserver hostname>  rw,user=root
    where <appserver hostname> is the hostname or IP address of the application server host. Ensure that you revert these settings to more restrictive settings after performing the next two steps, as discussed in step 5.
  3. Using a command line on the Application Server, cd to /opt/bmc/bladelogic/NSH/br/.bladelogic, the directory containing the id.pem file.
  4. Push the SHA1 fingerprint to managed servers by entering the following command: 
    /opt/bmc/bladelogic/NSH/sbin/putcert bladmin id.pem <agent1...agentN>
    where <agent1...agentN> is a space-separated list of the host names or IP addresses (IPv4 or IPv6) of the managed servers hosting agents or repeaters. 
    This command creates or updates a fingerprint file on each targeted agent or repeater. On a Windows machine, the fingerprint file for a Window Application Server is C:\Program Files\BMC Software\BladeLogic\RSCD\certs\bladmin; on a UNIX machine, the fingerprint file for a Windows Application Server is /opt/bmc/bladelogic/NSH/certs/bladmin
    In environments where multiple Application Servers communicate with agents, you should provision each Application Server with its own self-signed certificate. Performing this procedure for each of those Application Servers generates multiple fingerprints in the bladmin file.
  5. Revert the setting in the exports file on managed servers back to a more restrictive user mapping. Otherwise, all users accessing those agents from the application server host are mapped to root or Administrator.

Note

Ensure that you keep a backup of the id.pem file. This backup file will enable you to restore communication between the Application Server and your RSCD Agents if the id.pem file is inadvertently removed or changed. See also Backup and restore of the BladeLogic environment.

Step 4: Enable certificate authorization

Use the following manual procedures (depending on environment) to enable certificate authorization between Application Servers and agents or repeaters. 

On Windows

Use this procedure to update the rscd entry in each agent's secure file so it reads as follows:

rscd:port=4750:protocol=5:tls_mode=encryption_and_auth:encryption=tls

After agents are provisioned with the SHA1 fingerprint of an Application Server's self-signed certificate, the secure files on those agents must be updated so tls_mode=encryption_and_auth. This setting requires client authentication via client-side certificates.

To modify the rscd entry in the secure file on each targeted agent, use Network Shell to enter the following secadmin command:

secadmin -m rscd -p 5 -T encryption_and_auth -e tls

Tip

You can also run this command using nexec from the Application Server (using nexec <hostname> secadmin ...) or by using a NSH script job.

This procedure is also necessary if you are configuring a repeater to authenticate incoming requests from an Application Server.

 

On UNIX

Use this procedure to update the rscd entry in each agent's secure file so it reads as follows:

rscd:port=4750:protocol=5:tls_mode=encryption_and_auth:encryption=tls

After agents are provisioned with the SHA1 fingerprint of an Application Server's self-signed certificate, the secure files on those agents must be updated so tls_mode=encryption_and_auth. This setting requires client authentication via client-side certificates.

To modify the rscd entry in the secure file on each targeted agent, use Network Shell to enter the following secadmin command:

secadmin -m rscd -p 5 -T encryption_and_auth -e tls

Tip

You can also run this command using nexec from the Application Server (using nexec <hostname> secadmin ...) or by using a NSH script job.

This procedure is also necessary if you are configuring a repeater to authenticate incoming requests from an Application Server.

Step 5: Upgrade BMC Server Automation

Once you have confirmed that secure communication is working in your current BMC Server Automation version, you can upgrade your Application Server and other BMC Server Automation components.

See one of the following topics, depending on your environment:

Walkthrough: Upgrading to version 8.7 for Microsoft Windows

Walkthrough: Upgrading to version 8.7 for Linux

Walkthrough: Installing individual components for Linux and Oracle environments

Walkthrough: Installing individual components for Microsoft Windows and SQL Server environments

Was this page helpful? Yes No Submitting... Thank you

Comments