Session layer security

BMC Server Automation uses Transport Layer Security (TLS) for session layer security across all communications legs. In all contexts (excluding BMC BladeLogic Decision Support for Server Automation), BMC Server Automation system components employ TLS_RSA_WITH_AES_128_CBC_SHA for the TLS cipher suite, which includes the following capabilities:

• RSA key negotiation
• 128-bit AES block encryption algorithm
• Cipher Block Chaining (CBC) block cipher mode
• SHA1 HMAC construction for integrity protection.

The BMC Server Automation Application Server and all client applications use the certified OpenSSL FIPS 140-2 object module for cryptographic operations on all transported data.

Communication with middle tier

When a BMC Server Automation client establishes a TLS connection with a middle tier entity (that is, the Authentication, Application, or Network Shell Proxy Services), the client must validate a certificate from that entity. At installation, middle tier entities are provisioned with self-signed X.509 certificates. (Optionally, you can provision middle tier entities with certificates issued by a CA. For more information, see Securing communication with CA certificates.) When a client first accesses a middle tier entity (by necessity, the Authentication Service) to authenticate and obtain an SSO credential, the client establishes a TLS connection with the Application Server hosting the Authentication Service. In the course of the TLS handshake, the client is presented with the Application Server self-signed X.509 certificate. The client cannot recognize the certificate as trusted so the client prompts the user to accept or reject the self-signed certificate. The client application displays the certificate's content, as well as its SHA1 and SHA256 fingerprints. If the user chooses to:

• Trust the self-signed certificate, the certificate is added to the client's list of trusted certificates and a secure session is established for the client.
• Not to trust the self-signed certificate, the session is terminated.

All client services running on a BMC Server Automation Application Server (Authentication Service, Application Service, and Network Shell Proxy Service) share the same certificate. If your installation employs multiple Application Servers or stand-alone Network Shell proxy servers, they too share the same certificate. After a client application has added the Authentication Service's certificate to its list of trusted certificates, the user is no longer prompted to trust that certificate when establishing future sessions with any of these other related entities.

A client's list of trusted certificates are stored in a file written in the Privacy Enhanced Mail (PEM) format. This file is known as a keystore. The keystore resides in a default location, but you can modify that location, as described in Setting override locations for client SSO files. Client applications re-write the keystore file when a trusted X.509 certificate is added to or removed from the trusted certificate store.

Communication with server tier

Self-signed certificates are used to secure communication between entities that communicate directly with agents. These entities could be Application Servers, Network Shell proxy servers, repeaters, or Network Shell clients. By default, BMC Server Automation generates self-signed certificates when an agent is installed on a server. The certificate is stored in a file called certificate.pem in the directory where the agent is installed. If this file is:

• Present, it is read every time an Application Server, Network Shell proxy server, repeater, or Network Shell client establishes a connection with the agent.
• Not present, the agent creates one. Self-signed server-side certificates are used to secure the exchange of TLS session keys between agents and entities that communicate with agents.

By default, BMC Server Automation does not use client-side certificates. However, you can choose to use self-signed client-side certificates for TLS sessions with RSCD agents. To accomplish this, you must provision agents with the SHA1 fingerprints of trusted clients' self-signed certificates.