Controlling RSCD agent logging

This topic contains the following sections:

• Overview of the log4crc.txt file
• Syntax for the log4crc.txt file
• Using secure agent logging
• Using keystroke log files
• Configuring the UNIX syslog
• Default log4crc.txt file examples
• Related videos

Overview of the log4crc.txt file

The log4crc.txt file:

• Enables you to control Agent logging in TrueSight Server Automation so that all Agent events are logged using consistent formats.

• Is XML-based. By modifying XML tags in log4crc.txt, you can control which log files TrueSight Server Automation generates, how much information is included in each file, where each log file is generated, how often logs are rotated, and what sort of layout the contents of each log should use.

  Note

  The log4crc.txt file is used to control Agent logging. For Application Server logging, you control logging attributes using the Infrastructure Management window on the TrueSight Server Automation Console and in the Application Server profiles of each default and custom profiles. The log appender, logging level, and logging format for Application Server logs are controlled using the log4j2.XML file. For more information, see Controlling-Application-Server-logging.

• Resides in different locations on Windows and UNIX systems, as described in the following table. On Windows, you can have multiple instances of TrueSight Server Automation client applications, each with their own log4crc.txt file.

The following table shows how the location of the log4crc.txt file on Windows varies between the first instance and all subsequent instances.

Syntax for the log4crc.txt file

The syntax of the log4crc.txt file consists of three tags:
<category>
<appender>
<layout>

category tag

The <category> tag:

• Identifies the types of logging that TrueSight Server Automation generates.
• Can include three options: name, priority, and appender.

The following list shows the <category> tags included by default in the log4crc.txt file in a Windows installation. Default values vary somewhat for UNIX installations.

The name= option identifies the type of log file TrueSight Server Automation generates. The following table identifies all possible names:

The priority= option specifies the amount of information included in a log. The following table identifies the possible priority levels:

Keystroke logs (where name is set to keystroke ) support only the following options:

The appender= tag provides a name and path for a log file. Enter the path using a UNIX or Windows format. Do not use a Network Shell path.

appender

The <appender> tag specifies whether logging information is stored as a stream in a file or periodically rolled over into a new file, usually to prevent log files from getting excessively large. You can specify that log files are rolled at specified intervals or when log files reach a particular size. The <appender> tag also lets you specify secure agent logging and keystroke logging. 

When a log file is rolled, the file is renamed with a number appended to its name. For example, rscd.log is renamed to rscd.log1. All new information is then recorded in the rscd.log file. When the log file is rolled again, rscd.log1 is renamed to rscd.log2, the current log file is renamed to rscd.log1, and all new information is recorded in rscd.log. 

The <appender> tag can include three options: name, type, and layout. The following list shows the <appender> tags that are included by default in the log4crc.txt file.

The name= option must match the name (including its full path) assigned to an appender option in a <category> tag.

The type= option specifies the type of log file to generate. The following table identifies the possible types:

The layout= option specifies the type of layout used for logging information. The following table identifies all possible layouts:

layout tag

The <layout> tag defines the format of logging entries. Users should not modify the syntax of the <layout> tag. To develop additional logging formats, contact BMC customer support.

Using secure agent logging

Secure agent logging is a rolling log mechanism that protects your RSCD agent log files by:

• Securing each entry in the current log file with a Message Authentication Code (MAC) and sequence number.
• Protecting rolled log files with digital signatures.
• Verifying the integrity of log files, and recording the status of each verification. You can later check log file integrity by using the bllogman command.

For information about secure agent logging, see:

• Overview of the security processes
• Verifying the integrity of log files
• Enabling secure agent logging
• Disabling secure agent logging

Overview of the security processes

Here is an overview of the security processes that take place as an agent writes and rolls a log file.

1. Before beginning to write its first log file, the RSCD agent generates a random session key. The agent uses this key to calculate a Message Authentication Code (MAC) for each entry in the log file.
   Note that this session key is used only for the writing of this one log file. When this log file is rolled and it is time to start a new log file, the agent generates a new session key.

2. The RSCD agent starts writing its first log file — rscd.log. As it writes each log entry, it uses the session key to calculate a MAC and associate this MAC with each log entry. It also associates a sequence number with each entry. A sample log entry looks like this when digital signature is enabled:

   222007f8999c0a6bdbf3 0000000103 12/14/16 00:40:00.244   INFO    rscd -  ::ffff:10.129.115.156 7633 0/0 (root): nexec: nexec localhost ps

   The following diagram shows various parts of a log entry when digital signature is enabled. When it is not enabled, MAC and Sequence Number are not logged. Rest of the fields are the same as shown in the diagram.
   

   

   In the above diagram:

   • MAC – Represents a unique code for each entry in the log file. 
   • Sequence Number – Represents a unique number for each entry in the log file. 
   • Date – Represents the date on which an entry is logged in the log file.
   • Time – Represent the time at which an entry is logged in the log file.
   • Log – Specifies the level at which this entry is logged. For more information, see the priority option in the category tag.
   • Category Name  – Specifies the component name for which the log entry is generated. For more information, see the name option in the category tag.
   • Hostname/ IP address  – Specifies the host name or address of the peer computer communicating with the RSCD agent computer. Hostname specifies the host name of the computer or a full-qualified domain name. IP address can be IPv4 address (for example, 10.129.10.10), IPv6 address (for example, ::1), or IPv4-mapped-IPv6 address (for example, ::ffff:10.1.1.1).
   • PID/Thread ID – Each incoming request is handled by a new RSCD process (UNIX) or thread (Windows). This field specifies the respective Process ID (PID) or Thread ID. This field can be used to associate multiple log entries to a single request.
   • User Mapping – Based on the platform:
     
     • (Windows) This field provides details of the user privilege mapping. This mapping allows the RSCD agent to temporarily grant the group privileges of a local user to an unprivileged user account, called BladeLogicRSCD. For more information, see How TrueSight Server Automation grants access to RSCD agents. In the example, entry after bold shows the privilege mapping:
       12/20/16 21:10:36.940 INFO1    rscd -  ::1 8172 BladeLogicRSCD@CLM-PUN-026432->samali@CLM-BMC-02ABX:PrivilegeMapped (Administrator): nexec: > 01010 Execute remote command: cmd /c ps
     • (UNIX) This field shows Effective User Identification Number (EUID) or Effective Group Identification Number (EGID) of the user mapped on the RSCD computer. In the example, entry after bold (0/0) shows the EUID/EGID as the user mapping:
       222007f8999c0a6bdbf3 0000000103 12/14/16 00:40:00.244   INFO    rscd -  ::ffff:10.129.115.156 7633   0/0  (root) : nexec: : > 01010 Execute remote command: "ls | wc -l"
       
   • Client User Name – Represents name of the user on the client performing the current operation on the RSCD agent.
   • Program name – Represents the NSH command invoked against the RSCD computer.

   • Message – Represents the informational message, containing relevant description. In case of command execution, actual command is also logged under this field.

     Note

     In few cases, during RSCD operations, information about some of the above fields is not available. Also, there are some internal messages being logged by RSCD Agent. In such cases, the following values are used while logging messages:

     Hostname: Host name of the current machine

     User Mapping: -1/-1 (UNIX) or SYSTEM (Windows)

     Client User Name: Not_available

     Command name: Not_available

3. When it is time for a rollover, rscd.log is rolled to rscd.log1.
   The following events take place at rollover:
   • MAC verification test and sequencing test. 
     
     The agent verifies the integrity of each log entry in the rolled log file, rscd.log1, against each entry's MAC. It also verifies the sequence number (or in other words, the order) of each log entry. 
     
     If either the MAC test or the sequencing test fails, the agent raises an event (in EventLog on Windows and syslog on UNIX systems) indicating that the file has been tampered with.
   • Digital signature file. 
     
     The agent creates a corresponding digital signature file for the rolled log file rscd.log1. In this case, the corresponding signature file would be called rscd.log.sig1. 
     
     The signature file has a status field. If the rolled log file failed the MAC test or the sequencing test, the status field is set to Inconsistent. If the rolled log file passed the MAC test and the sequencing test, the status field is set to Consistent. 
     
     You can use the information stored in the status field to verify the integrity of a rolled log file, as described in Verifying the integrity of log files. 
     
     The MAC and sequence number fields are stripped as part of the process of signing the rolled log file. 
     
     At the next roll, the signature file is rolled along with its associated log file.

4. The cycle begins again, with the creation of a new random session key for use in creating MACs for the next version of rscd.log.

   Note

   If an agent is restarted, the previous log file is automatically rolled and signed at agent startup. The agent also does the MAC verification test and sequencing test on the rolled log file.

Verifying the integrity of log files

You can verify the integrity of all agent log files by using the NSH command, bllogman. 

For example:

In the above example, there are five log files on the agent machine, engrhes40vm10. One file ( rscd.log3 ) is reported as Inconsistent, which indicates that it has been tampered with.
For additional information about bllogman, see the bllogman man page.

Enabling secure agent logging

You can enable secure agent logs as part of your initial installation (see Post-install-configuration-Windows-and-Windows-R2) or later on, as described the procedure below.

1. Back up all your existing agent log files (if any). These files have names like rscd.log, rscd.log1, rscd.log2, and so on.

2. Make the following changes to the log4crc.txt configuration file:
   In the <appender> section, remove or comment out the rscd.log appender entry that has type set to rollfile:

   <-- appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/rscd.log" type="rollfile" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10" layout="dated"/-->
   

   Uncomment or add the following entry where type is set to digisign:

   <appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/rscd.log"
   type="digisign" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10"
   layout="dated" certfile="C:/WINDOWS/rsc/certificate.pem" privatekeyfile=
   "C:/WINDOWS/rsc/certificate.pem"/>
3. Stop the RSCD agent.
4. Delete all the agent log files.

5. Start the RSCD agent.

   Note

   On UNIX agents, secure agent logs are only enabled (even if you have followed these steps) if the server on which the agent is running has either a working random number generator or PRNGD installed. Otherwise, usual rolling logs are generated.

   Click here for more information on PRNGD.

    TrueSight Server Automation uses random numbers to encrypt communication securely. On HP-UX, AIX, and Linux 7 systems without a suitable Random Number Generator, the TrueSight Server Automation Application Server installation program gives you the option to install the TrueSight Server Automation Pseudo Random Number Generator Daemon (PRNGD). PRNGD is an entropy gathering daemon (EGD) that performs a variety of actions to generate random data in a secure manner. If a server includes a hardware device for generating random numbers, it usually resides in /dev/random or /dev/urandom. If that device is not functioning properly or is not FIPS compliant, the installation program advises you to install a patch to correct the problem. BMC recommends this approach for setting up random number support on a server. Most computers have a random number generator, so you can choose not to install the PRNGD. However, if you choose not to install, the installation program aborts installation because it found no suitable random number generator and you chose to install a working device of your own. You can restart the installation when that has been done.

Disabling secure agent logging

If you have enabled secure agent logging and you now want to disable it:

1. Back up the certificate.pem file and the signature files.
2. Back up all your existing agent log files (if any). These files have names like rscd.log, rscd.log1, rscd.log2, and so on.

3. Make the following changes to the log4crc.txt configuration file:
   In the <appender> section, add or uncomment the rscd.log appender entry that has type set to rollfile:

   appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/rscd.log"
   type="rollfile" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10"
   layout="dated"

   Comment out or delete the following entry where type is set to digisign:

   <!--<appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/rscd.log"
   type="digisign" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10"
   layout="dated" certfile="C:/WINDOWS/rsc/certificate.pem" privatekeyfile=
   "C:/WINDOWS/rsc/certificate.pem"/>-->
   
4. Stop the RSCD agent.
5. Delete all the agent log files.
6. Start the RSCD agent.

Using keystroke log files

You can configure the TrueSight Server Automation RSCD agent to generate keystroke logs that record nexec sessions. Whenever a remote user uses the NSH command nexec to execute a command on an agent machine, the keystroke log captures and stores the command's STDIN, STDOUT, and STDERR streams. 

Keystroke logs are:

• Similar to the secure agent logs described in Using secure agent logging.
• Rolled periodically and are digitally signed after they are rolled.
• Encrypted and so are not readable.
• Accompanied by a digital signature file, which lets you verify the integrity of a keystroke log file.

By using the NSH command blkeylogman, you can verify the integrity of all the keystroke logs on an agent machine, or a particular keystroke log file on an agent machine. 

For example:

Keystroke Logfile(s) for host engrhes40vm10 with status:\
/opt/bmc/bladelogic/NSH/log/keystroke.log () --> Consistent
/opt/bmc/bladelogic/NSH/log/keystroke.log1 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log2 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log3 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log4 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log5 () --> Inconsistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log6 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log7 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log8 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log9 () --> Consistent 
/opt/bmc/bladelogic/NSH/log/keystroke.log10 () --> Consistent engw2k3agt1%

In the above example, there are ten keystroke log files on the agent machine. One file (keystroke.log5 ) is reported as Inconsistent, which indicates that it has been tampered with. 

The active keystroke log file (/opt/bmc/bladelogic/NSH/log/keystroke.log in the above example) is also protected by MAC codes and sequence numbers.

When the active keystroke log file is rolled, the agent tests it for consistency using the MACs and the sequence numbers. These two are then stripped off from the file and a digital signature is computed for it. If the log file was detected Inconsistent during this process, an event is raised (In the Eventlog on Windows and syslog on UNIX systems). 

The blkeylogman utility also lets you:

• View the decrypted contents of keystroke log files.
• View a list of various nexec sessions that have been recorded in the keystroke logs.
• Copy a (decrypted) keystroke log file from an agent to the client host.

For more details, see the blkeylogman man page. 

You can enable keystroke logs as part of your initial installation or later on, as described in Enabling keystroke logging. To disable keystroke logging, see Disabling keystroke logging.

Enabling keystroke logging

You can enable keystroke logging as part of your initial installation or later on, as described in the procedure below.

1. Make the following changes to the log4crc.txt configuration file:
   In the <category> section, uncomment or add the following entry, where name is set to keystroke:

   <category name="keystroke" priority="info1" appender="C:/Program Files/BMC
   Software/BladeLogic/RSCD/keystroke.log"/>

   In the <appender> section, uncomment or add the following entry, where type is set to encrypt:

   <appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/keystroke.log"
   type="encrypt" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10"
   layout="rawtime" certfile="C:/WINDOWS/rsc/certificate.pem" privatekeyfile=
   "C:/WINDOWS/rsc/certificate.pem"/>

2. Restart the RSCD agent.

   Note

   On UNIX agents, keystroke logging is only enabled (even if you have followed these steps), if the server on which the agent is running has either a working random number generator or PRNGD installed.

   Click here for more information on PRNGD.

    TrueSight Server Automation uses random numbers to encrypt communication securely. On HP-UX, AIX, and Linux 7 systems without a suitable Random Number Generator, the TrueSight Server Automation Application Server installation program gives you the option to install the TrueSight Server Automation Pseudo Random Number Generator Daemon (PRNGD). PRNGD is an entropy gathering daemon (EGD) that performs a variety of actions to generate random data in a secure manner. If a server includes a hardware device for generating random numbers, it usually resides in /dev/random or /dev/urandom. If that device is not functioning properly or is not FIPS compliant, the installation program advises you to install a patch to correct the problem. BMC recommends this approach for setting up random number support on a server. Most computers have a random number generator, so you can choose not to install the PRNGD. However, if you choose not to install, the installation program aborts installation because it found no suitable random number generator and you chose to install a working device of your own. You can restart the installation when that has been done.

Disabling keystroke logging

If you have enabled keystroke logging and you now want to disable it:

1. Back up the certificate.pem file.

2. Make the following changes to the log4crc.txt configuration file:
   In the <category> section, comment out or delete the following entry, where name is set to keystroke:

   <\!--<category name="keystroke" priority="info1" appender="C:/Program Files/BMC
   Software/BladeLogic/RSCD/keystroke.log"/>-->
   

   In the <appender> section, comment out or delete the following entry, where type is set to encrypt:

   <\!--<appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/keystroke.log"
   type="encrypt" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10"
   layout="rawtime" certfile="C:/WINDOWS/rsc/certificate.pem" privatekeyfile=
   "C:/WINDOWS/rsc/certificate.pem"/>-->
   
3. Restart the RSCD agent.

Configuring the UNIX syslog

When the RSCD agent is configured to log to syslog, the local6 facility is used. To send these logs to a remote syslog system, the local syslog configuration must be setup to send the logs to the remote system. Alternatively, the logs could be directed to a local log file.  

In the /etc/rsc/log4crc.txt file, add the following:

This entry sends all events at the info1 level or higher to the local6 facility on the local system. To direct those events to a local log file, the /etc/syslog.conf file requires an entry like the following:

To direct those events to a remote host,add the following to the /etc/syslog.conf file:

Your syslog daemon configuration specifics may differ.

If the RSCD agent is configured to send logs to syslog, the log created by syslog is not managed by the RSCD Agent’s log4crc.txt settings; it is managed by the operating system facility that manages the syslog log files.

Default log4crc.txt file examples

The following is an example of a default log4crc.txt file for a Windows installation.

The following is an example of a default log4crc.txt file for a UNIX installation.

Related videos