Troubleshooting the RSCD Agent connectivity issues

The RSCD logs are typically located inside the installation directory. The live log file is called the rscd.log. The rolled log files are named as rscd.logN, where N is a number. On Windows, the additional rscdsvc.log file contains logs for the Windows service startup. The log file locations:

• UNIX

/opt/bmc/bladelogic/RSCD/log

• Windows

C:\Program Files\BMC Software\BladeLogic\RSCD

If the log files are not present at this location, inspect the log4crc.txt file that is present in the C:\Windows\rsc or the /etc/rsc directory, and search for the following line:

<category name="rscd" priority="info1" appender="/opt/bmc/bladelogic/NSH/log/rscd.log" debugappender="stderr"/>

Do the following:

1. Edit the log4crc.txt file that is present in the C:\Windows\rsc or /etc/rsc directory and locate this line:
   <category name="rscd" priority="info1" appender="/opt/bmc/bladelogic/NSH/log/rscd.log" debugappender="stderr"/>
2. Change the priority to debug.
   
3. (For Windows RSCD Agents) Set the rollmaxfiles setting to 60. Windows RSCD agents need an additional change to retain more rolled log files. During startup, the service attempts to start the RSCD process 50 times. To capture the first restart, the rollmaxfiles setting change is required. Locate the following line:
   

   <appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/rscd.log" type="digisign" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10" layout="dated" certfile="C:/W
indows/rsc/certificate.pem" privatekeyfile="C:/Windows/rsc/certificate.pem"/>

   The line may look as follows when secure agent logging is not used:

   <appender name="C:/Program Files/BMC Software/BladeLogic/RSCD/rscd.log" type="rollfile" rollsize="10000000" rolltimeinsec="2419200" rollmaxfiles="10" layout="dated"/>

4. Save the changes.
5. Restart the RSCD Agent.

You can use the agentctl command to change the logging level of the running Agent to debug. A restart of the RSCD reverts to the logging specified in the log4crc.txt file.

You can run this command remotely or locally.

To run this command remotely:

nexec <targetName> agentctl toggle

To run this command locally:

• UNIX
  /opt/bmc/bladelogic/RSCD/sbin/agentctl toggle
• Windows
  C:\Program Files\BMC Software\BladeLogic\RSCD\agentctl toggle

Running the command again reverts to the previous logging level.

• Identify what is not able to connect to the RSCD Agent. For instance, all Application Servers, one Application Server, one client, or all clients.
• Verify whether the issue is common for all RSCD Agents or specific Agents. For example, RSCD Agents in a certain location or the Agents running on targets with a specific operating system.
• Verify whether the issue is intermittent.
• Narrow down the problem as much as possible. For example, if the problem is intermittent and occurs on multiple Application Servers and Agents, focus on a single Application Server and a small number of Agents first.

From the system with the connectivity problem, start nsh and run agentinfo <targetName>

Do you get a response ? An error message?

The table in the "Resolutions for common RSCD Agent error messages" section contains a list of common messages received from agentinfo and how to resolve connectivity problems or acl issues if an error message is returned.

4

If you do not receive one of the specified messages, start investigating the state of the RSCD on the target.

Connect to the target system via other means such as ssh or RDP as an administrative user.

5

Verify that the RSCD Agent is running.

After you have connected to the target system, check the agent processes:

UNIX:
Log in to the target server and view the process queue with the ps command, and check the process 'rscd'.

ps -ef | grep 'rscd'

Windows:
Start the Windows Task Manager or run

tasklist | findstr RSCD

and check the RSCDsvc.exe and RSCD.exe processes.

If the specified processes are not present, start the RSCD service.

• On UNIX: Use the startup scripts (usually found in /etc/init.d).
• On Windows: Use the service control manager and look for TrueSight Server Automation RSCD Agent. After starting the service, confirm that you see the processes.

If the Agent does not start, enable debug logging, start the Agent, collect the RSCD log files, open a case with BMC Support and include the log files.

6

Verify that the Agent is running on the expected port (typically 4750).

After confirming the agent has started, confirm that it is running on the expected port. The default port is 4750 for TCP. You can configure the port by updating the secure file (/etc/rsc/secure or C:\Windows\rsc\secure).

If the agent is listening correctly, the following output is expected:

• UNIX:
  netstat -anp
  tcp        0      0 0.0.0.0:4750            0.0.0.0:*               LISTEN      1249/bin/rscd  
• Windows:
  netstat -abno
  TCP 0.0.0.0:4750 0.0.0.0:0 LISTENING 5592
  [RSCD.exe]

If the process are not bound to the port defined in the secure file, enable debug logging, start the agent, collect the RSCD log files, open a case with BMC Support and include the gathered log files.

7

Verify whether you can connect to the RSCD Agent port on the target system from the system with the connectivity issue.

Do the following to initiate a connection to the target:

1. telnet <targetSystem> <rscd port>
2. Inspect the rscd.log file on the target system for an entry such as:
   211dd2c0d8ac9182fe04 0000000093 06/20/20 16:00:04.003 WARN rscd - ::ffff:<source system> 1033 -1/-1 (Not_available): (Not_available): TLS setup failed for agent: Protocol mismatch.
   Check that client and server "secure" files match. Exiting and terminating connection.

If you do not see such a message, start investigating the network path between the systems.

8

Verify that the firewall is not blocking access.

Ensure that the target system does not have the RSCD Agent port blocked by a firewall, which is installed either locally on the target or on the network between the Application Server and the target.

If a firewall is blocking access, configure it to allow connectivity from the Application Server to the target on port 4750 for TCP (default port).

9

Verify whether the AntiVirus or Host Intrusion Protection installed on the target is blocking the Agent.

Review the logs of the security agent and check for indications that it is blocking access to the RSCD process or port.

Check that a mapping exists for the incoming user in the users.local or users file and the nouser entry is present in the users file. If there is no mapping entry for the incoming user, the nouser entry blocks access to any unmatched requests. In this case, the following message is displayed in the rscd.log such as:

e2d2854dd45bc9ce7c2f 0000000050 06/24/20 16:43:44.182 WARN rscd - ::ffff:192.168.12.20 2058 0/0 (BLAdmins:BLAdmin): agentinfo: Failed to map user to local user

To correct this problem, grant RBAC permissions on the server object to the user and role, and then push acls to the target system, or add an entry to users.local to grant the role and user access.

No authorization to access host

Check that the exports file on the RSCD host grants the connecting system access. If access is not granted, the following message is displayed in the rscd.log:

be695c0f233edde4635b 0000000049 06/24/20 16:39:41.945 WARN rscd - ::ffff:192.168.12.20 2046 -1/-1 (Not_available): (Not_available): Host not granted access

In this case, update the exports file to grant access to the connecting system.

Configuring the exports file

Configuring the users or users.local files

Controlling server access with agent ACLs

If the following message is displayed:

3f0f1204dbf7ebfc8d0a 0000000058 06/24/20 17:17:56.293 WARN rscd - ::ffff:192.168.12.20 2117 1001/1001 (BLAdmins:BLAdmin): agentinfo: command: "agentinfo" not authorized

Check the mapping entry for the user and verify whether any commands restrictions are applied to the mapping entry. For example,

BLAdmins:BLAdmin rw,map=root,commands=mv,cp

Alter the command authorizations the role has on the server object.

Login not allowed for user

• Check that the mapped user exists on the target system.
• If the problem target is a domain controller, check that you do not have duplicate BladeLogicRSCDDC accounts in your domain.
  • If present, stop the RSCD service on all domain controllers and delete the BladeLogicRSCD account in the domain.
  • Force replication from the PDC emulator, start the RSCD service on the PDC emulator, and then force the replication again.
  • Start the RSCD service on the other domain controllers.
• Check that the BladeLogicRSCDDC account is not locked out.
  
  
  
  

Permission denied

Check the account that is mapped to and whether that role has the required access permission. For example, if you are trying to read the system log of a target and get the following Permission denied message:

% cat //red8-2002/var/log/messages
cat: //red8-2002/var/log/messages: Permission denied

Check what user you are mapped to by running agentinfo:

% agentinfo red8-2002
[...]

User Permissions: 1001/1001 (dba/dba)

Ensure the mapped account can read the /var/log/messages file.

No route to host

If the target agent is running, this may indicate that a host-based firewall is blocking access.

Error in TLS protocol/ encryption configuration error

This issue occurs when the secure file is different on the two hosts that are interacting. Ensure that both hosts are communicating using the same protocol and encryption settings.

Confirm that the RSCD is listening on 4750 port.

This error indicates connectivity issues with the target. Verify the network connectivity to the target.

I/O error

This sometimes is shown in place of No authorization to access host errors. Use the same methods for resolving this issue. It is also seen when the secure files on each host are different.

Remote host is unknown

This error will happen when either the application server can't resolve the host, or your client can't resolve the host.

Ensure the client system can resolve the target hostname that is registered in the console.

Connection timed out

You might see this error in the following situations:

• The target is offline
• A firewall is blocking the agent port
• The agent is not running

Connection refused

This error occurs when the remote host is down and/or the Agent is not running.

It can also happen when there is a mismatch between the port the Agent communicates over (configured in the secure file) and the port configured on the Agent from the originating connection.