Troubleshooting the BladeLogicRSCD user lockout issues

When a BladeLogicRSCD account is locked, User Privilege Mapping (UPM) access fails. The BladeLogicRSCD account is a local Windows service account that the RSCD Agent uses to grant access to the managed Windows servers. For more information, see Impersonation-and-privilege-mapping.

Issue symptoms

When you attempt to access a managed Windows server with a locked BladeLogicRSCD account, you may see the following error:
Can't access host "<hostname>": Login not allowed for user in a job run log or RCP console message.

On the managed server, the rscd.log file (<RSCD install directory>\rscd.log) might show the following message:

865d94a4ab5624bbf931 0000001331 02/07/21 12:51:23.401 ERROR rscd - WIN19-1 2192 SYSTEM (Not_available):
(Not_available): User Impersonation Failed for mapped user BLAdmins:BLAdmin; Error Location: 
RSCD_WinUser::logonPassword:LsaLogonUser() ; Error Message: The referenced account is currently locked out
and may not be logged on to. ; Auxiliary Error Message: BladeLogicRSCD@WIN19-1

Issue scope

  • The issue may occur with a single managed Windows server or with multiple servers.
  • The affected server may be a Windows Domain Controller or a member server.

For information about other issues that cause the RSCD Agent connectivity failure, see Troubleshooting-the-RSCD-Agent-connectivity-issues.

Diagnosing and reporting an issue

Task

Action

Steps

Reference

1

Understand the problem scope.

  1. Determine the affected target servers.
  2. Verify whether the lockout occurs at the domain level BladeLogicRSCD account thereby, locking out the BladeLogicRSCD account on the domain controllers or at the local BladeLogicRSCD on some member servers.


2

Determine the source of the failed authentication attempts that lead to the account lockout.

  1. Verify whether the failed authentication attempts are due to a system other than the one with the locked account. For example, Server A has the locked BladeLogicRSCD account, but the source of the failed authentication attempts was Server B
  2. Check the Windows Event Log for Event ID 4625 on the system where the lockout occurred. In the Event Log message, locate the Account Domain and Source Network Address fields to determine:
    • The server that was the source of the failed authentication attempts
    • Whether the server is a local server or a remote server

An account failed to log in

Example Windows event log message

An account failed to log on.

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

Account For Which Logon Failed:
 Security ID: NULL SID
 Account Name: BladeLogicRSCD
 Account Domain: WIN19-2

Failure Information:
 Failure Reason: Unknown user name or bad password.
 Status: 0xC000006D
 Sub Status: 0xC000006A

Process Information:
 Caller Process ID: 0x0
 Caller Process Name: -

Network Information:
 Workstation Name: WIN19-2
 Source Network Address: 192.168.8.215
 Source Port: 51799

3

Correlate the failed authentication attempts to an action.

Check what was running on that source server at the time of lockout.

  • If the system causing the lockout is enrolled in TrueSight Server Automation:
    • Check the <RSCD install>\rscd.log for any activity around the time of the lockout. Determine if the logged actions could result in failed authentication attempts being sent to the problematic system.
    • Live Browse in the TrueSight Server Automation console, and look for any jobs running around the time of the lockout.
  • If the system is not enrolled in TrueSight Server Automation, inspect the event log or other application logs to determine what was running around the time of the failed authentication attempts and account lockout. Look for the activity that can cause a connection from the source system to the system with the lockout.


4

If the source of the failed authentication attempts is the system itself, check whether you can restart the RSCD service without any issue.

  1. Stop the RSCD Service.
  2. Unlock the BladeLogicRSCD account
  3. Start the RSCD Service

If the account lockout happens upon service restart, then the BladeLogicRSCD password may need to be reset. Follow the steps in referenced KA 000379333.

5

Determine whether the lockouts are occurring with any frequency.

If the lockout occurs multiple times, determine whether there is a pattern for the lockouts using the Event Log and other information. You can use this information to investigate only those lockouts that are occurring with the same frequency.


6

Investigate any actions that can cause the lockout.


If a Job ran around the time of the lockout, determine what the job is running. Common causes of the lockout could be:

  • Running a net use to map a share on the locked system
  • Running certutil on a domain member server causing a lock of the BladeLogicRSCD domain
  • Enumerating the file system on the source when a share is mapped to the locked system


7

Identify lockout candidates and test them in isolation.

If the cause cannot be immediately identified, then set up a time to test run each candidate in isolation when no activity is running on the source and locked system.

Depending on the account lockout threshold defined on the locked system, the candidate action may need to be run multiple times.


8

Match error messages with those in the "Resolutions for common issues" table.

Review the information in the "Resolutions for common issues" section to understand the common issues that can result in BladelogicRSCD user account lockout issues along with how they can typically be resolved.

If you are unable to identify and resolve the problem, create a BMC Support Case.


9

Creating a BMC Support Case

Provide the following information and log files when creating a case with BMC Customer Support:


    • Scope of the issue as identified in steps 1 and 2.
    • Event Log 4625 entries that include occurrences of the failed authentication attempts in step 3.
    • RSCD.log and rolled over log files from the system with the lock and any systems identified in the Event Logs.


Resolutions for common issues

Symptom

Action

Reference

rscd.log of a Member Server displays this error:

"The referenced account is currently locked out and may not be logged on to." The local BladeLogicRSCD account on the member server is being locked


Unlock the locked BladeLogicRSCD account.

This action will provide an immediate resolution to the issue, however, until the root case of the lockout is determined, it's likely the lockout might occur again. See referenced KA 000290455.

rscd.log of the domain controller displays the following error:

The referenced account is currently locked out and may not be logged on to. The domain level BladeLogicRSCD (or BladeLogicRSCDDC) account is being locked


  1. Unlock the locked account.
  2. Set up separate service accounts for each domain controller to limit the scope of the problem during investigation.

Failed authentication attempts due to BladeLogicRSCD from the same system and the account locks after starting the RSCD service.


  1. Stop the RSCD service, unlock the BladeLogicRSCD account.
  2. Use the agentctl passwd option to reset the BladeLogicRSCD user's password.
  3. Start the RSCD service.

The only supported method to reset the BladeLogicRSCD password is to use the chapw or agentctl passwd commands. Verify whether some other method was used to change the password.

Failed authentication attempts due to BladeLogicRSCD from another system, which causes the account lock on the problem system.


  1. Verify whether a relationship exists between the two systems, and also check what is being run on the system with the failed authentication attempts.
  2. Rename the BladeLogicRSCD account on the problematic system to prevent lockouts. The same registry change noted for domain controllers can also be applied to standalone or member servers.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*