Troubleshooting an Update Server Properties Job

This topic contains the following sections:

Agent verification

After successfully completing the Virtual Guest Job, BMC Cloud Lifecycle Management starts the Update Server Properties (USP)  job in BMC Server Automation. When the USP job completes, BMC Cloud Lifecycle Management obtains the RSCD agent status from the target virtual machine to determine if the agent is responding and licensed.

The log section shown below illustrates the process of verifying the RSCD agent.

USP Job Windows-4_USP_Job_1316723136960 is complete. Attempting to validate agent status. Poll count: 1
Looking for agentStatus
VM agent status could not be validated. Waiting to rerun USP job...
USP Job Windows-4_USP_Job_1316723136960 is complete. Attempting to validate agent status. Poll count: 2
VM agent is already licensed.

In BMC Cloud Lifecycle Management 3.0 and later releases, you do not have to license RSCD agents because agents are licensed by default beginning with BMC Server Automation 8.2. 

However, if you are dealing with earlier releases and the RSCD agent is not licensed, BMC Cloud Lifecycle Management triggers the licensing flow. The RSCD license process requires BMC Server Automation to have the license portal settings configured and an Internet connection.

After the RSCD agent is responsive and licensed, BMC Cloud Lifecycle Management retrieves certain properties from the target virtual machine that are required for cloud operations.

Re-run mechanism

After the USP job executes, BMC Cloud Lifecycle Management polls for the agent status.  If the agent-status poll indicates the agent is not licensed or responding, BMC Cloud Lifecycle Management repeats the process of executing the USP job and checking the agent status.  This process repeats until the agent responds or the maximum number of polls are performed. This maximum is configured by BBSA_VG_AGENT_STATUS_POLLS.

To control the number of retries, configure the following AccessAttributes in providers.json:

• BBSA_VG_AGENT_STATUS_POLLS 

• BBSA_VG_AGENT_STATUS_POLL_INTERVAL_SECONDS  

Troubleshooting

Check the USP job run logs from the BMC Server Automation Console for failures.

Ideally, the USP job should successfully complete within 2 or 3 attempts. If the USP job executes more than 3 times, investigate the USP job run logs from the BMC Server Automation Console. Be aware that you may see a green status for all the USP job runs, even though the Application Server could not reach the target VM or the agent is not responding. You must drill down to the job run to see the details.

Right-click on the specific USP job from  the CSM_Jobs folder and click on Show Results. From the Show Results pane, right click on the job run and select Show Log. Expand the tree downwards and select the host name, as shown below.

For the USP job to complete successfully, the Application Server must be able to reach the target virtual machine by the name used to enroll the machine in BMC Server Automation.

A USP Job can fail for the following reasons:

• Networking or DNC Issues: Target VM is NOT reachable from BMC Server Automation Appserver.
• Insufficient RSCD Permissions - Renaming Administrator account on source template
• OS level firewall settings on the source template
• Custom Object Error - Incorrect version of RSCD agent on the source template

Networking or DNS Issues

Ensure the Application Server can ping the target VM by both host name and IP address. Cross-verify the IP address from the target VM.

Fix any issues at the OS and DNS level to ensure connectivity.

Refer to the KA # KA304875 to learn more about Java DNS caching.

DNS Settings in VGP

BMC Cloud Lifecycle Management 2.x and 3.x do not use the DNS configuration settings from the Virtual Guest Package for template-based provisioning. However, for bare metal provisioning, BMC Cloud Lifecycle Management can use the DNS configuration settings in the Network tab of OS packages created in BMC Server Automation.

Static IP Assignment

When static IP assignment is used for a management NIC, BMC Network Automation only provides the IP address and not the DNS server address to the VMs. Due to this limitation, DNS registration does not occur in the environment. Thus, if the VM is enrolled by host name in BMC Server Automation, the Application Server is not able to resolve the target VM.

Also, in secured DNS environments, hosts cannot update the DNS. In these scenarios, use the following workarounds:

• Use a pre-provisioning BMC Atrium Orchestrator callout (customization) to create an entry in the DNS.
• Register the VMs by IP address rather than name.
  This is a system-wide requirement. Any provisioned server should be enrolled by IP Address instead of host name in BMC Server Automation as well as in VMware.

The property BBSA_REGISTER_SERVERS_BY_HOSTNAME in providers.json on the Platform Manager controls how the VM is registered in BMC Server Automation. By default it is set to false, but the property is not used if DHCP assignment is used for the management NIC. When you start using a static IP, by default BMC Server Automation begins registering the VMs by IP address based on this property.

Insufficient RSCD Permissions: Renaming Administrator account on source template

To perform certain tasks on the target VM via the RSCD agent, you must have certain permissions on the source template so all VMs that are provisioned from this template get those permissions.

Ensure that the exports, users and users.local files have sufficient permissions configured so that the BMC Server Automation Application Server can connect to the target VM via the BLAdmin user and retrieve server information.

In some scenarios, based on the customer environment, the built-in Administrator user account must be assigned a different name on the source template. In this case, the exports file must be modified accordingly. For example, if the built-in Administrator account was modified to clmAdmin, the exports file should reflect that user name as * rw,user='clmAdmin'.

The user mapped in the exports file must be present on the target server. However, when the VM is cloned and customized in the Virtual Center, the user account is renamed back to Administrator, thus invalidating the exports file mapping. This issue occurs specifically for Windows 2008 OS on VMWare, and it is by design. In this case, the USP job can never complete successfully. The service offering instance fails after completing all the USP job runs, (60 times by default).

*\[Class=VirtualGuestTaskExecHelper:execute\] - Error creating VG: <hostname>*

*java.lang.IllegalStateException: Done running USP jobs.&nbsp; The agent is not responding.*

If there is a requirement to change the built-in Administrator user on the source template, test the scenario first in Virtual Center directly. 

Ensure the user account is not renamed again by the Guest OS Customization process. As in case of Windows 2008, you can avoid this by using scripts that are executed after USP job completes.

OS level firewall settings on the source template

In certain scenarios, due to security reasons, restrictions such as OS-level firewall settings are applied to the source template. This may block incoming requests, including ping requests. Due to such restrictions, the USP job may not complete successfully.

Configure the source template so the USP job is not affected by any firewall rules.

Custom Object Error: Incorrect version of RSCD agent on the source template

After the USP job is complete and BMC Cloud Lifecyle Management finds the RSCD agent is responding and licensed, BMC Cloud Lifecyle Management attempts to retrieve the custom object SystemInfo (Hardware Information) from the Application Server.

A mismatch in the RSCD version between the BMC Server Automation Application Server, the VC Server, and the source template could cause the following error:

\[ERROR\] API - \[Class=VirtualGuestTaskExecHelper:execute\] - *Error  creating VG: <hostname>*
java.lang.IllegalStateException: *Custom Object version was not found in the BBSA response*
at com.bmc.cloud.provider.bbsaprovider.task.BaseBBSATaskExecHelper.getAssetCOVersion(BaseBBSATaskExecHelper.java:728)

Every CLM version has a set of certified BMC Server Automation versions. BMC recommends using RSCD versions similar to the BMC Server Automation Application Server version across the BMC Cloud Lifecyle Management environment, including the Virtual Center Server and source VM templates.

Upgrade the RSCD agent of the source VM template if it is not the same version as BMC Server Automation and the VC Server.