Troubleshooting an Update Server Properties Job
This topic contains the following sections:
After successfully completing the Virtual Guest Job, BMC Cloud Lifecycle Management starts the Update Server Properties (USP) job in BMC Server Automation. When the USP job completes, BMC Cloud Lifecycle Management obtains the RSCD agent status from the target virtual machine to determine if the agent is responding and licensed.
The log section shown below illustrates the process of verifying the RSCD agent.
USP Job Windows-4_USP_Job_1316723136960 is complete. Attempting to validate agent status. Poll count: 1 Looking for agentStatus VM agent status could not be validated. Waiting to rerun USP job... USP Job Windows-4_USP_Job_1316723136960 is complete. Attempting to validate agent status. Poll count: 2 VM agent is already licensed.
In BMC Cloud Lifecycle Management 3.0 and later releases, you do not have to license RSCD agents because agents are licensed by default beginning with BMC Server Automation 8.2.
However, if you are dealing with earlier releases and the RSCD agent is not licensed, BMC Cloud Lifecycle Management triggers the licensing flow. The RSCD license process requires BMC Server Automation to have the license portal settings configured and an Internet connection.After the RSCD agent is responsive and licensed, BMC Cloud Lifecycle Management retrieves certain properties from the target virtual machine that are required for cloud operations.
After the USP job executes, BMC Cloud Lifecycle Management polls for the agent status. If the agent-status poll indicates the agent is not licensed or responding, BMC Cloud Lifecycle Management repeats the process of executing the USP job and checking the agent status. This process repeats until the agent responds or the maximum number of polls are performed. This maximum is configured by
To control the number of retries, configure the following AccessAttributes in providers.json:
Check the USP job run logs from the BMC Server Automation Console for failures.
Ideally, the USP job should successfully complete within 2 or 3 attempts. If the USP job executes more than 3 times, investigate the USP job run logs from the BMC Server Automation Console. Be aware that you may see a green status for all the USP job runs, even though the Application Server could not reach the target VM or the agent is not responding. You must drill down to the job run to see the details.
UspJobSucceedsWhenAgentDown property controls the status of the USP job. BMC does not recommend modifying the default value of this property because that can affect the BMC Cloud Lifecycle Management use cases.
Right-click on the specific USP job from the CSM_Jobs folder and click on Show Results. From the Show Results pane, right click on the job run and select Show Log. Expand the tree downwards and select the host name, as shown below.
For the USP job to complete successfully, the Application Server must be able to reach the target virtual machine by the name used to enroll the machine in BMC Server Automation.
A USP Job can fail for the following reasons:
- Networking or DNS Issues: Target VM is NOT reachable from BMC Server Automation Appserver.
- Insufficient RSCD Permissions - Renaming Administrator account on source template
- OS level firewall settings on the source template
- Custom Object Error - Incorrect version of RSCD agent on the source template
Networking or DNS Issues
Ensure the Application Server can ping the target VM by both host name and IP address. Cross-verify the IP address from the target VM.
Fix any issues at the OS and DNS level to ensure connectivity.
Refer to the KA # KA304875 to learn more about Java DNS caching.
DNS Settings in VGP
BMC Cloud Lifecycle Management 2.x and 3.x do not use the DNS configuration settings from the Virtual Guest Package for template-based provisioning. However, for bare metal provisioning, BMC Cloud Lifecycle Management can use the DNS configuration settings in the Network tab of OS packages created in BMC Server Automation.
Static IP Assignment
When static IP assignment is used for a management NIC, BMC Network Automation only provides the IP address and not the DNS server address to the VMs. Due to this limitation, DNS registration does not occur in the environment. Thus, if the VM is enrolled by host name in BMC Server Automation, the Application Server is not able to resolve the target VM.
Also, in secured DNS environments, hosts cannot update the DNS. In these scenarios, use the following workarounds:
- Use a pre-provisioning BMC Atrium Orchestrator callout (customization) to create an entry in the DNS.
- Register the VMs by IP address rather than name.
This is a system-wide requirement. Any provisioned server should be enrolled by IP Address instead of host name in BMC Server Automation as well as in VMware.
BBSA_REGISTER_SERVERS_BY_HOSTNAME in providers.json on the Platform Manager controls how the VM is registered in BMC Server Automation. By default it is set to false, but the property is not used if DHCP assignment is used for the management NIC. When you start using a static IP, by default BMC Server Automation begins registering the VMs by IP address based on this property.
Creating BMC Atrium Orchestrator callouts involves product customization. This task may require assistance from BMC Professional Services.
Insufficient RSCD Permissions: Renaming Administrator account on source template
To perform certain tasks on the target VM via the RSCD agent, you must have certain permissions on the source template so all VMs that are provisioned from this template get those permissions.
Ensure that the exports, users and users.local files have sufficient permissions configured so that the BMC Server Automation Application Server can connect to the target VM via the BLAdmin user and retrieve server information.
In some scenarios, based on the customer environment, the built-in Administrator user account must be assigned a different name on the source template. In this case, the exports file must be modified accordingly. For example, if the built-in Administrator account was modified to clmAdmin, the exports file should reflect that user name as
The user mapped in the exports file must be present on the target server. However, when the VM is cloned and customized in the Virtual Center, the user account is renamed back to Administrator, thus invalidating the exports file mapping. This issue occurs specifically for Windows 2008 OS on VMWare, and it is by design. In this case, the USP job can never complete successfully. The service offering instance fails after completing all the USP job runs, (60 times by default).
*\[Class=VirtualGuestTaskExecHelper:execute\] - Error creating VG: <hostname>* *java.lang.IllegalStateException: Done running USP jobs. The agent is not responding.*
If there is a requirement to change the built-in Administrator user on the source template, test the scenario first in Virtual Center directly.
Ensure the user account is not renamed again by the Guest OS Customization process. As in case of Windows 2008, you can avoid this by using scripts that are executed after USP job completes.
OS level firewall settings on the source template
In certain scenarios, due to security reasons, restrictions such as OS-level firewall settings are applied to the source template. This may block incoming requests, including ping requests. Due to such restrictions, the USP job may not complete successfully.
Configure the source template so the USP job is not affected by any firewall rules.
Custom Object Error: Incorrect version of RSCD agent on the source template
After the USP job is complete and BMC Cloud Lifecyle Management finds the RSCD agent is responding and licensed, BMC Cloud Lifecyle Management attempts to retrieve the custom object SystemInfo (Hardware Information) from the Application Server.
A mismatch in the RSCD version between the BMC Server Automation Application Server, the VC Server, and the source template could cause the following error:
\[ERROR\] API - \[Class=VirtualGuestTaskExecHelper:execute\] - *Error creating VG: <hostname>* java.lang.IllegalStateException: *Custom Object version was not found in the BBSA response* at com.bmc.cloud.provider.bbsaprovider.task.BaseBBSATaskExecHelper.getAssetCOVersion(BaseBBSATaskExecHelper.java:728)
Every CLM version has a set of certified BMC Server Automation versions. BMC recommends using RSCD versions similar to the BMC Server Automation Application Server version across the BMC Cloud Lifecyle Management environment, including the Virtual Center Server and source VM templates.
Upgrade the RSCD agent of the source VM template if it is not the same version as BMC Server Automation and the VC Server.