Troubleshooting the Application Server

This topic presents information about troubleshooting the BMC Server Automation Application Server.

For additional performance issues, see Application Server performance recommendations.

Troubleshooting Presentation

The following video and presentation file (.ppt) are from a live demo that reviewed the BMC Server Automation Application Server troubleshooting steps, along with best practices. 

Flowchart

The following image is a flowchart of how to troubleshoot issues in starting up the Application Server. Troubleshooting issues with an Application server typically involve these areas:

• Database connectivity
• File server issues
• Keystore issues
• For File Server issues you might need to troubleshoot the agent.

Application server startup issues can happen in any environment. Remember that it is up to you to prevent application server startup issues.

Errors

The following sections discuss several typical errors.

An error occurred that prevented filtering the ids used in an ACL

[22 Oct 2010 13:30:24,007] [Client-Connections-Thread-5] [ERROR] [BLAdmin:BLAdmins:127.0.0.1] [Clien
t] An error occurred that prevented filtering the ids used in an ACL.  The error was com.bladelogic.
app.db.DBException: An error occurred while attempting to access the database:
Message : No entry in mapping files for query with ID SELECT_DELETED_ROLE_IDS_IN_SET SQLState: null
ErrorCode: 0

Solution: Copy the sqlmap.props from _template out to the other deployments. sqlmap.properties usually has those queries: look closely at them and compare to another machine.

A connection cannot be made to the database

An error occurred while attempting to access the database:
Message : Cannot get a connection, pool error Timeout waiting for idle object 
SQLState: null ErrorCode: 0

Explanation: The number of database connections that you set for jobs is divided across two job connection pools, with 75% of the value that you set applied to a primary pool and 25% to a secondary pool that is reserved for nested jobs. For example, if you set the maximum number of job-related connections to a value of 100, the primary pool has 75 connections and the secondary pool has 25 connections.

This means that you might run out of job-related connections sooner than expected. Although you set the maximum number of connections to a value of 100, you might receive an error that no more connections are available when all 75 connections from the primary pool are being used, even though not all 25 connections in the secondary pool are being used.

Solution: Set the number of database connections in the job connection pool to a higher value through the blasadmin utility.

Failed to parse request; error msg = Unknown request id

You may get this message flooding the logs twice every second:

[08 Oct 2010 16:06:20,611] [Client-Connections-Thread-7] [DEBUG] [BLAdmin:BLAdmins:127.0.0.1] [Client] Failed to parse request; error msg = Unknown request id : com.bladelogic.app.api.BlAclService_filterValidIds(java.util.Map)
com.bladelogic.mfw.util.BlException: Failed to parse request; error msg = Unknown request id : com.bladelogic.app.api.BlAclService_filterValidIds(java.util.Map)
                at com.bladelogic.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:79)
                at com.bladelogic.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:370)
                at com.bladelogic.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:280)
                at com.bladelogic.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:180)
                at com.bladelogic.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:99)
                at com.bladelogic.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:25)
                at com.bladelogic.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:92)

Possible Solution: Most likely cause is that your UI and application server run different builds. If you recently upgraded, double-check that all of the UIs were upgraded as well.

Invalid App Servers shown under Infrastructure Management details

After attempting setup of multiple app servers on the same physical host there are now invalid app servers listed under Tools -> Infrastructure Management (formerly known as App Server Details) from either previous AppServerProfiles.xml entries or a previous BladeLogic configuration.

Explanation: The entries are recorded in the database under the database table "application_server."

Solution: Using SQL "select * from application_server" - you'll need to be able to confirm the valid entries by either the app_server_name, display_name, last_start_date, last_alive_date, or the port numbers, clear_port or rmi_port.
The invalid entries can then have the is_deleted value set to 1 with

update application_server set is_deleted=1 where column_name = name;

(followed by "commit;" for Oracle db).

Application Server thread states

Details about the state of threads that are run by the Application Server can help you troubleshoot problems in the Application Server. You can obtain thread details from the following sources:

• Application Server thread dump, which you can obtain through the BLCLI, using the AppServerProfiler getThreadDump command.
• Infrastructure Management window, the Application Servers node. See Getting information about Application Servers.

The following table describes several common thread states and suggests actions to take in the case of problematic states: