Troubleshooting the Application Server

This topic was edited by a BMC Contributor and has not been approved.  More information.

This topic presents information about troubleshooting the BMC Server Automation Application Server.

For additional performance issues, see Application Server performance recommendations.

Troubleshooting Presentation

The following video and presentation file (.ppt) are from a live demo that reviewed the BMC Server Automation Application Server troubleshooting steps, along with best practices. 

Flowchart for Application Server troubleshooting

The following image is a flowchart of how to troubleshoot issues in starting up the Application Server. Troubleshooting issues with an Application Server typically involve these areas:

  • Database connectivity
  • File server issues
  • Keystore issues
  • For File Server issues you might need to troubleshoot the agent.

App Server Troubleshooting

Application Server startup issues can happen in any environment. Remember that it is up to you to prevent Application Server startup issues.

Application Server errors

The following sections discuss several typical errors.

An error occurred that prevented filtering the ids used in an ACL

[22 Oct 2010 13:30:24,007] [Client-Connections-Thread-5] [ERROR] [BLAdmin:BLAdmins:127.0.0.1] [Clien
t] An error occurred that prevented filtering the ids used in an ACL.  The error was com.bladelogic.
app.db.DBException: An error occurred while attempting to access the database:
Message : No entry in mapping files for query with ID SELECT_DELETED_ROLE_IDS_IN_SET SQLState: null
ErrorCode: 0

Solution: Copy the sqlmap.props from _template out to the other deployments. sqlmap.properties usually has those queries: look closely at them and compare to another machine.

A connection cannot be made to the database

An error occurred while attempting to access the database:
Message : Cannot get a connection, pool error Timeout waiting for idle object 
SQLState: null ErrorCode: 0

Explanation: The number of database connections that you set for jobs is divided across two job connection pools, with 75% of the value that you set applied to a primary pool and 25% to a secondary pool that is reserved for nested jobs. For example, if you set the maximum number of job-related connections to a value of 100, the primary pool has 75 connections and the secondary pool has 25 connections.

This means that you might run out of job-related connections sooner than expected. Although you set the maximum number of connections to a value of 100, you might receive an error that no more connections are available when all 75 connections from the primary pool are being used, even though not all 25 connections in the secondary pool are being used.

Solution: Set the number of database connections in the job connection pool to a higher value through the blasadmin utility.

Failed to parse request; error msg = Unknown request id

You may get this message flooding the logs twice every second:

[08 Oct 2010 16:06:20,611] [Client-Connections-Thread-7] [DEBUG] [BLAdmin:BLAdmins:127.0.0.1] [Client] Failed to parse request; error msg = Unknown request id : com.bladelogic.app.api.BlAclService_filterValidIds(java.util.Map)
com.bladelogic.mfw.util.BlException: Failed to parse request; error msg = Unknown request id : com.bladelogic.app.api.BlAclService_filterValidIds(java.util.Map)
                at com.bladelogic.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:79)
                at com.bladelogic.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:370)
                at com.bladelogic.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:280)
                at com.bladelogic.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:180)
                at com.bladelogic.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:99)
                at com.bladelogic.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:25)
                at com.bladelogic.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:92)

Possible Solution: Most likely cause is that your UI and application server run different builds. If you recently upgraded, double-check that all of the UIs were upgraded as well.

Invalid App Servers shown under Infrastructure Management details

After attempting setup of multiple app servers on the same physical host there are now invalid app servers listed under Tools -> Infrastructure Management (formerly known as App Server Details) from either previous AppServerProfiles.xml entries or a previous BladeLogic configuration.

Explanation: The entries are recorded in the database under the database table "application_server."

Solution: Using SQL "select * from application_server" - you'll need to be able to confirm the valid entries by either the app_server_name, display_name, last_start_date, last_alive_date, or the port numbers, clear_port or rmi_port.
The invalid entries can then have the is_deleted value set to 1 with

update application_server set is_deleted=1 where column_name = name;

(followed by "commit;" for Oracle db).

Application Server thread states

Details about the state of threads that are run by the Application Server can help you troubleshoot problems in the Application Server. You can obtain thread details from the following sources:

The following table describes several common thread states and suggests actions to take in the case of problematic states:

Thread stateDescription
RUNNABLE

The thread is actively working.

This is a normal state and it does not indicate a problem.

WAITING

The thread is waiting for another action to complete.

This is a normal state and does not indicate a problem.

BLOCKED

The thread is blocked by another action and cannot resume until that other action completes.

If the thread remains in this state for a long time, then this state indicates a problem. Save a copy of the thread dump and contact BMC Support.

STUCK

The thread is stuck and cannot complete its task within the defined timeout period.

This state indicates a problem. Save a copy of the thread dump and contact BMC Support.

The following video presents a few methods of obtaining the Application Server thread dump, which you can use to troubleshoot performance issues — either by generating an Application Server status report when generating data for support, or by accessing the Infrastructure Management window and copying or exporting the thread data. 

  https://youtu.be/WOuJWZg4-5k

Was this page helpful? Yes No Submitting... Thank you

Comments