Troubleshooting the Application Server
This topic presents information about troubleshooting the BMC Server Automation Application Server.
For additional performance issues, see Application Server performance recommendations.
The following video and presentation file (.ppt) are from a live demo that reviewed the BMC Server Automation Application Server troubleshooting steps, along with best practices.
Flowchart for Application Server troubleshooting
The following image is a flowchart of how to troubleshoot issues in starting up the Application Server. Troubleshooting issues with an Application Server typically involve these areas:
- Database connectivity
- File server issues
- Keystore issues
- For File Server issues you might need to troubleshoot the agent.
Application Server startup issues can happen in any environment. Remember that it is up to you to prevent Application Server startup issues.
Application Server errors
The following sections discuss several typical errors.
An error occurred that prevented filtering the ids used in an ACL
[22 Oct 2010 13:30:24,007] [Client-Connections-Thread-5] [ERROR] [BLAdmin:BLAdmins:127.0.0.1] [Clien t] An error occurred that prevented filtering the ids used in an ACL. The error was com.bladelogic. app.db.DBException: An error occurred while attempting to access the database: Message : No entry in mapping files for query with ID SELECT_DELETED_ROLE_IDS_IN_SET SQLState: null ErrorCode: 0
Solution: Copy the sqlmap.props from _template out to the other deployments. sqlmap.properties usually has those queries: look closely at them and compare to another machine.
A connection cannot be made to the database
An error occurred while attempting to access the database: Message : Cannot get a connection, pool error Timeout waiting for idle object SQLState: null ErrorCode: 0
Explanation: The number of for jobs is divided across two job connection pools, with 75% of the value that you set applied to a primary pool and 25% to a secondary pool that is reserved for nested jobs. For example, if you set the maximum number of job-related connections to a value of 100, the primary pool has 75 connections and the secondary pool has 25 connections.
This means that you might run out of job-related connections sooner than expected. Although you set the maximum number of connections to a value of 100, you might receive an error that no more connections are available when all 75 connections from the primary pool are being used, even though not all 25 connections in the secondary pool are being used.
Solution: Set the number of database connections in the job connection pool to a higher value through the blasadmin utility.
Failed to parse request; error msg = Unknown request id
You may get this message flooding the logs twice every second:
[08 Oct 2010 16:06:20,611] [Client-Connections-Thread-7] [DEBUG] [BLAdmin:BLAdmins:127.0.0.1] [Client] Failed to parse request; error msg = Unknown request id : com.bladelogic.app.api.BlAclService_filterValidIds(java.util.Map) com.bladelogic.mfw.util.BlException: Failed to parse request; error msg = Unknown request id : com.bladelogic.app.api.BlAclService_filterValidIds(java.util.Map) at com.bladelogic.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:79) at com.bladelogic.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:370) at com.bladelogic.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:280) at com.bladelogic.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:180) at com.bladelogic.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:99) at com.bladelogic.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:25) at com.bladelogic.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:92)
Possible Solution: Most likely cause is that your UI and application server run different builds. If you recently upgraded, double-check that all of the UIs were upgraded as well.
Invalid App Servers shown under Infrastructure Management details
After attempting setup of multiple app servers on the same physical host there are now invalid app servers listed under Tools -> Infrastructure Management (formerly known as App Server Details) from either previous AppServerProfiles.xml entries or a previous BladeLogic configuration.
Explanation: The entries are recorded in the database under the database table "application_server."
Solution: Using SQL "select * from application_server" - you'll need to be able to confirm the valid entries by either the app_server_name, display_name, last_start_date, last_alive_date, or the port numbers, clear_port or rmi_port.
The invalid entries can then have the is_deleted value set to 1 with
update application_server set is_deleted=1 where column_name = name;
(followed by "commit;" for Oracle db).
Application Server thread states
Details about the state of threads that are run by the Application Server can help you troubleshoot problems in the Application Server. You can obtain thread details from the following sources:
- Application Server thread dump, which you can obtain through the BLCLI, using the AppServerProfiler getThreadDump command.
- Infrastructure Management window, the Application Servers node. See Getting information about Application Servers.
The following table describes several common thread states and suggests actions to take in the case of problematic states:
The thread is actively working.
This is a normal state and it does not indicate a problem.
The thread is waiting for another action to complete.
This is a normal state and does not indicate a problem.
The thread is blocked by another action and cannot resume until that other action completes.
If the thread remains in this state for a long time, then this state indicates a problem. Save a copy of the thread dump and contact BMC Support.
The thread is stuck and cannot complete its task within the defined timeout period.
This state indicates a problem. Save a copy of the thread dump and contact BMC Support.
The following video presents a few methods of obtaining the Application Server thread dump, which you can use to troubleshoot performance issues — either by generating an Application Server status report when generating data for support, or by accessing the Infrastructure Management window and copying or exporting the thread data.