Troubleshooting Application Server startup issues
A Truesight Server Automation Application Server fails to start. This topic helps you locate and review the Application Server logs to determine the root cause of the problem and either help you identify and resolve the issue or create a BMC Customer Support case.
Issue symptoms
The Application Server fails to start using any of the documented methods. The following symptoms might be observed:
- On a Windows Application Server, the "BladeLogic Application Server" Windows Service fails to start.
- On a Linux Application Server, the systemctl start blappserv command fails with an error.
- The Application Server fails to start when attempted from the "Configuration - Infrastructure Management - Application Servers" node in the TrueSight Server Automation Console.
- The following message is displayed while attempting to connect to the Application Server from the TrueSight Server Automation Console:
"Could not connect to "service:authsvc.bladelogic:blauth://appserver:port"
Issue scope
The problem may affect all or specific Application Servers in your environment.
Diagnosing and reporting an issue
Task | Action | Steps | Reference |
|---|---|---|---|
1 | Understand problem scope | Verify whether the problem affects all the Application Servers or a specific Application Server. This can help determine whether the problem is a global issue (for example, database or File server is down) or is a configuration issue on a specific Application Server. | |
2 | Identify recent changes | Is this the first attempt to start a freshly-installed Application server? Or is this an existing Application Server which was previously running successfully? If this is an existing Application Server that was running successfully, when was it last observed to be running and has anything notable changed in the environment since then? For instance,
This event may not be the root cause of the problem but being aware of this information can be useful while troubleshooting. | |
3 | Identify and locate the Application Server Log files | The following Application Server log files can be used to troubleshoot Application Server startup issues. These log files are located in the installDirectory/br/ directory on the Application Server:
*If you have multiple Application Server deployments, the specified log file names are the log files of the default Application Server deployment. The names of the log files for other Application Server deployments are prefixed with the name of the deployment, connected with an underscore.For example, jobservera_appserver.log, jobserverb_console.log, and so on. | |
4 | Review the AppserverLauncher.log file for errors during startup | Check the AppServerLauncher.log file to see if there are any errors at the Launcher level. A healthy AppServerLauncher.log file contains the entries highlighted in bold, which indicate the successful launching of the Java process(es) for the Application Server(s) on that host. [28 Aug 2017 13:44:40,017] [main] [INFO] [::] [] Starting application server launcher ... When the Application Server Launcher detects that the Application Server did not start correctly, it displays a message similar to that in red below. In this case, AppServerLauncher.log indicates that there was a problem launching the Application Server, but may not contain the root cause. [28 Aug 2017 16:27:52,093] [main] [ERROR] [::] [] Application server myAppServer failed to start. Please check the deployment log for 'default' deployment and the deployment console log file 'console.log' for more details. In some cases, AppServerLauncher.log may contain more specific details about the root cause of the startup failure. For example, Caused by: com.bladelogic.om .infra.app.bladmin.BlAdminException: Error while creating database connection. Please check the following for possible reasons of connection failure : at com.bladelogic.om.infra.app.bladmin.BootstrapDatabaseConnection.getConnection(BootstrapDatabaseConnection.java:87) Verify whether AppServerLauncher.log shows errors, the type of errors, and the date and time when they occurred and then proceed to step 5 to examine the console.log and appserver.log files. | |
5 | Review the console.log and appserver.log files | The next step is to check whether the console.log and appserver.log files were updated when the Application Server startup was attempted. The following entries represent the first three lines of the console.log update corresponding to the Application Server startup. Start reading the file from the beginning of the startup sequence and scroll down until the first error message is displayed. [28 Aug 2017 16:27:49,351] [main] [INFO] [::] [] Starting Agent Service...
| |
6 | When console.log contains error messages related to the failed Application Server startup | If the Application Server is failing to start, and the console.log file was updated with events since the Application Server was started, this file is likely contain the root cause of the issue. [28 Aug 2017 15:31:37,709] [main] [INFO] [::] [] BLManager Version = 8.9.00.138 If the Application Server startup has failed, the console.log should contain errors instead of the specified entries. See the "Resolution for common issues" section for details on specific error messages which may be observed in console.log when the Application Server fails to start. | |
7 | When console.log contains no error messages or events related to the failed Application Server startup | If the Application Server is failing to start but no new entries are written to console.log, or the console.log file does not exist at all, then it indicates that the Application Server startup process is failing early in the startup process and the logging mechanisms are not even being initialized. Examples of such situations could include:
In such a case, verify whether the AppServerLauncher.log file contains any messages that indicate the root cause. If AppServerLauncher.log is not created or updated, it indicates that the problem is happening even before the Launcher's logging has initialized and and the updated AppServerLauncher.log or console.log is not available to troubleshoot. touch AppServerLauncher_console.log chown bladmin:bladmin AppServerLauncher_console.log chmod 755 AppServerLauncher_console.log ii) Restart the Application Service. The updates are available in the AppServerLauncher_console.log file is created. However, no updates are available in the AppServerLauncher.log or console.log. Linux Application Server Test 2: Run the following commands in the order indicated: i) Change to the NSH/br directory. ii) Run this command: su - bladmin iii) Run this command: ./blappserv Note down any error that you might encounter while running commands mentioned in step ii or iii. Windows Application Server: Do the following: The Application Server startup issue might persist, but the console shows the error message that indicates the root cause. | |
8 | Review the specific error message found in Application Server logs | Use the table in the "Resolutions for common issues" section to review common errors that can result in the Application Server startup failure and how they can typically be resolved. If you are unable to identify and resolve the problem, see Step 9 to create a BMC Support Case. | |
9 | Create a BMC Support Case | Provide the following information and log files when creating a case with BMC Customer Support:
|
Resolutions for common issues
Symptom | Action | Reference |
|---|---|---|
The TrueSight Server Automation service installed on SuSE 12 SP5 fails to start, both when triggered manually and after a system reboot. The application server log shows the following error: App server.log or Console.log file content: [69.528s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. [69.607s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. | Indicates an out of memory issue when the application server service is manually started or post reboot. Add the parameter TasksMax=infinity in the configuration file (/etc/system/system/blappser.service) to fix the issue as shown in the following snippet: For more information, see the KA article in the BMC Community page. | |
console.log contains the following error: [28 Aug 2019 15:41:44,742] [main] [ERROR] [::] [] File server validation failed because either the file server is not specified, cannot be contacted, or sufficient permissions are not granted on the file server agent to the appserver. Please verify the connectivity between appserver and file server and ensure that the appserver has read write permissions on the file server. | Indicates a problem communicating with the RSCD agent on the designated File Server host. Follow the referenced KA to troubleshoot and resolve. | |
AppServerLauncher.log or console.log contains the following error: [15 Jun 2020 15:23:24,658] [main] [INFO] [::] [] Starting application server launcher ... | Indicates a problem connecting to the specified database. This problem might occur due to incorrect Application Server configuration or a database issue. Examples of common errors:
Do the following:
If the problem is on a database side, work with the site DBA to resolve the issue. | |
console.log contains the following error: [28 Aug 2017 16:27:51,505] [main] [ERROR] [::] [] Port already in use: 9700; nested exception is: | Indicates that another process on the system is using one of the designated Appserver ports (9700 in this example). Follow the referenced KA to troubleshoot and resolve. | |
console.log, appserver.log and AppServerLauncher.log are empty/not updated but creating an AppServerLaunch_console.log file (step 7 above) displays the following error from the attempted startup: [root]# more AppServerLauncher_console.log /opt/bmc/bladelogic/NSH/br/blappserv: line 27: /opt/bmc/bladelogic/NSH/br/javaxx/bin/java: No such file or directory | Indicates problem finding Java on the Application Server.
| |
console.log, appserver.log and AppServerLauncher.log are empty/not updated but creating an AppServerLaunch_console.log file (step 7 above) displays the following error from the attempted startup: [root]# more AppServerLauncher_console.log Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object | Indicates problem loading the main Java class object on the Application. This suggests there is a problem (corruption, file missing) with the following file: <AppserverInstallDir>/NSH/br/java/lib/rt.jar Check for the presence of this file on the Application Server where this issue is occurring. If missing, restore from the backup or copy from the working Application Server of the same version. | |
During disaster recovery, if the hostname of the Application Server or PXE Server changes (for example, from the Fully-Qualified Domain Name to short name), the Application Server or PXE Server service fails to start on the disaster recovery site with the following error message: Please check if property value is set properly or _pxe deployment exists against hostname: <short hostname> | If there is a modification to the Application Server or PXE server name (for example, from the Fully-Qualified Domain Name to short name), this will be treated as the introduction of a new server instance. Therefore, during Disaster Recovery (DR) operations, it is crucial to cross-verify the host names of both the Application Server and PXE server against those stored in the database, ensuring these names are exactly the same. Follow the referenced KA to troubleshoot and resolve. |