Troubleshooting Application Server startup issues

A Truesight Server Automation Application Server fails to start. This topic helps you locate and review the Application Server logs to determine the root cause of the problem and either help you identify and resolve the issue or create a BMC Customer Support case.

Issue symptoms

The Application Server fails to start using any of the documented methods. The following symptoms might be observed:

  • On a Windows Application Server, the "BladeLogic Application Server" Windows Service fails to start.
  • On a Linux Application Server, the /etc/init.d/blappserv start command fails with an error.
  • The Application Server fails to start when attempted from the "Configuration - Infrastructure Management - Application Servers" node in the TrueSight Server Automation Console.
  • The following message is displayed while attempting to connect to the Application Server from the TrueSight Server Automation Console:
    "Could not connect to "service:authsvc.bladelogic:blauth://appserver:port"

Issue scope

The problem may affect all or specific Application Servers in your environment.

Diagnosing and reporting an issue

Task

Action

Steps

Reference

1

Understand problem scope

Verify whether the problem affects all the Application Servers or a specific Application Server. This can help determine whether the problem is a global issue (for example, database or File server is down) or is a configuration issue on a specific Application Server.


2

Identify recent changes

Is this the first attempt to start a freshly-installed Application server? Or is this an existing Application Server which was previously running successfully?

If this is an existing Application Server that was running successfully, when was it last observed to be running and has anything notable changed in the environment since then? For instance,

  • The database user password was changed.
  • The database was moved to a different host.     
  • The File Server was moved to a different host.     
  • TrueSight Server Automation was upgraded to a new version.
  • OS patches were applied to the Application Server or File Server host.
  • The Application Server host was rebooted.

This event may not be the root cause of the problem but being aware of this information can be useful while troubleshooting.


3

Identify and locate the Application Server Log files

The following Application Server log files can be used to troubleshoot Application Server startup issues. These log files are located in the installDirectory/br/ directory on the Application Server: 

  • AppServerLauncher.log
  • appserver.log*    
  • console.log*

*If you have multiple Application Server deployments, the specified log file names are the log files of the default Application Server deployment. The names of the log files for other Application Server deployments are prefixed with the name of the deployment, connected with an underscore.For example, jobservera_appserver.log, jobserverb_console.log, and so on.

4

Review the AppserverLauncher.log file for errors during startup

Check the AppServerLauncher.log file to see if there are any errors at the Launcher level.

A healthy AppServerLauncher.log file contains the entries highlighted in green, which indicate the successful launching of the Java process(es) for the Application Server(s) on that host.

[28 Aug 2017 13:44:40,017] [main] [INFO] [::] [] Starting application server launcher ...
[28 Aug 2017 13:44:42,887] [main] [INFO] [::] [] Starting RMI registry on host: clm-aus-019331
[28 Aug 2017 13:44:42,943] [main] [INFO] [::] [] Setting client socket factory: com.bladelogic.om.infra.app.service.client.ClientSocketFactory
[28 Aug 2017 13:44:42,943] [main] [INFO] [::] [] Setting server socket factory: com.bladelogic.om.infra.app.service.client.ServerSocketFactory
[28 Aug 2017 13:44:42,943] [main] [INFO] [::] [] Setting authenticator: com.bladelogic.om.infra.app.service.manage.auth.BlSessionServerAuthenticator
[28 Aug 2017 13:44:42,944] [main] [INFO] [::] [] Starting Management Service ...
[28 Aug 2017 13:44:43,104] [main] [INFO] [::] [] Management Service started.
[28 Aug 2017 13:44:43,104] [main] [INFO] [::] [] Setting up app server message listener on port: 9701
[28 Aug 2017 13:44:44,378] [main] [INFO] [::] [] Updating application server profile 'clm-aus-019331'
[28 Aug 2017 13:44:44,378] [main] [INFO] [::] [] No changes made to application server profile 'clm-aus-019331'
[28 Aug 2017 13:44:44,380] [main] [INFO] [::] [] Starting Application Server:myAppserver with arguments: [/opt/bmc/bladelogic/NSH/br/java/bin/java, -Djava.library.path=/opt/bmc/bladelogic/NSH/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib, -Djava.security.egd=file:/dev/../dev/urandom, -Djava.io.tmpdir=/opt/bmc/bladelogic/NSH/tmp, -Djava.class.path=/opt/bmc/bladelogic/NSH/br:< many jar file paths here>, -Xss1m, -Xmx954728448, -D
blx.rootdir=/opt/bmc/bladelogic/NSH, -Dblx.cmrootdir=/opt/bmc/bladelogic/NSH/br, -Dblx.hostname=clm-aus-019331, com.bladelogic.om.infra.mfw.fw.BlManager,myAppserver
,myAppserver, true, ALL, AppSvcPort=9841, AuthSvcPort=9840, CLRProxyPort=0, ConsoleLogfileName=console.log, HttpsPortNumber=9843, JMXManagementPort=9838, LogfileNam
e=appserver.log, MaxJobs=20, MaxPort=9899, MaxWorkItemThreads=50, MinPort=9850, ProxySvcPort=0, RegistryPort=9836, SSLPort=0, SocketsBindAddress=all, SqlFiles=sql%2Fsql
map.properties%2Csql%2Fstreamable_sqlmap.properties, TempDirectoryName=application_server, LauncherIncomingMessagePort=9701, LauncherRegistryPort=9700]
[22 Aug 2017 16:04:12,513] [Thread-4] [INFO] [::] [] Application server myAppserver has started successfully
[22 Aug 2017 16:04:12,806] [main] [INFO] [::] [] Registering shutdown hook
[22 Aug 2017 16:04:12,807] [main] [INFO] [::] [] Application server launcher started
[22 Aug 2017 16:04:12,807] [main] [INFO] [::] [] Ready

When the Application Server Launcher detects that the Application Server did not start correctly, it displays a message similar to that in red below.

In this case, AppServerLauncher.log indicates that there was a problem launching the Application Server, but may not contain the root cause.

[28 Aug 2017 16:27:52,093] [main] [ERROR] [::] [] Application server myAppServer failed to start. Please check the deployment log for 'default' deployment and the deployment console log file 'console.log' for more details.
com.bladelogic.om.infra.mfw.util.BlException: Application server myAppServer failed to start. Please check the deployment log for 'default' deployment and the deployment console log file 'console.log' for more details.
at com.bladelogic.om.infra.app.profile.AppServerLauncher.checkServer(AppServerLauncher.java:1060)
at com.bladelogic.om.infra.app.profile.AppServerLauncher.startAppServersByProfile(AppServerLauncher.java:980)
at com.bladelogic.om.infra.app.profile.AppServerLauncher.launchApplicationServers(AppServerLauncher.java:1327)
at com.bladelogic.om.infra.app.profile.AppServerLauncher.start(AppServerLauncher.java:220)
at com.bladelogic.om.infra.app.profile.AppServerLauncher.main(AppServerLauncher.java:1530)

In some cases, AppServerLauncher.log may contain more specific details about the root cause of the startup failure. For example,

Caused by: com.bladelogic.om.infra.app.bladmin.BlAdminException: Error while creating database connection. Please check the following for possible reasons of connection  failure :
 1. Authentication failure
 2. Database configuration like uri are incorrectly configured
 3. Network connectivity issues between this application server and the database
        at com.bladelogic.om.infra.app.bladmin.BootstrapDatabaseConnection.getConnection(BootstrapDatabaseConnection.java:87)
        at com.bladelogic.om.infra.app.profile.AppServerLauncher.getConfigTimePropertiesValue(AppServerLauncher.java:378)
        ... 5 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'sa'. ClientConnectionId:9bbe8340-c23a-4e7b-9559-6905d4353c71

Verify whether AppServerLauncher.log shows errors, the type of errors, and the date and time when they occurred and then proceed to step 5 to examine the console.log and appserver.log files.

5

Review the console.log and appserver.log files

The next step is to check whether the console.log and appserver.log files were updated when the Application Server startup was attempted.

The following entries represent the first three lines of the console.log update corresponding to the Application Server startup. Start reading the file from the beginning of the startup sequence and scroll down until the first error message is displayed.

[28 Aug 2017 16:27:49,351] [main] [INFO] [::] [] Starting Agent Service...
[28 Aug 2017 16:27:49,355] [main] [INFO] [::] [] Agent Service started.
[28 Aug 2017 16:27:49,496] [main] [INFO] [::] [] String ClientValidation Service...

  • If the console.log and appserver log are updated with log entries since the Application Server startup failure, review these log files. The contents of the console.log file are usually a super-set of the appserver.log contents (console.log = appserer.log + standard error/output). Therefore, focus on the log entries in the console.log for startup issues. Proceed to step 6 for reviewing console.log.
  • If the console.log and appserver.log are not updated at all during the failed Application Server startup, it indicates that the Application Server startup attempt is failing early in the process and that the logging mechanisms have not even been initialized. In this case, skip to step 7.


6

When console.log contains error messages related to the failed Application Server startup

If the Application Server is failing to start, and the console.log file was updated with events since the Application Server was started, this file is likely contain the root cause of the issue.

A successful Application Server startup results in the following log file entries in console.log:

[28 Aug 2017 15:31:37,709] [main] [INFO] [::] [] BLManager Version = 8.9.00.138
..........
BLManager Current Time = Mon Aug 28 15:31:37 CDT 2017
BLManager Started = Mon Aug 28 15:31:10 CDT 2017
BLManager PID = 25068
[28 Aug 2017 15:31:37,711] [main] [INFO] [::] [] Accepting requests...
[28 Aug 2017 15:31:37,711] [main] [INFO] [::] [] Ready.
[28 Aug 2017 15:31:37,711] [main] [INFO] [::] [] Performing post-startup deployment operations...

If the Application Server startup has failed, the console.log should contain errors instead of the specified entries.

See the "Resolution for common issues" section for details on specific error messages which may be observed in console.log when the Application Server fails to start.


7

When console.log contains no error messages or events related to the failed Application Server startup

If the Application Server is failing to start but no new entries are written to console.log, or the console.log file does not exist at all, then it indicates that the Application Server startup process is failing early in the startup process and the logging mechanisms are not even being initialized. Examples of such situations could include: 

  • Database connectivity issues
  • Problems locating or launching Java

In such a case, verify whether the AppServerLauncher.log file contains any messages that indicate the root cause.

If AppServerLauncher.log is not created or updated, it indicates that the problem is happening even before the Launcher's logging has initialized and and the updated AppServerLauncher.log or console.log is not available to troubleshoot.

In this rare case, you can use the following approach depending on whether the Application Server is running on Linux or Windows.

Linux Application Server Test 1:

i) Create the following file in the NSH/br directory and set the ownership and permissions as specified:

touch AppServerLauncher_console.log
chown bladmin:bladmin AppServerLauncher_console.log
chmod 755 AppServerLauncher_console.log

ii) Restart the Application Service.

The updates are available in the AppServerLauncher_console.log file is created. However, no updates are available in the AppServerLauncher.log or console.log.
The following example shows that there is a problem locating Java on this Application Server:
more AppServerLauncher_console.log
/opt/bmc/bladelogic/NSH/br/blappserv: line 27: /opt/bmc/bladelogic/NSH/br/java/bin/java: No such file or directory

Linux Application Server Test 2:

Run the following commands in the order indicated:

i) Change to the NSH/br directory.

ii) Run this command: su - bladmin

iii) Run this command: ./blappserv

Note down any error that you might encounter while running commands mentioned in step ii or iii.

Windows Application Server:

Do the following:
i) Launch the Windows command prompt as an administrator.
ii) Change to the Application Server NSH/bin directory.
iii) Run the following command:  appsvc.exe

The Application Server startup issue might persist, but the console shows the error message that indicates the root cause.


8

Review the specific error message found in Application Server logs

Use the table in the "Resolutions for common issues" section to review common errors that can result in the Application Server startup failure and how they can typically be resolved.

If you are unable to identify and resolve the problem, see Step 9 to create a BMC Support Case.


9

Create a BMC Support Case

Provide the following information and log files when creating a case with BMC Customer Support:

  • Scope of the issue as identified in step 1
  • Recent environment changes as identified in step 2
  • The Application Server OS vendor and version
  • The Application Server product version
  • AppserverLauncher.log
  • appserver.log for the Application Server Deployments(s) that are failing to start (may have different name as discussed in step 3)
  • console.log for the Application Server Deployments(s) that are failing to start (may have different name as discussed in step 3)


Resolutions for common issues

Symptom

Action

Reference

console.log contains the following error:

[28 Aug 2019 15:41:44,742] [main] [ERROR] [::] [] File server validation failed because either the file server is not specified, cannot be contacted, or sufficient permissions are not granted on the file server agent to the appserver. Please verify the connectivity between appserver and file server and ensure that the appserver has read write permissions on the file server.

Indicates a problem communicating with the RSCD agent on the designated File Server host. Follow the referenced KA to troubleshoot and resolve.

AppServerLauncher.log or console.log contains the following error:

[15 Jun 2020 15:23:24,658] [main] [INFO] [::] [] Starting application server launcher ...
[15 Jun 2020 15:23:30,996] [main] [ERROR] [::] [] Error while creating database connection. Please check the following for possible reasons of connection failure :
 1. Authentication failure
 2. Database configuration like uri are incorrectly configured
 3. Network connectivity issues between this application server and the database

Indicates a problem connecting to the specified database. This problem might occur due to incorrect Application Server configuration or a database issue.

Examples of common errors:

  • Caused by: java.sql.SQLException: ORA-01017: invalid username/password; logon denied
  • Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
  • ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
  • Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)

Do the following:

  1. Review the Database connection information via blasadmin:
    blasadmin -a show database connection

    blasadmin -a show database user

  2. If needed, update the Database connection information via blasadmin:
    blasadmin -a set database conn XXXXX

    blasadmin -a set database user bladelogic
    blasadmin -a set database password mypassword

  3. Review the database connection information in the installDirectory/br/deployments/global.properties file to ensure it is set as expected. If not, update the global.properties file and retest the startup:
    # pwd
    /opt/bmc/bladelogic/NSH/br/deployments

    # more global.properties

    #
    #Thu Jun 18 09:12:12 CDT 2020
    user=BLADELOGIC
    url=jdbc\:oracle\:thin\:@DBServer1\:1521\:ORA12C
    driver=oracle.jdbc.driver.OracleDriver

If the problem is on a database side, work with the site DBA to resolve the issue.

console.log contains the following error:

[28 Aug 2017 16:27:51,505] [main] [ERROR] [::] [] Port already in use: 9700; nested exception is:
java.net.BindException: Address already in use

Indicates that another process on the system is using one of the designated Appserver ports (9700 in this example).

Follow the referenced KA to troubleshoot and resolve.


console.log, appserver.log and AppServerLauncher.log are empty/not updated but creating an AppServerLaunch_console.log file (step 7 above) displays the following error from the attempted startup:

[root]# more AppServerLauncher_console.log
/opt/bmc/bladelogic/NSH/br/blappserv: line 27: /opt/bmc/bladelogic/NSH/br/javaxx/bin/java: No such file or directory

Indicates problem finding Java on the Application Server.

  • Review the location of Java in the error message to make sure it is located in the specified directory.
  • On Linux Application Servers, review the JAVA_HOME variable in the NSH/br/blappserv script.


console.log, appserver.log and AppServerLauncher.log are empty/not updated but creating an AppServerLaunch_console.log file (step 7 above) displays the following error from the attempted startup:

[root]# more AppServerLauncher_console.log

Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object

Indicates problem loading the main Java class object on the Application.

This suggests there is a problem (corruption, file missing) with the following file:

<AppserverInstallDir>/NSH/br/java/lib/rt.jar

Check for the presence of this file on the Application Server where this issue is occurring.

If missing, restore from the backup or copy from the working Application Server of the same version.


During disaster recovery, if the hostname of the Application Server or PXE Server changes (for example, from the Fully-Qualified Domain Name to short name), the Application Server or PXE Server service fails to start on the disaster recovery site with the following error message:

Please check if property value is set properly or _pxe deployment exists against hostname: <short hostname>

If there is a modification to the Application Server or PXE server name (for example, from the Fully-Qualified Domain Name to short name), this will be treated as the introduction of a new server instance.

Therefore, during Disaster Recovery (DR) operations, it is crucial to cross-verify the host names of both the Application Server and PXE server against those stored in the database, ensuring these names are exactly the same.

Follow the referenced KA to troubleshoot and resolve.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*