Default language.

Troubleshooting issues when the AR System server does not start


Use this information to obtain the appropriate logging and troubleshooting steps when the does not start.

Related topic

Symptoms

The  does not start.


Scope

  • Fresh deployment or upgrade of 
  • Patch deployment
  • Scheduled restart of the 
  • Any troubleshooting step that requires the  restart


Resolution

Perform the following steps to troubleshoot the issue:

Step

Action

Description

1

Collect  logs

Collect logs from the following files:

  • arerror.log
  • armonitor.log
  • ardebug.log
  • ccsd_config.log
  • ccs_server.log
  • arstartup_trace.log

For information about collecting these logs, see the knowledge article on the BMC community page How to Use the AR System Management Console to View Log Files in a Server Group.


2

Review the armonitor.log file

In the armonitor.log file, check the following tags:

  • Processes Started
  • Processes failed to Start

In a functional environment, the Processes Started tag contains the names of all processes as shown in the example below:

Processes Started :
[ ARSERVER BMC:DefaultJavaPluginServer
BMC:DSOJServer BMC:NativePluginServer
BMC:ARDispatcher BMC:FtsJavaPluginServer
BMC:CarteServer BMC:CMDBDispatcher
BMC:ReconcilationEngine BMC:NormalizationEngine   BMC:AtriumSharedPluginServer  
BMC:SLMBRService BMC:SLMCollectorService
BMC:FileDeployer ].

A process that fails is not included in the Processes Started tag. The failed process is included in the Processes failed to Start list.

In a functional environment, when there are no processes that failed, the Processes failed to Start list is empty, as shown in the example below:

Processes failed to Start : [  ]

When a process fails to start, review the list and identify the failed process that is related to ARSERVER.

In some cases, the  might take more time to start. 

When the  does not start, review the following lines:

"Process [ ARSERVER ]
stopped with exit value"

Also, review some of the following useful keywords:

  • ERROR (552):
    The SQL database operation failed.
  • IO Error:
    The Network Adapter could not establish the connection
  • org.eclipse.persistence.exceptions.DatabaseException

Use the table to analyze the specific errors. If there are no specific errors, continue to perform the next steps.


3

Review the arerror.log file

Review the arerror.log file, located in the ARSystemInstallDir\Arserver\Db\ folder and analyze the approximate time of the last restart of the .

Identify the error messages that explain why the  did not start.

Use the table to analyze the specific errors.


4

Review the ardebug.log file

In the ardebug.log file, located in the ARSystemInstallDir\Arserver\Db\ folder, search for any error message that explain why the  did not start.

Use the table to analyze the specific errors.


5.

Review the Centralized Configuration logs

The Centralized Configuration logs capture the activity of connecting the  and loading the Centralized Configuration data.

These logs mainly include SQL statements. This also helps in resolving the database issues. The following example shows the CCS_Config.log file:

INFO  c.b.a.c.s.i.ConfigurationImpl
THREAD [42]:-
Loading local database configuration.
INFO  c.b.a.c.s.i.ARDatabaseConfiguration
THREAD [42]:-
Loading configurations for query
- SELECT C3204, C3205, C3210 FROM T25 WHERE  C3207 = 'CPGAA5V0FHYCOAR92TLRR83ETCEQZE'
INFO  c.b.a.c.s.i.
ConfigurationImpl THREAD [42]:
- Loading global database configuration.
INFO  c.b.a.c.s.i.ARDatabaseConfiguration
THREAD [42]:-
Loading configurations for query
- SELECT C3204, C3205, C3210
FROM T25 WHERE  C3207
= 'AGGAA5V0FHYCOAQIS9K8QHU9EJO5BR'
INFO  c.b.a.c.s.i.ConfigurationImpl
THREAD [42]:-
Loading shared configuration.

The following example shows the CCS_Server.log file:

INFO  c.b.a.s.d.r.i.
ConfigurationRepositoryImpl
This server is running in normal mode.
INFO  c.b.a.s.d.r.i.
ConfigurationRepositoryImpl
The database connection pool
size is configured to 118

Review the error messages, especially related to the database and loading configurations.


6

Review the arstartup_trace-<date/time>.log file

In the arstartup_trace-<date/time>.log file, locate the log that matches the time when the did not start.

This log file shows the successful deployment.

When an error occurs, you might not see the error message. However, you will also not see the next stage complete.

In the following example, the  tries connecting to the database but the attempt is not successful:

RestApiActivator:starting bundle...
Registering Restful servlet container...
Registered Restful servlet
container with the base URI as "/api"
Registering Restful SSO Filter...
Registered SSO Filter for
all API and applications
Restapi bundle started.
<STARTUP> <0000000039>
/* Tue Oct 25 2022 09:59:28.7270 */
Initializing datasource
and jdbc connection.
<STARTUP> <0000000039>
/* Tue Oct 25 2022 09:59:28.7830 */
Establish datasource session variables.
<STARTUP> <0000000039>
/* Tue Oct 25 2022 09:59:28.7880 */
ALTER SESSION SET NLS_COMP=BINARY
<STARTUP> <0000000039>
/* Tue Oct 25 2022 09:59:28.7880 */
ALTER SESSION SET NLS_SORT=BINARY
<STARTUP> <0000000039>
/* Tue Oct 25 2022 09:59:28.7880 */
ALTER SESSION SET EVENTS
'60025 TRACE NAME CONTEXT FOREVER'
<STARTUP> <0000000039>
/* Tue Oct 25 2022 09:59:28.7890 */
ALTER SESSION SET DDL_LOCK_TIMEOUT=1800
UnRegistering Restful servlet container...
Restful Servlet Container unregistered...

<end of log>

The following example shows a successful  startup:

RestApiActivator:starting bundle...
Registering Restful servlet container...
Registered Restful servlet
container with the base URI as "/api"
Registering Restful SSO Filter...
Registered SSO Filter for
all API and applications
Restapi bundle started.
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:56.3630 */
Initializing datasource and jdbc connection.
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:56.4120 */
Establish datasource session variables.
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:56.4120 */
ALTER SESSION SET NLS_COMP=BINARY
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:56.4120 */
ALTER SESSION SET NLS_SORT=BINARY
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:56.4120 */
ALTER SESSION SET EVENTS
'60025 TRACE NAME CONTEXT FOREVER'
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:56.4120 */
ALTER SESSION SET DDL_LOCK_TIMEOUT=1800
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:59.8420 */
Entity manager initialized.
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:59.8690 */
Install dir =
C:\Program Files\BMC Software\ARSystem
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:58:59.9610 */
Initializing TimeStamp Manager.
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:59:00.2030 */
Database charset = UTF-8
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:59:00.2050 */
Loading the AR Server configuration
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:59:01.1600 */
Language = en;UTF-8
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:59:04.0080 */
Escalation scheduler running.
<STARTUP> <0000000042>
/* Fri Oct 07 2022 07:59:05.8290 */
Initializing Access Control
Interceptor Aspects.

Analyze the logs and identify errors when the  does not start successfully.


7

Review the API and SQL logs

Enable the API and SQL logs and restart the .

  • Review API logs
    API logs capture the activity when the  starts and accepts connections. This log helps determine if the  has started or not.
  • Review SQL logs
    SQL logs show all the SQLs that were used to load the required data dictionary definitions. Analyze and review the errors to identify the issue. 

8

Determine a solution

  • Use the table to analyze the specific errors and troubleshoot the issues.
  • If you cannot identify the cause, collect the logs and raise a case with BMC Support

9

Collect logs

Rename the zipped file to include the server name.

10

Analyze logs

Review the logs and identify error messages or behavior. Also, use the  table to analyze possible symptoms and solutions.


11

Create a support ticket

Collect and send the following information when you create a case with BMC Support:

  • User(s) that experienced the problem (if not all users)
  • Approximate timestamp when the problem was observed
  • What happened prior to the startup issue? 
  • Reason for the server restart.
  • Any recent changes to the application, database, or other components of your environment.

For more information, see the knowledge article on BMC communities SFT - Steps to send logs, files, screenshots, etc to BMC Support for a Product related case.



After you determine a specific symptom or error message, use the following table to identify the solution:

Error message

Location

Action

Reference

Failed to fetch database name.

armonitor.log

Make sure that the database is active and accessible.

IO Error: 
The network adapter could not establish the connection

armonitor.log

Make sure that you can connect to the database with a non-AR Server connection.

org.eclipse.persistence.
exceptions.DatabaseException

armonitor.log

Make sure that the database is active and accessible.

Error encountered in bundle: com.bmc.arsys.messaging  

arerror.log

Delete bundle cache.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*