Troubleshooting issues when the AR System server does not start
Symptoms
The does not start.
Scope
- Fresh deployment or upgrade of
- Patch deployment
- Scheduled restart of the
- Any troubleshooting step that requires the restart
Resolution
Perform the following steps to troubleshoot the issue:
Step | Action | Description |
1 | Collect logs | Collect logs from the following files:
For information about collecting these logs, see the knowledge article on the BMC community page How to Use the AR System Management Console to View Log Files in a Server Group. |
2 | Review the armonitor.log file | In the armonitor.log file, check the following tags:
In a functional environment, the Processes Started tag contains the names of all processes as shown in the example below: Processes Started : [ ARSERVER BMC:DefaultJavaPluginServer BMC:DSOJServer BMC:NativePluginServer BMC:ARDispatcher BMC:FtsJavaPluginServer BMC:CarteServer BMC:CMDBDispatcher BMC:ReconcilationEngine BMC:NormalizationEngine BMC:AtriumSharedPluginServer BMC:SLMBRService BMC:SLMCollectorService BMC:FileDeployer ]. A process that fails is not included in the Processes Started tag. The failed process is included in the Processes failed to Start list. In a functional environment, when there are no processes that failed, the Processes failed to Start list is empty, as shown in the example below: Processes failed to Start : [ ] When a process fails to start, review the list and identify the failed process that is related to ARSERVER. In some cases, the might take more time to start. When the does not start, review the following lines: "Process [ ARSERVER ] stopped with exit value" Also, review some of the following useful keywords:
Use the table to analyze the specific errors. If there are no specific errors, continue to perform the next steps. |
3 | Review the arerror.log file | Review the arerror.log file, located in the ARSystemInstallDir\Arserver\Db\ folder and analyze the approximate time of the last restart of the . Identify the error messages that explain why the did not start. Use the table to analyze the specific errors. |
4 | Review the ardebug.log file | In the ardebug.log file, located in the ARSystemInstallDir\Arserver\Db\ folder, search for any error message that explain why the did not start. Use the table to analyze the specific errors. |
5. | Review the Centralized Configuration logs | The Centralized Configuration logs capture the activity of connecting the and loading the Centralized Configuration data. These logs mainly include SQL statements. This also helps in resolving the database issues. The following example shows the CCS_Config.log file: INFO c.b.a.c.s.i.ConfigurationImpl THREAD [42]:- Loading local database configuration. INFO c.b.a.c.s.i.ARDatabaseConfiguration THREAD [42]:- Loading configurations for query - SELECT C3204, C3205, C3210 FROM T25 WHERE C3207 = 'CPGAA5V0FHYCOAR92TLRR83ETCEQZE' INFO c.b.a.c.s.i. ConfigurationImpl THREAD [42]: - Loading global database configuration. INFO c.b.a.c.s.i.ARDatabaseConfiguration THREAD [42]:- Loading configurations for query - SELECT C3204, C3205, C3210 FROM T25 WHERE C3207 = 'AGGAA5V0FHYCOAQIS9K8QHU9EJO5BR' INFO c.b.a.c.s.i.ConfigurationImpl THREAD [42]:- Loading shared configuration. The following example shows the CCS_Server.log file: INFO c.b.a.s.d.r.i. ConfigurationRepositoryImpl This server is running in normal mode. INFO c.b.a.s.d.r.i. ConfigurationRepositoryImpl The database connection pool size is configured to 118 Review the error messages, especially related to the database and loading configurations. |
6 | Review the arstartup_trace-<date/time>.log file | In the arstartup_trace-<date/time>.log file, locate the log that matches the time when the did not start. This log file shows the successful deployment. When an error occurs, you might not see the error message. However, you will also not see the next stage complete. In the following example, the tries connecting to the database but the attempt is not successful: RestApiActivator:starting bundle... Registering Restful servlet container... Registered Restful servlet container with the base URI as "/api" Registering Restful SSO Filter... Registered SSO Filter for all API and applications Restapi bundle started. <STARTUP> <0000000039> /* Tue Oct 25 2022 09:59:28.7270 */ Initializing datasource and jdbc connection. <STARTUP> <0000000039> /* Tue Oct 25 2022 09:59:28.7830 */ Establish datasource session variables. <STARTUP> <0000000039> /* Tue Oct 25 2022 09:59:28.7880 */ ALTER SESSION SET NLS_COMP=BINARY <STARTUP> <0000000039> /* Tue Oct 25 2022 09:59:28.7880 */ ALTER SESSION SET NLS_SORT=BINARY <STARTUP> <0000000039> /* Tue Oct 25 2022 09:59:28.7880 */ ALTER SESSION SET EVENTS '60025 TRACE NAME CONTEXT FOREVER' <STARTUP> <0000000039> /* Tue Oct 25 2022 09:59:28.7890 */ ALTER SESSION SET DDL_LOCK_TIMEOUT=1800 UnRegistering Restful servlet container... Restful Servlet Container unregistered... <end of log> The following example shows a successful startup: RestApiActivator:starting bundle... Registering Restful servlet container... Registered Restful servlet container with the base URI as "/api" Registering Restful SSO Filter... Registered SSO Filter for all API and applications Restapi bundle started. <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:56.3630 */ Initializing datasource and jdbc connection. <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:56.4120 */ Establish datasource session variables. <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:56.4120 */ ALTER SESSION SET NLS_COMP=BINARY <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:56.4120 */ ALTER SESSION SET NLS_SORT=BINARY <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:56.4120 */ ALTER SESSION SET EVENTS '60025 TRACE NAME CONTEXT FOREVER' <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:56.4120 */ ALTER SESSION SET DDL_LOCK_TIMEOUT=1800 <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:59.8420 */ Entity manager initialized. <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:59.8690 */ Install dir = C:\Program Files\BMC Software\ARSystem <STARTUP> <0000000042> /* Fri Oct 07 2022 07:58:59.9610 */ Initializing TimeStamp Manager. <STARTUP> <0000000042> /* Fri Oct 07 2022 07:59:00.2030 */ Database charset = UTF-8 <STARTUP> <0000000042> /* Fri Oct 07 2022 07:59:00.2050 */ Loading the AR Server configuration <STARTUP> <0000000042> /* Fri Oct 07 2022 07:59:01.1600 */ Language = en;UTF-8 <STARTUP> <0000000042> /* Fri Oct 07 2022 07:59:04.0080 */ Escalation scheduler running. <STARTUP> <0000000042> /* Fri Oct 07 2022 07:59:05.8290 */ Initializing Access Control Interceptor Aspects. Analyze the logs and identify errors when the does not start successfully. |
7 | Review the API and SQL logs | Enable the API and SQL logs and restart the .
|
8 | Determine a solution |
|
9 | Collect logs | Rename the zipped file to include the server name. |
10 | Analyze logs | Review the logs and identify error messages or behavior. Also, use the table to analyze possible symptoms and solutions. |
11 | Create a support ticket | Collect and send the following information when you create a case with BMC Support:
For more information, see the knowledge article on BMC communities SFT - Steps to send logs, files, screenshots, etc to BMC Support for a Product related case. |
After you determine a specific symptom or error message, use the following table to identify the solution:
Error message | Location | Action | Reference |
---|---|---|---|
Failed to fetch database name. | armonitor.log | Make sure that the database is active and accessible. | |
IO Error: | armonitor.log | Make sure that you can connect to the database with a non-AR Server connection. | |
org.eclipse.persistence. exceptions.DatabaseException | armonitor.log | Make sure that the database is active and accessible. | |
Error encountered in bundle: com.bmc.arsys.messaging | arerror.log | Delete bundle cache. |