Troubleshooting issues when the AR System server takes a long time to start
Symptom
The takes a long time (approximately 20 to 30 minutes) to start.
Scope
Affects one or more servers in a server group.
Resolution
Perform the following steps to troubleshoot the issue:
Step | Action | Description |
1 | Collect logs. | The following logs are important to identify startup issues with the :
You can also check the API logs to confirm when the connection with the was established first time. Enable the API and SQL logging if not enabled and capture logs for the slow startup. Consider performing the following actions if the does not start or takes a longer time to start and you cannot enable the API and SQL logs:
|
2 | Review the arerror.log file. | Review the arerror.log file and check if there are any errors reported during the startup. Use the table to analyze the specific errors. |
3 | Review the arstartup_trace log file. | Review the arstartup_trace log file. These logs represents the time duration of the startup. Review the file and identify time gaps in the log. See the example below: */ Fri Dec 10 17:27:40.009 2021 Establishing database connection...success */ Fri Dec 10 17:27:40.466 2021 Loading the AR Server configuration..success */ Fri Dec 10 17:54:23.135 2021 null */ Fri Dec 10 17:54:40.485 2021 Action Request System Server Starting (default locale: en) */ Fri Dec 10 17:55:12.313 2021 Action Request System Server Started. This example shows that the is taking a longer time to load metadata. Review the sample startup log that shows a successful startup. .Use the table to analyze the specific symptoms and errors. |
4 | Review the Centralized Configuration logs. | The Centralized Configuration logs are updated during the Centralized Configuration activities. The configurations are loaded into memory during the startup. Use the timestamp of the actual server startup as a guideline to review the Centralized Configuration logs. See the example below: Fri Nov 18 2022 10:14:48.041 INFO c.b.a.c.s.i.ARDatabaseConfiguration THREAD [359]:- Loading configurations for query - SELECT C3204, C3205, C3210 FROM T25 WHERE C3207 = 'AGGAA5V0FHYCOAQIS9KLQHU9E2O5FV' Fri Nov 18 2022 10:14:51.822 INFO c.b.a.c.s.i.ConfigurationImpl THREAD [287]:- Loading local database configuration. Fri Nov 18 2022 10:14:51.822 INFO c.b.a.c.s.i.ConfigurationImpl THREAD [287]:- Loading global database configuration. Fri Nov 18 2022 10:14:51.838 INFO c.b.a.c.s.i.ConfigurationImpl THREAD [287]:- Loading shared configuration. Fri Nov 18 2022 10:14:51.854 INFO c.b.a.c.s.i.ConfigurationImpl THREAD [287]:- Loading local ARDB configuration. Fri Nov 18 2022 10:14:51.854 INFO c.b.a.c.s.i.ConfigurationImpl THREAD [287]:- Loading global ARDB configuration. Fri Nov 18 2022 10:14:51.869 INFO c.b.a.c.s.i.ARDatabaseConfiguration THREAD [287]:- Loading configurations for query - SELECT C3204, C3205, C3210 FROM T25 WHERE C3207 = 'CPGAA5V0FHYCOAR92TLRR83ETCEQZE' If you observe a gap during the time duration, that might be the reason for the slow startup of the . Use the table to analyze the specific symptoms and errors. |
5 | Review the SQL logs. | The SQL logs represents data dictionary loading. The data dictionary includes a large amount of data and might be a reason for slow startups of the . The SQL logs show activity for Client-RPC:390600 and a NULL or blank user name, which indicates that this was performed by the server itself. You can see SELECT statements for multiple metadata tables, such as arschem, aschema_group_ids, char_menu, filter_mapping, field_dispprop. Search for the statements in the SQL logs that show a large gap from the prior statement. You might not see a long SQL statement but, you might see many SQLs that add up for a larger duration. Analyze logs and identify the activities that might cause slowness. Use the table to analyze the specific symptoms and errors. |
6 | Review the API logs. | Review API logs and analyze the first API call made to an external source. For example, a call from queue Pvr:390680, or queue Fast. Analyze the duration from starting the until the external call. With this information, you can use other logs to analyze the time required for the startup. |
7 | Determine a solution |
|
8 | Collect relevant logs |
|
9 | Analyze logs | Review the logs and identify error messages or behavior. Also, use the table to analyze possible symptoms and solutions. |
10 | Create a Support Ticket | Collect and send the following information when you create a case with BMC Support:
Attach the Log Zipper zip file up to 2 GB or share the file over FTP with BMC Support. For more information, see the knowledge article on BMC communities SFT - Steps to send logs, files, screenshots, etc to BMC Support for a Product related case. |
After you determine a specific symptom or error message, use the following table to identify the solution:
Symptom | Location | Action | Reference |
---|---|---|---|
The overall startup is slow. | SQL logs | Check the network between and the database. | |
A single SQL is very slow. | SQL logs | Test the query directly against the database to see if it is slow outside of the startup. Let the database administrator determine cause of slowness. | |
Large number of rows in ARSystem:RLS AutoDiscovery form. See the example below: /* Wed Nov 03 2021 10:10:56.2670 */ Initializing RLSAutoDiscoveryServiceImpl. .... /* Wed Nov 03 2021 10:55:44.3460 */ Done initializing 33 services concurrently. | arstartup logs | Truncate the relevant table from the database. | |
Slowness due to statistics. See the example below: Oct 08 17:17:42.977 2020 updated statistics configuration: enabled=true aggregation interval (sec)=3600 Tue Oct 08 17:28:02.739 2020 Extension loaded: com.bmc.arsys.alertws 9.1.10.SNAPSHOT ARSYS.ALRT.WEBSERVICE | arstartup logs | In the Centralized Configuration, disable the statistics gathering by setting the value of AR-Calculate-Object-Cache-Statistics to False. |