| | |
| Downloading the log analyzer | |
| | - Enable the SQL, API, and Filter logging on every server in the server group to get the best information to understand the performance issue and to help identify a root cause.
- To enable logs on multiple servers in a server group, use the Managing-logs-for-server-group option from the Server Group Admin Console.
- The performance issue may take some time to occur. During this time, the logs may grow larger in number and they get rolled over so you see only some part of the logs at a given time. Capture the logs shortly after the problem occurs. Allow the logs to run for 5-10 minutes after the reported problem occurs to capture the last logs of the SQLs and APIs logs which take a long time to complete.
- You can combine the logs into a single file but this will cause the logs to grow quickly, and the total logs gathered will be shorter. Ensure that you have at least 8 backups set for log history. Also ensure that you have enough disk space to contain all the logs.
|
| Reproduce the problem or wait for the problem to occur | - Perform the activities where you experience the slowness.
- If the performance issue cannot be reproduced, wait for the issue to occur.
|
| | - Disable the logs 5-10 minutes after the issue is reproduced.
- Collect and zip up the SQL/API/Filter logs from every server in the server group. The logs should include the time frame (from a time before the problem was observed (at least 5 minutes) to 5-10 minutes after the problem).
- Rename the zip file to include the server name where the problem was identified.
- On the system running the AR System Log Analyzer, copy each of the server's log set into its own directory. Perform the following step after you copy the server's log set:
- Move or remove the zip file from the directory where the logs are placed so that a reference to *.* does not include the zip file.
For example:
D:\LogAnalysis D:\LogAnalysis\Server1 Contains: arapisqlfilter.log, arapisqlfilter.log.1, arapisqlfilter.log.2 D:\LogAnalysis\Server2 Contains: arapisqlfilter.log, arapisqlfilter.log.1, arapisqlfilter.log.2, arapisqlfilter.log.3 |
| | Installation: - To install, simply unzip the attached package into a directory of your choice such a C:\ARLogAnalyzer (Windows) or /opt/ARLogAnalyzer (Unix)
- For ease of use, you can add this directory to your PATH environment variable. Or you can simply include the full path when you execute ARLogAnalyzer.
Note: It is assumed that the Java directory is in your path.
Usage: There are multiple ways to run the utility:
1. Drag and Drop a single file or a folder onto the appropriate batch file provided (Windows).
2. Execute one of the batch files or shell scripts according to your needs (Windows or Unix)
3. Execute the java command line to run ARLogAnalyzer.jar with the appropriate options
For example, if your log files are stored in a directory called D:\ARLogs, you can simply drag and drop the ARLogs folder onto the analyzeFolderWebZip.bat file.
This will create both a new folder called ARLogs analysis with the web-based analysis and a zip file called ARLogs analysis.zip that contains a compressed copy of the ARLogs analysis folder. For steps to use ARLogAnalyzer and create an Analysis report, see theKnowledge Article 000373578. |
| View the AR System Log Analyzer output | - Verify that the logs captured the activity you want to analyze.
- Verify that the output shows statistics that indicate further investigation is required.
- Identify any pattern of individual slow API calls or SQL statements.
- Identify specific APIs, SQLs, or filters that are causing slowness.
To view AR System Log Analyzer output: - Navigate to the directory that AR System Log Analyzer created (In our example directory name is “ARLogs analysis”)
- Open index.htm in a browser.
|
| Review the results of the AR System Log analyzer | Review each of the following sections as you review the results from the AR System Log Analyzer: |
| | - Verify appropriate elapsed time in the log and note start time, end time, and elapsed time.
- Capture the logs during a time when the performance issue occurs.
- Capture at least 30 minutes of activity as per the general guidelines.
- Note the user count, which provides a high-level indication of whether other user activity might be involved in the reported performance issue.
- Note API and SQL exceptions
- API exceptions indicate API calls that are not completely captured in the log due to the beginning or end of the logs being outside the range.
- SQL exceptions are SQL statements that reported an unexpected error.
- Run the logs for a few minutes after the performance issue is reported. This increases the likelihood that complete results are recorded in the logs.
|
| | This section lists top n (100) long running API calls. API Aggregates is generally the focus area for performance troubleshooting. - Check the form name, API call, and run time values for the top reported APIs and verify if they are matching to the problem.
- Use the table in the following section to troubleshoot long running APIs.
|
| | |
| | This section shows the API calls which are not completed. - Long running API calls with no end time should be analyzed if the start time is well before the end time of the API logs. For example, if the start time is more than 10 seconds before the end time.
- Use the table in the following section to find a possible cause.
|
| | This section shows long-running SQL statements. - If you see long running UPDATE, DELETE, or INSERT statements, which are generally very fast in their operation, the problem might be database blocking.
- If you see long running SELECT statements, there might be a SQL that needs tuning.
|
| | - Use the table in the following section to troubleshoot specific problems found in the analysis.
- If the cause has not been found or no solution is available, proceed to the next step to gather logs and create a Support Case.
|
| | - Zip up each "max" directory and include the server name in the filename. When you create a case with BMC Support, include these zip files and any observations you made from your analysis.
- Provide the following information as part of your case:
- Users that experienced the problem
- Approximate timestamp when the slowness was noticed
- If identified, provide the SQL and API that were causing the slowness
- Note any recent changes to the application, database, or other components of the Remedy environment
3. Attach any materials or reports provided by the DBA: - This list includes AWR, ADDM, or ASH reports, execution plans, observations.
4. Attach the zipped log files for each server to your case (up to 2 GB) or transfer the files to BMC using FTP - Steps to send logs, files, screenshots, etc to BMC Support for a Remedy Product related case. |
| |
|---|
| This issue is identified in the Log Analyzer report. - Click the line number of the long running API call noticed in the Top N section of the API aggregates
It displays the API call, SQLs, and filters triggered for the selected API. - Scroll down through the log line (while watching the timestamp column) and check if any long running SQL or filter is involved or API call taking time due to filter.
- To view additional details, click the next pages. The page links are on the top right of the output.
For example:
The cause for the API call taking too much time is seen in this section and might be one of the following:
- One or more specific SQL statements takes a long time to compete. See the long running SQL steps.
- One or more filter actions takes a long time to complete. See the filter processing steps.
- Many activities taking place at the same time take long time to complete. In this case, check if there is a workflow problem (such as a filter push fields action updating a large number of records and causing subsequent filters to fire), or a general performance issue.
|
API call taking time due to long running SQL | - Run the SQL statement directly on the database client and check the response time.
- You can use the QueryTest Tool to test the query outside of AR System using JDBC.
- If response time is higher than expected, ask the Database administrator (DBA) to check if any indexes can be created to fine tune the SQL.
- Follow the steps in Troubleshooting-SQL-Tuning-issues.
- If the response time is fast yet the SQL logs (via AR System Log Analyzer) show it is slow, create a Support Ticket and provide this information.
|
API call taking time due to filter processing | - Using the AR System Log Analyzer output, identify the filter which takes a longer time of execution.
- Check the actions triggered by the filter using Developer Studio.
- Check if filter is calling any plugin to get data from or set data on any vendor form.
- Check the respective plugin logs if vendor form is involved.
For example if a call is happening on Report creator form, check arjavaplugin.log to see why the plugin call is taking time.
- Check if the filter is performing a Filter API action which calls a plugin.
- Check the respective plugin logs if a Filter API call is involved.
- If the filter is performing a Push Fields action, determine if too many records are matching the qualification and causing many record updates. If this is the case, create a Support Ticket with that information.
|
Large queued time observed | |
API calls in exception list | - Ignore the calls with message: “End of API call has no corresponding start”. These are hard to follow since we don't have the beginning of the API call.
- Check API calls with message: “Start of API call has no corresponding end”.
- Note down the start time of the API call and end time of the logs from the general statistics section and identify how long the API call was running.
- If it is less than 20 seconds, we can ignore the API calls.
- If the API call was running for more than 20 seconds, perform the steps in the Long Running API steps.
|