This documentation supports the 23.3 version of BMC Helix Innovation Suite (AR System and BMC Helix Innovation Studio).

To view an earlier version, select the version from the Product version menu.

Troubleshooting performance issues by using AR System Log Analyzer

When you observe performance issues in AR System server, use the following information to either troubleshoot and resolve the issues by using AR System Log Analyzer or create a BMC Support case.

Symptoms

  • Slow performance speed.
  • Slowness or timeout errors while loading the forms (For example, Overview console, Incident or Change management console)
  • Slowness or timeout errors while creating or modifying a record
  • Timeout errors logged in the arerror.log file

Scope

  • One or more users experience this problem
  • One or more servers in a server group experience this problem 


Diagnosing and reporting an issue

Step

Task

Description

1Download the log analyzer
2

Enable logging

  • Enable the SQL, API, and Filter logging on every server in the server group to get the best information to understand the performance issue and to help identify a root cause.
  • To enable logs on multiple servers in a server group, use the Managing logs for server groups option from the Server Group Admin Console.
  • The performance issue may take some time to occur. During this time, the logs may grow larger in number and get rolled over. Hence, you may only see some part of the logs at a given time.
    Capture the logs shortly after the problem occurs.
    Allow the logs to run for 5-10 minutes after the reported problem occurs and then capture the last logs of the SQLs and APIs, which take a long time to complete.

  • You can combine the logs into a single file but this will cause the logs to grow quickly, and the total logs gathered will be shorter. 
    Make sure that you have at least 8 backups set for log history, and have enough disk space to contain all the logs.

3

Reproduce the problem or wait for the problem to occur 

  • Perform the activities where you experience the slowness.
  • If you cannot reproduce the performance issue, wait for the issue to occur.
4

Collect the logs

  • Disable the logs 5-10 minutes after the issue is reproduced.
  • Collect and zip the SQL/API/Filter logs from every server in the server group. The logs should include the time frame (from a time before the problem was observed (at least 5 minutes) to 5-10 minutes after the problem).

  • Rename the zip file to include the server name where the problem was identified.
  • On the system running the AR System Log Analyzer, copy each of the server's log set into its own directory. Perform the following step after you copy the server's log set:
    • Move or remove the zip file from the directory where the logs are placed so that a reference to *.* does not include the zip file.
      For example:

D:\LogAnalysis

D:\LogAnalysis\Server1

Contains: arapisqlfilter.log, arapisqlfilter.log.1, arapisqlfilter.log.2

D:\LogAnalysis\Server2

Contains: arapisqlfilter.log, arapisqlfilter.log.1, arapisqlfilter.log.2, arapisqlfilter.log.3

5

Running the  AR System Log Analyzer

Perform the following steps from the server running the AR System Log Analyzer:

AR System Log Analyzer can be CPU intensive while analyzing logs.

  1. Unzip the “ARLogAnalyzer9104.zip” into the directory from which you ran the AR System Log Analyzer.
    For example: D:\ARLOGAN
  2. Open a command prompt window and navigate to the working directory that contains the log files.

  3. Run the following command:

    arpreparelogs -o <outputLogFileName> <log filename mask>

    For example:  

    arpreparelogs -o final.log *.*  

    The output log file (final.log) is created in the working directory. The size of the output file is the sum of the input file sizes.

    If there are multiple servers, run the command once from each log directory where you copied the logs. Only log files should be present in each of these directories.

  4. After you run the arpreparelogs command, run the following command: 

    arwklga -sMAX -wMAX -n <number of Top items to report> <outputLogFileName>

    For example: 

    arwklga -sMAX -wMAX -n 100 final.log

    This command will create a directory named MAX.

Other parameters available in arwklga can be configured. In some cases, 100 might not be enough lines for the logs. This is true when the AR System Log Analyzer shows the duration of the FTS indexing longer than it actually is, due to the way it is logged. 

Use a larger -n value, such as "-n 200".

6

View the  AR System Log Analyzer output

  1. Verify that the logs captured the activity you want to analyze.

  2. Verify that the output shows statistics that indicate further investigation is required.

  3. Identify any pattern of individual slow API calls or SQL statements.

  4. Identify specific APIs, SQLs, or filters that are causing slowness.


To view AR System Log Analyzer output:

  1. Navigate to the directory that AR System Log Analyzer created (In our example directory name is MAX ).
  2. Open index.htm in a browser.

   

7

Review the results of the AR System Log Analyzer

Review each of the following sections as you review the results from the AR System Log Analyzer:


7.1 - General Statistics 
  1. Verify appropriate elapsed time in the log and note start time, end time, and elapsed time:
    1. Capture the logs during a time when the performance issue occurs.
    2. Capture at least 30 minutes of activity as per the general guidelines.
  2. Note the user count, which provides a high-level indication of whether other user activity might be involved in the reported performance issue.
  3. Note API and SQL exceptions:
    1. API exceptions indicate API calls that are not completely captured in the log due to the beginning or end of the logs being outside the range.
    2. SQL exceptions are SQL statements that reported an unexpected error.
  4. Run the logs for a few minutes after the performance issue is reported. This increases the likelihood that complete results are recorded in the logs.

7.2 - API Aggregates

The API Aggregates section lists top n (100) long running API calls. API Aggregates is generally the focus area for performance troubleshooting.

  • Check the form name, API call, and run time values for the top reported APIs and verify if they are matching to the problem.
  • Use the table in the following section to troubleshoot long running APIs.

7.3 - Top Queued N

7.4 - Exceptions

The Exceptions section shows the API calls which are not completed.

  • Long running API calls with no end time should be analyzed if the start time is well before the end time of the API logs. For example, if the start time is more than 10 seconds before the end time.
  • Use the table in the following section to find a possible cause.

7.5 - SQL Aggregates

The SQL Aggregates section shows long-running SQL statements.

  • If you see long running UPDATE, DELETE, or INSERT statements, which are generally very fast in their operation, the problem might be database blocking.
    Follow the steps in Troubleshooting Database Blocking Issues .
9Find a solution
  1. Use the table in the following section to troubleshoot specific problems found in the analysis.
  2. If the cause has not been found or no solution is available, proceed to the next step to gather logs and create a Support Case.
10Create a Support Ticket
  1. Zip each Max  directory and include the server name in the filename. When you create a case with BMC Support, include these zip files and any observations you made from your analysis.
  2. Provide the following information as part of your case:
    • Users that experienced the problem
    • Approximate timestamp when the slowness was noticed
    • If identified, provide the SQL and API that were causing the slowness
    • Any recent changes to the application, database, or other components of your environment
  3. Attach any material or report provided by the DBA:
    This list includes AWR, ADDM, or ASH reports, execution plans, observations.
  4. Attach the zipped log files for each server to your case (up to 2 GB) or transfer the files to BMC using FTP. (See Steps to send logs, files, screenshots, etc to BMC Support for a Product related case Open link .)

Error messages and resolution

Issue

Resolution

Long running API calls 

Perform the following steps if this issue is identified in the AR System Log Analyzer report:

  1. Click the line number of the long running API call noticed in the Top N section of the API aggregates
    It displays the API call, SQLs, and Filters triggered for the selected API.
  2. Scroll down through the log line (while watching the timestamp column) and check if any long running SQL or Filter is involved or API call taking time due to filter.
    To view additional details, click the next pages. The page links are at the top right of the output.
    For example:

The API call might be taking too much due to one of the following reasons:

    1. One or more specific SQL statements takes a long time to compete. See the long running SQL steps.
    2. One or more filter actions takes a long time to complete. See the filter processing steps.
    3. Many activities taking place at the same time take long time to complete. In this case, check if there is a workflow problem (such as a filter push fields action updating a large number of records and causing subsequent filters to fire), or a general performance issue.

API call taking time due to long running SQL

Run

  1. the SQL statement directly on the database client and check the response time.
  2. You can use the QueryTest Tool to test the query outside of AR System using JDBC. (See How can I test a DB Query using JDBC outside of ARServer? Open link .)
  3. If response time is higher than expected, ask the Database administrator (DBA) to check if any indexes can be created to fine tune the SQL.
  4. Follow the steps in Troubleshooting SQL Tuning issues.
  5. If the response time is fast yet the SQL logs (by using the AR System Log Analyzer) show it is slow, create a Support Ticket and provide this information.

API call taking time due to filter processing 

Perform the following steps if this issue is identified:

  1. Using the AR System Log Analyzer output, identify the filter which takes a longer time of execution.
  2. Check the actions triggered by the Filter using Developer Studio .
  3. Check if the Filter is calling any plugin to get data from or set data on any vendor form.
    1. Check the respective plugin logs if vendor form is involved.
      For example, if a call is happening on Report creator form, check arjavaplugin.log to see why the plugin call is taking time.
  4. Check if the Filter is performing a Filter API action which calls a plugin.
    1. Check the respective plugin logs if a Filter API call is involved.
  5. If the Filter is performing a Push Fields action, determine if too many records are matching the qualification and causing many record updates. If this is the case, create a Support Ticket with that information.
Large queued time observed 

See the Troubleshooting ARServer Thread Tuning and Queue Wait Time issues troubleshooting guide.

API calls in exception list

Perform the following steps if this issue is identified:

  1. Ignore the calls with message: “End of API call has no corresponding start”. These are hard to follow since we don't have the beginning of the API call.
  2. Check API calls with message: “Start of API call has no corresponding end”.
    1. Note down the start time of the API call and end time of the logs from the general statistics section and identify how long the API call was running.
    2. If it is less than 20 seconds, we can ignore the API calls.
    3. If the API call was running for more than 20 seconds, perform the steps in the Long Running API steps.
Was this page helpful? Yes No Submitting... Thank you

Comments