Troubleshooting overall Mid Tier performance issues


Actions processed by Mid Tier and the resource available on the Mid Tier might impact the performance of Mid Tier. This section helps to obtain the appropriate logging and troubleshooting steps to either resolve the problem or create a BMC Support case.

Symptom

  • The overall Mid Tier performance is slow.
  • Mid Tier stops responding.
  • Performance issue is observed on a specific Mid Tier.
  • Web server process continues to run, however the Mid Tier does not respond to the requests. 

Scope

All users connected to the specific Mid Tier experience the problem.

Resolution

Perform the following steps to troubleshoot the performance issues of Mid Tier:

Step

Task

Description

1

Review the configuration.

Verify the Mid Tier and AR System server configuration. 

2

Determine the frequency of the performance issue.


Analyze when the performance issue occurs - at start-up or after running for a predictable interval.


3

Enable logs and reproduce the issue.

To understand the behavior of the Java Virtual Machine (JVM) supporting the Mid Tier, perform the following steps:

  1. Gather thread dumps when the performance is slow or when the Mid Tier is not responding.
    Use the following options to take the thread dump:
    • Use the jVisualVM to gather thread dumps or heap dumps.
      For more information, see the knowledge article Remedy - Mid Tier - How to enable the Java JVisualVM process to collect logging

    • Use the Thread Dump feature on the Log Settings page of Mid Tier Configuration Tool. 
      This will create five separate thread dumps at a set interval.  The Mid Tier thread dumps are stored in the midtier/logs folder on the specific Mid Tier.
  2. Fetch detailed Mid Tier logs by using SERVLET and PERFORMANCE log categories.
    For more information, see Enabling-logs-for-Mid-Tier-troubleshooting.
    Detailed Mid Tier logging gives a perspective of the Mid Tier to AR System server
    processing. You can track the HTTP request to the Mid Tier log and determine if the delay is due to Mid Tier.
  1. Fetch Webserver logs.
    For more information, see Gathering-additional-logging-information.

4.

Disable logging

Perform the steps listed in Enabling detailed Mid Tier logging and Enabling-server-side-AR-System-logs to disable the logging.

Remember to disable the logs from each Mid Tier and AR System server, on which logging was enabled.

5

Collect the logs.


Copy the logs to another location, where you can review them, so that they do not get overwritten.

The logs may be large depending on volume of activity during the logging period.

Collecting Mid Tier logs and thread dumps

Create a zip file of all logs in the log location specified on the Log Settings page of Mid Tier Configuration Tool. 
For Microsoft Windows, the default location is as follows:

<Installation directory>\BMC Software\ARsystem\midtier\logs 

For Linux, the default location is as follows:

/opt/bmc/ARsystem/midtier/logs

For more information, see Gathering-additional-logging-information.

7

Create a BMC Support Case.

Collect and send logs with detailed information when creating a case with BMC Support:

  1. Provide the following information as part of your case:
    • Name of the user who performed the search
    • The search criteria used
    • Time when the problem occurred
    • Any error messages that the user received
  2. Collect the logs.
    • Create a zip file of each Mid Tier log set.
    • Create a zip file of each AR System server log set.
    • Create a zip file of the network capture file (*.har or *.saz file).
  3. Attach the zip files to your case. You can attach files up to 2 GB. You can also upload the files on FTP.
    For more information, see Steps to send logs, files, screenshots, etc to BMC Support for a Remedy Product related case.

8

Analyze the logs.

Review the logs for error messages or behaviors.

Use the following table to troubleshoot based on the symptoms and error messages.

Error messages and resolution

Error

Where

Resolution

{{display reference="Service-Management.Innovation-Suite.BMC-Helix-Innovation-Suite.is233._SvIncludeLibrary._SvIncludePlusContent-_midtier_short-1.WebHome"/}}

 9.x becomes unresponsive due to logs

Mid Tier 9.1.03 or later.

Disable the Internal log category on the Mid Tier Config Tool - Log settings.

Sync cache feature causes server lockup or performance problems

Mid Tier 9.1.02 or earlier.

Upgrade to Mid Tier 9.1.03 or later.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*