Troubleshooting performance issues

This topic provides information about the steps that you can take to troubleshoot various performance issues.   

Troubleshooting overall product performance issues

Symptoms:

• Java heap OutOfMemoryError (%BMC_ITDA_HOME%\logs)
• GC overhead limit exceeded error (%BMC_ITDA_HOME%\logs)
• Slow operations (UI/search)

Solution: Use one or more of the following steps to troubleshoot overall performance issues.

1. Ensure that the maximum Java heap sizes are properly set for each component.

   For more information, see Component configuration recommendations.
2. Ensure that the system has enough physical RAM to prevent paging and swapping. Having additional RAM is beneficial, because the Indexer uses OS-level disk file caching, which can significantly increase performance. For more information, see Hardware sizing recommendations.
3. Check the CPU utilization of the computer on which the product is installed. If the CPU utilization is extremely high, first determine which component has the highest CPU utilization, and then go to one of the following sections to troubleshoot for that specific component:
   
   • Troubleshooting the Indexer performance issues
   • Troubleshooting the Collection Station performance issues
   • Troubleshooting the Console Server performance issues
   • Troubleshooting the Configuration Database performance issues
4. Ensure that you have enough CPU cores for the expected load. For more information, see Hardware sizing recommendations.
5. Verify that your Disk I/O sub system has adequate performance and is not acting as a bottleneck.
6. Verify that the data per index (defined by the intervalInHrs property) is properly set for both the Collection Station and the Console Server + Search. For more information, see Component configuration recommendations.
7. Ensure that you do not have an antivirus software actively scanning files or another third-party application running on the same system as the product components, so that the product is not competing for resources.
8. Check network connectivity between the web browser and the IT Data Analytics server.
9. Refresh the web browser to make certain that it is running the latest code.
10. Restart the web browser.
11. Verify that the computer running the web browser has enough physical RAM available and that CPU cycles are available.
12. Check the itda.log file at %BMC_ITDA_HOME%\logs (in the event of high CPU utilization) to see the number of concurrent searches that are in progress.
     If it appears that too many searches are running concurrently, you might want to reduce the number by using one or more of the following suggestions:
    • Execute memory-intensive searches during off-peak hours. Such searches include search strings that are likely to process a large number of search results, complex search strings, or search strings containing generic key words followed by multiple search commands.
      
    • Distribute the scheduled executions for notifications so that they are do not triggered at the same time.
      
    • Reduce the number of viewlets that you have added.

Troubleshooting Indexer performance issues

Symptom: High CPU utilization on the server hosting the Indexer.

Solution: Use one or more of the following steps to troubleshoot the Indexer performance issues:

1. Check the Collection Station’s collection.log file (located at %BMC_ITDA_HOME%\station\collection\logs) to find the following error:
   org.apache.flume.ChannelException: The channel has reached it's capacity. This might be the result of a sink on the channel having too low of batch size, a downstream system running slower than normal, or that the channel capacity is just too low.
   Ensure that the intervalInHrs property is properly set based on the expected load. By default, every index is optimized every 2 hours. If you have 6-hour interval indices and too much data in the index, the optimizer takes a long time to run and significantly degrades the overall system performance. For more information, see Component configuration recommendations.
2. Even if you do not see the OutOfMemoryError or the GC overhead limit exceeded error in the log file, the Indexer might still have higher CPU utilization due to excessive Java garbage-collection activity. This issue can occur if the maximum Java heap size setting is too low and needs to be adjusted. For more information, see Component configuration recommendations.
3. Remove high cardinality fields (fields with large number of unique values) from the Filter pane on the Search tab.
4. Reduce the number of fields that are calculated in the Filter pane on the Search tab.
5. Narrow the time range for your search query.
6. Make it a practice to explicitly pause or stop search queries that are processing a large quantity of results or that take a long time to complete, when you find that you have enough data.
7. While using search commands with the by parameter, make sure that the by field has low cardinality (a low number of unique values).
8. Reduce the number of concurrent searches.
9. When searching, try to reduce the number of matches first, before piping the data to the search commands.
10. Consider adding more CPU cores. For more information, see Hardware sizing recommendations.
11. Restart the Indexer.

Troubleshooting Collection Station performance issues

Symptom: High CPU utilization on the server hosting the Collection Station.

Solution: Use one or more of the following steps to troubleshoot the Collection Station performance issues:

1. If you notice that the Collection Station’s CPU is taking up most of the CPU resources, it is possible that for some reason the Indexer was not available. Check the Collection Station’s collection.log file (located at %BMC_ITDA_HOME%\station\collection\logs) to see if it had problems sending events to the Indexer. Check the Indexer’s log file (located at %BMC_ITDA_HOME%\logs\indexer) for clues as to why it was too busy or for why the process went down.
2. Check the Collection Station’s collection.log file and verify that the Collection Agent’s flume channel is not full (that is, the Indexer is able to keep up with the rate of data being sent by the Collection Agents). If the Indexer is unable to keep up, ensure that intervalInHrs is properly set based on the expected load. By default, every index is optimized every 2 hours. If you have 6-hour interval indices and too much data in the index, the optimizer takes a long time to run and significantly degrades the overall system performance. For more information, see Component configuration recommendations.
3. Check the Collection Station’s collection.log file to see if the OutOfMemoryError or GC overhead limit exceeded errors occurred. If such errors did occur, increase the Collector Station’s maximum Java heap size. For more information, see Component configuration recommendations.
4. Restart the Collection Station.

Troubleshooting Console Server performance issues

Symptom: High CPU utilization on the server hosting the Console Server.

Solution: Use one or more of the following steps to troubleshoot the Console Server performance issues:

1. Check the itda.log file (at %BMC_ITDA_HOME%\logs) for the OutOfMemoryError or the GC overhead limit exceeded error. If you find either of these errors, increase the Console Server’s maximum Java heap size. For more information, see Component configuration recommendations.
2. Reduce the number of fields that are calculated in the Filter pane on the Search tab.
   
3. Narrow the time range for your search query.
4. Make it a practice to explicitly pause or stop search queries that are processing a large quantity of results or that take a long time to complete, when you find that you have enough data.
5. When using search commands with the by parameter, ensure that the by field has low cardinality (a low number of unique values).
6. Reduce the number of concurrent search commands.
7. If the data collector’s polling interval (default, 1 minute), is consistent with the scheduled poll frequency, and you previously increased the collection.thread.pool.size property to a value much higher than the default value, then try decreasing this value. 
   For more information, see Modifying the configuration files.
8. Reduce the number of notification alerts and reports.
9. Restart the Console Server.

Troubleshooting Configuration Database performance issues

Troubleshooting data collection falling behind schedule

Symptom: Poll interval of the data collectors is falling behind schedule.

Solution: Use the following steps to resolve this issue:

1. In the Collection Station’s agent.properties file, increase the value of the collection.thread.pool.size property.
   For more information, see Modifying the configuration files.
2. Decrease the intervalInHrs setting for number of hours data per index (for both the Console Server and the Collection Agent).
   For more information, see Component configuration recommendations .
3. Increase the Indexer’s maximum Java heap size setting.
   For more information, see Component configuration recommendations.