This topic describes some of the issues and troubleshooting related to TrueSight Infrastructure Management Java memory management. The goal is to help you avoid potential performance impacts.
Memory issues can appear on almost any component, but the most common areas of concern are pserver, jserver, csr, index server, and agent controller.
If you experience performance and/or memory issues, run the TrueSight Health Check Tool. See
The Infrastructure Management Maintenance Tool utility is included with the Infrastructure Management installation, and is made up of tabbed pages that provide options for administering Infrastructure Management. You can use this utility to:
Administering Infrastructure Management Maintenance Tool - Documentation link
You can use the Infrastructure Management Maintenance Tool Tune Parameters tab to configure the performance tuning parameters.
Perform the following steps to configure the performance tuning parameters using the Tune Parameters tab:
After you identify the symptoms and scope of the issue, use the troubleshooting guide to help diagnose and resolve the issue or to contact Customer Support.
Jserver process crash due to insufficient memory:
If the process has crashed due to insufficient memory then a memory dump file will be created in <PnServerPath>\pw\pronto\logs directory with .hprof extension.
In case of a JVM crash, a core dump file with this naming convention 'hs_err_pid*.log' will be created under <PnServerPath>\pw\pronto\tmp.
Jserver process in hung state
|Determining this state is not straight forward, typically Operations/Admin console could become unresponsive because the requests to jserver take more time to respond. Please not this is a possibility and may not be an exact symptom.|
We can collect the thread dump in this case.
In windows : pw threaddump jserver
In solaris : kill –QUIT <pid>
In windows, a file by name jserverTD.out will be created under <PnServerPath>\pw\pronto\logs directory.
In case of solaris, the output will be captured in <PnServerPath>\pw\pronto\logs\jserver.out.
How to enable debug logs for jserver process
The command to enable debug of jserver process is
pw debug on -p jserver [-s <subsystem>]
Enabling debug for all the subsystem is expensive and the log files will be flooded with messages. So it is always advisable to enable for the required subsystems. The subsystem to be enabled should be determined based on the nature of the issue.
Following command can be used to find the various subsystems under jserver process
pw debug list -p jserver
On enabling debug, the log messages will be captured in <BPPM_HOME>\pw\prontp\logs\debug\jserver.log.
The debug should be turned off after collecting the logs as this is an expensive operation.
If you are seeing poor GUI performance and jserver is consuming high levels of CPU when attempting to navigate the tool.
Slowness can have many causes, so Support will ask for hardware specifications as well as load on the application, and if there are any other outside factors (patching,other applications on the box, network connectivity issues, etc). The goal is to get an overall view of your environment to help narrow down the focus of the search.
While there may be a variety of factors, one way to clear up slowness is to enable lazyloading. This is a property which can be set in order to allow more effective loading of objects in the console.
Certain performance problems can be observed within the navigation tree of the Ops Console. Examples of this could be:
Setting the lazy loading parameter to true can help to resolve these problems. The performance problems seen in the navigation tree can be caused by a number of things such as:
We recommend that you uncheck those items in the navigation tree preferences that you are not interested in. For example, if you are interested in events only, then uncheck the boxes related to service model and folders.
This is because of the way the navigation tree works, in that data is loaded in the order of top to down. So, it will be event collectors first, then groups, then service model CIs, then component folders, then event folders. All of this can take time as they have to be loaded from various sources such as cell, jserver, backend database, and it can result in a bottleneck.
The lazyloading option says to load only the top-level elements. Then as soon as a user wishes to expand that top-level it will fetch the data at that point. That is why traversing from top-level down can seem slower, but it is greatly reducing the load on the jserver.
To turn the lazyloading option on, add the following property to the pw/custom/conf/pronet.conf file:
Then reload the jserver properties with command:
pw jproperties reload
Then monitor the performance of the TrueSight Infrastructure Management Operations Console.
If the performance continues to be an issue, there is another property to set which can also help with TrueSight Infrastructure Management slowness, the nearcache property.
Add the following properties to the pw/custom/conf/pronet.conf
Then restart the TrueSight Infrastructure Management server (pw system start). These properties will help to solve performance issues in the Operations Console and Administration Console and these properties are set by default in TrueSight 11.3.01
If the issue remains after enabling the properties above, collect the pw dump 1 output and send the details to Support for further assistance.
While there may be many factors involved in this type of situation, it is always best to send the logs to Support to review them for other possible causes of the issue.
Due to the elastic search errors below the component was unable to communicate (sync) with TSPS which caused it to fail.
INFO 06/21 09:32:53.393 [EventMsg_28] TsimAudit Failed to insert events to ES
nested: ScriptException[compile error]; nested: IllegalArgumentException[Scripts may be no longer than 16384 characters. The passed in
script is 21057 characters. Consider using a plugin if a script longer than this length is a requirement.];
This issue with elastic search is fixed in TrueSight 11.3.01. A workaround is to delete the elastic search data folder and restart the Presentation Server. The elastic search folder can be found here: <TSPS_HOME>/modules/elasticsearch/
Delete the entire folder as it will be recreated upon restart of the Presentation Server.
If clearing the elastic search folder does not allow the TrueSight Infrastructure Management component to initialize, gather up the output of the tssh dump export command and send it to Support for further review.