This documentation supports the 22.1 version of BMC Helix ITSM: Smart IT. To view an earlier version, select the version from the Product version menu.The documentation for BMC Helix ITSM 23.3.00 has changed. To understand the changed documentation structure see, Where did the Smart IT documentation go?

Troubleshooting the dashboard graphs and KPI Indicators issues

While working with dashboard graphs or social timeline in Smart IT, you might observe performance issues or incorrect data being shown in the graphs. 

Use the information in this topic to identify the problem and to gather relevant information and logs to create a case with BMC Support.

Related topic

Support-information


Symptoms

  • The dashboard and social timeline in Smart IT load slowly.
  • The dashboard graphs show incorrect data.
  • The Smart IT screen displays an error message while loading the data.

Scope

  • One or more users experience these problems. 
  • In a server group environment, these issues could affect one or more servers.


Diagnosing and reporting an issue

Perform the following tasks to troubleshoot the issues:

Step

Task

Description

1

Review the configuration.

Verify that Smart IT is correctly installed and is pointing to an Action Request System load balancer. Pointing Smart IT to a single AR System server is also possible, but it may impact performance on busy environments. Also, check if Smart IT is connected to the user-facing AR System load balancer or to a server with enough resources.

For more information, see KB Article 000219036.

To learn how to change AR System server references, see Changing Action Request System server references.

2

Determine the scope of the issue.


  1. Determine if all users of specific support groups experience issues.
  2. Identify the actual issue with the dashboard. Determine whether the issue is performance related or data related.
  3. Is the data shown on the dashboard incorrect?

Check if you face these issues even when you log in as a non admin user. RLS (Row level security) may cause a delay while retrieving the data. Therefore, log in as a non admin user and verify.

Enable the F12 Network Capture logging, which provides information about delayed responses or incorrect data. For information about how to monitor network response times in Smart IT, see KB Article 000219900.

3

For performance related issues, enable Filter, API, and SQL logging as well as smartit.log.


If users are experiencing slow load times while loading the dashboard, it could be due to association with several support groups, which in turn could cause RLS-related queries to be time consuming.

To find out the affected users and the reason for slow load time of the dashboard, first log in as an admin user. Admin users have more dependence on RLS as opposed to regular users. If you do not experience any issues when you are logged in as an admin user, it indicates that there might be an API or SQL related performance problem. However, if the dashboard takes some time to load even when you are logged in as an admin user, there might be issues with either the database or the Tomcat web server due to the following reasons:

  • Action Request System level
  • The database is not using proper indexes.
  • The Smart IT Tomcat client provides delayed responses.

For load balanced environments, enable API and SQL logging from all user facing servers in the group. For more information, see Enabling server-side AR System logs.

To enable debug searcher logging from indexing server:

  1. Remove any old or unneeded logs from ..\db directory to reduce the size of the file being sent to BMC Support.
  2. Rename log files to make it easier to identify.
  3. Make sure the log files include the time period when the issue was reproduced.

Starting with version 18.05, you can enable logs across servers and server groups. For information about how to enable logs across server groups, see Managing logs for server groups

To enable the Smart IT logs in DEBUG mode for 18.05 and later versions, see How to enable Smart IT debug logs.

For versions 20.08 and later, you can enable them via the log.level Centralized Configuration setting. Change the default value from WARN to DEBUG.

4

Reproduce the issue and note down the search-related information.

Enable logs and reproduce the problem. If you are unable to reproduce the issue, wait until the problem occurs on its own.

Collect the following information to provide to BMC Support:

  • Name of the user who executed the search
  • The search criteria used
  • Time period when the problem occurred

5

Disable logging.

Disable the Action Request System Server Log.

For more information, see Enabling server-side AR System logs

6

Performance-related errors: Collect the logs and run them through the log analyzer to disregard database performance-related issues.


If the issue is mainly related to performance, analyze the logs with theAction Request System Server Log Analyzer. Copy the logs to another location where you can review them so that they do not get overwritten. For more information, see AR Log Analyzer - Recorded Session

By default, Smart IT uses a combined likes algorithm to provide RLS security. Starting with version19.08, on Action Request System Server and Developer Studio, the subquery algorithm has been introduced.

The following snippet is an example of regular RLS algorithm being used:

SELECT * FROM (SELECT AR_SQL_Alias$1.*, rownum as AR_RowNumber_Alias$1 FROM (SELECT T2310.C1000000161, T2310.C1000000018, T2310.C1000000019, T2310.C1000000000, T2310.C303497300, T2310.C1000000164, T2310.C7, T2310.C1000000217, T2310.C1000000218, T2310.C1000005261, T2310.C1000003009, T2310.C1 FROM T2310 WHERE ((T2310.C1000000161 LIKE 'INC0000%') AND (((T2310.C2 = '21001-Test') OR (T2310.C4 = '21001-Test')) OR exists ( select /*+ NO_EXPAND  */ 1 from dual where ((T2310.C112 LIKE '%;'21001-Test';%') OR (T2310.C112 LIKE '%;0;%') OR (T2310.C112 LIKE '%;-20042;%') OR (T2310.C112 LIKE '%;-20032;%') OR (T2310.C112 LIKE '%;-20003;%') OR (T2310.C112 LIKE '%;-20000;%') OR (T2310.C112 LIKE '%;-1099;%') OR (T2310.C112 LIKE '%;-1098;%') OR (T2310.C112 LIKE '%;440;%') OR (T2310.C112 LIKE '%;803;%') OR (T2310.C112 LIKE....

You can observe OR statements being compared against column C112.


The following snippet is an example of the subquery algorithm being used:

SELECT * FROM (SELECT AR_SQL_Alias$1.*, rownum as AR_RowNumber_Alias$1 FROM (SELECT T2310.C1000000161, T2310.C1000000018, T2310.C1000000019, T2310.C1000000000, T2310.C303497300, T2310.C1000000164, T2310.C7, T2310.C1000000217, T2310.C1000000218, T2310.C1000005261, T2310.C1000003009, T2310.C1 FROM T2310 WHERE ((T2310.C1000000161 LIKE 'INC0000%') AND (EXISTS (SELECT 1 FROM S2310 WHERE (S2310.entryId = T2310.C1) AND (((S2310.labelId IN (2, 4)) AND (S2310.labelValue IN ('21001-Test'))) OR ((S2310.labelId IN (112, 60900, 60901, 60989)) AND (S2310.labelValue IN ('''21001-Test''', '0', '-20042', '-20032', '-20003', '-20000', '-1099', '-1098', '440', '803', '804', '1057', '1058', '7113', '13005', '13006', '13007', '13008', '13009', '13010', '20000', '20003', '20007', '20012', '20013', '20032', '20042', '20043', '20055', '20077', '20078', 

The tables get converted to a new 'S' table that handles RLS in a better manner, which is very effective for large environments.

In order to enable the subquery algorithm, see KB Article 000366961: How To Enable Subquery In Smart IT To Improve Overall Performance.

7

Check for AR exception or data-related issues such as KPI data is not updated, is incorrect, or is not displayed at all.

Acquire a copy of the Filter, SQL, API, Escalation logs as well as smartit.log in DEBUG mode and note the user who is performing the action.

  1. If no data is displayed on the dashboard, ensure that the user executing the action has the correct set of permissions pertaining to the company and support groups.
  2. Review the permissions on field id 1 for the HPD:HelpDesk, WOI:WorkOrder, and SRM:Request.
    1. Log in as the affected user and press the F12 key with the browser open. The developer tools open. Once in the F12 Developer Tools, navigate to the Network tab, where you can check the REST calls. You can then reload the dashboard to identify if slowness is caused by the network or if there are other problems. The Network tab also reveals if there are missing objects while querying the Smart IT dashboard or the Ticket Console.

8

Create a BMC Support case.

When creating a case with BMC Support, collect and send logs and detailed information:

  1. Provide the following information as part of your case:
    • Name of the user who performed the search
    • The search criteria used
    • Time when the problem occurred
    • Any error messages received
    • Include the pluginsvr_config.xml from the ..\pluginsvr\fts\secondary directory. If there is more than one indexer, include this file from all indexers.
  2. Run the Log Zipper and select Zip Logs.
    For information about how to run the Log Zipper utility by using the maintenance tool, see AR System Maintenance tool.

  3. Attach the zip file to your case. You can attach a zip file up to 2 GB.
    For more information, see SFT - Steps to send logs, files, screenshots to BMC Support for a Remedy Product related case..


Resolutions for common issues

After you have determined a specific symptom or error message, use the following table for solutions:

Symptom

Where

Action

Reference

The dashboard takes a very long time to load (endless spinning wheel).

Dashboard

  • Apply the necessary indexes to improve performance
  • Enable Subquery algorithm (Action Request System Server 19.08)

Search takes long time to retrieve results.

Global search

  1. Enable FILTER SQL API logs and run them through the AR Log Analyzer and review with DB team if there are indexes that can be implemented.
  2. Review the configuration checklist from the BMC R&D Performance team.
  3. Tune the JVM Tomcat Settings for Smart IT.

The KPI:DataCollection is

arerror.log

  1. Review the KPI:KDC:CleanUp escalation. By default, it runs on Pool 1. If needed, you can select this escalation to run on its own separate pool.
  2. Enable FILTER SQL API and Escalation Log and ensure that the escalations are processed without any issues.
  3. Review the arerror.log and check that there are no issues with escalations running on the designated escalation Action Request System Server.

Data backlog shows incorrect data.

Dashboard

  1. Make sure that the escalations are enabled on the environment.
  2. Review all the escalations that run on KPI:DataCollections as they are responsible for updating the data. An example for incidents: KPI:KDC:Backlog_of_Incident.
  3. By default they run on Pool 1. If there are performance issues, you can select these escalations to run on separate individual pools.
  4. Make sure that the latest hotfix is applied for Smart IT.


Log in to Smart IT. The dashboard is not loaded, instead, users are redirected to the Ticket Console.

arerror.log

arsql.log

  1. Ensure that the landingPage parameter is set to dashboard.

Dashboard graphs are blank.

Dashboard

  1. Make sure that permissions are correctly added into the following forms in Developer Studio: HPD:HelpDesk, WOI:WorkOrder and SRM:Request.
  2. Make sure that the support staff users have IT Agent permissions.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*