Troubleshooting the Investigate tool

Investigate enables you to generate charts or tables to view and monitor the performance issues by using the required metrics. You can save these charts or tables as an investigate study. You can launch the study to view the related metrics in charts or tables. You can drill down to specific systems and metrics of interest, and perform root-cause analysis for any capacity or performance issues. Typically, the Investigate tool is used to analyze near real-time data and short-term history data to address the capacity and performance bottlenecks in your environment.  

The following questions may help you to resolve some of the issues that you might encounter while using the Investigate tool.

How to access the Investigate tool?

Do the following:

  1. Log in to the BMC Helix Capacity Optimization console.
  2. In the left pane, click Investigate.

What roles and permissions are required to access the Investigate tool?

Make sure the user role has the associated permissions (specific to the Investigate tool only). In the BMC Helix Capacity Optimization console, make sure that the required activities are granted to the role so that the user can access the Investigate tool. 

How to check and collect the relevant logs?

If you encounter an issue with the Investigate tool, review the following logs.

Data Provider Service logs

To review the Data Provider Service logs, do the following:

Using command line:

  1. Log in to the host computer on which you have installed the Application Server that is running the Data Hub component.
  2. At the shell prompt, log in as the user that you specified during installation (example: cpit).
  3. Navigate to the <base_direcory>/datahub/log directory (example: $CPITBASE/datahub/log).
  4. Check data-prv* and agent-data* logs.

    image2018-3-19 14 33 38.png
    You can also collect these logs using the logGrabber utility. 

Using the UI:

  1. Log in to the BMC Helix Capacity Optimization console.
  2. Navigate to Administration > SYSTEM > Status and select the Data Hub component.
  3. Click the API Providers - Data Provider Service backend service.
  1. Click Show log.

Gateway Server logs

To review the Gateway Server logs, do the following:

  1. Log in to the host where the Gateway Server is installed.
  2. On the command line, navigate to the following directory:
    <GWS_install_dir>/local/manager/log/ 
  1. Check the following logs:
    • <GWS_system_name>-webInvestigate.log
    • <GWS_system_name>-General-Manager.log

Agent logs from the Gateway Server

To collect logs from the agent that is installed on the Gateway Server host, do the following:

  1. Navigate to the following directory:
    <GWS_install_dir>/bgs/bin

  2. Run the following command:

    ./bgsagent_logs -n <agent_hostname>

  3. Check the output to see whether it includes any connection errors. 

  4. Redirect the output to a file because it is large.

    ./bgsagent_logs -n <agent_hostname> > /tmp/agent_logs.txt

Time statistics information from the Gateway Server

You can use the time statistics information to check for connection and other errors.

Do the following:

  1. Log in to the Gateway Server host.
  2. Navigate to the following directory:
    <GWS_install_dir>/bgs/bin

  3. Run these commands as the same user that starts the Gateway Server and the agent process on the Gateway Server:

    end=`date +%s`;let begin=$end-3600; time /usr/adm/best1_11.3.00/bgs/bin/webInvestigate 
    getAgentHistoryMetricGroupDataList -A <hostname> -M "System Statistics"  -t 180 -n "Run Q" -b $begin -e $end
    time /usr/adm/best1_11.3.00/bgs/bin/webInvestigate getAgentHistoryMetricGroupDataList -A <hostname> -M
    "System Statistics"  -t 180 -n "Run Q"

    Replace the <hostname> with the host name value from the agent.
    The output of these commands is large. Hence, we recommend redirecting the command outputs to separate text files.

How to make computers with the Capacity Agent installed visible for charting in the Investigate tool?

If a computer running an agent is a part of the Manager run that is scheduled in the BMC Helix Capacity Optimization console, the computer is available for selection in the Investigate tool. 

The following computers are not available for selection in the Investigate tool:

  • Computers that are not added to Manager runs
  • Computers managed by Gateway Servers that are not defined in the Gateway Manager GUI

To make these computers visible for selection in the Investigate tool, add them as unmanaged computers. 

Adding unmanaged computers

Do the following:

  1. Log in to the BMC Helix Capacity Optimization console.
  2. Navigate to Gateway Manager > Gateway Servers.
  3. Click the required Gateway Server.
  4. Click Edit Uncataloged Systems.
  5. In the text box, add the host names of IP addresses of the computers that you want to add as unmanaged computers.
  6. Save the changes.

How to enable or disable the Investigate history data collection?

You can enable or disable collection of the Investigate history data as required.

Enabling the Investigate history data collection

Do the following:

  1. Log in to the computer where the agent is installed.
  2. Navigate to the following directory:
    • (Windows) %BEST1_COLLECT_HOME%\bgs\bin
    • (Linux) $BEST1_HOME/bgs/bin
  3. Run the command remotely:
    remoteHistoryConfig [-n node -q |-e |-d |-D]
    where
    -n node  - remote agent node
    -q: Query existing Agent configuration
    -e: Enable historical data collection
    -d: Disable historical data collection
    -D: Disable historical data collection and delete existing history files
    For example, to query the current Investigate History Status on remote node 'topgun' and then to enable history, run these commands:
    remoteHistoryConfig -n topgun -q
    Info: history disabled on node :topgun
    remoteHistoryConfig -n topgun -e
    Info : history enabled on node :topgun

Disabling the Investigate history data collection

You can use any of the following methods to disable the Investigate history data collection.

Using the remoteHistoryConfig command 

Do the following:

  1. Log in to the computer where the agent is installed.
  2. Navigate to the following directory:
    • (Windows) %BEST1_COLLECT_HOME%\bgs\bin
    • (Linux) $BEST1_HOME/bgs/bin
  3. Run the command remotely:
    remoteHistoryConfig [-n node -q |-e |-d |-D]
    where
    -n node  - remote agent node
    -q: Query existing Agent configuration
    -e: Enable historical data collection
    -d: Disable historical data collection
    -D: Disable historical data collection and delete existing history files
    For example, to query the current Investigate History Status on remote node 'topgun' and then to enable history, run these commands:
    remoteHistoryConfig -n topgun -q
    Info: history enabled on node :topgun
    remoteHistoryConfig -n topgun -d
    Info : history disabled on node :topgun

Using the repository.cfg file 

Do the following:

  1. Log in to the host where the agent is installed.
  2. Change to the following directory:
    • (Windows) cd C:\Program Files\BMC Software\Patrol3\BEST1\v.v.vv\bgs\mgroup
    • (Linux) cd /usr/adm/best1_default/bgs/mgroup
  3. Back up the existing Repository.cfg file.
    cp Repository.cfg Repository.cfg.bak
  4. Open the Repository.cfg file with a text editor such as VI.
  5. Under the "UDR history configuration" section, change the value of ENABLE parameter to false. For example,
    ENABLE=false
    Make this change for all the occurrences of ENABLE.
    Note: Do not modify any parameter values under the "one shot configuration" section.
  6. Save the Repository.cfg file.
  7. Stop the agent:
    • (Windows) C:\Program Files\BMC Software\Patrol3\BEST1\v.v.vv\bgs\bin\best1agent_stop
    • (Linux) /usr/adm/best1_default/bgs/scripts/best1agent_stop -b /usr/adm/best1_default
  8. Start the agent:
    • (Windows) C:\Program Files\BMC Software\Patrol3\BEST1\v.v.vv\bgs\bin\best1collect  -q
    • (Linux) /usr/adm/best1_default/bgs/scripts/best1collect -B /usr/adm/best1_default -q

(Windows) Using the Control Panel GUI 

Do the following:

  1. Log in to the Windows system where the agent is installed.
  2. Click Start > Settings > Control Panel > PATROL - Perform Agent.
  3. On the History tab, select Disable.
  4. Click OK to save the changes.
  5. To stop the agent, click Start > Programs > BMC Patrol > Collect Data.
  6. Click Query Information from a collect process. 
  7. Enter the host name of the Windows Server where the Investigate history is being disabled. This step initiates a collect query, which restarts the agent.

What is the frequently reported issue with the Investigate tool?

The following issue is frequently reported:

Issue: Investigate study error

In the console, BCO_PS_ERR104 error occurs while creating an Investigate study. 

Symptoms

The console shows the systems for selection. However, study creation fails with the 403 forbidden error. 

Error_1.jpg

The coRest.log shows the following warning:

WARN  03/22 13:43:15.896 [https-xxx-xxx-8043-exec-2] coRest General exception while processing request: [POST HTTPS://<hostname>.com:8443/dh-services/agent-data/v1/collection-groups returned 403 and response: 403-FORBIDDEN]

Cause

BMC Helix Capacity Optimization is configured with an incorrect port.

Solution

The default port while connecting to BMC Helix Capacity Optimization is 8280. In the following example, 8443 port is shown that is used to access the Investigate tool.

Change this port to 8280 to create Investigate studies.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*