Using Orchestration actions to enable triage and remediation of events




Supported with version 11.3.02 and later

The capabilties of integrating TrueSight Presentation Server with TrueSight Orchestration and initiating Orchestration actions from events are available only with TrueSight Presentation Server 11.3.02 and later.

TrueSight Orchestration is licensed separately and not bundled with TrueSight Operations Management. You must download and install it separately. For more information, see Downloading the installation files for TrueSight Orchestration.

TrueSight Orchestration enables you to automate known remediation scenarios. So, when an event occurs on the TrueSight console, you can easily initiate an Orchestration action from that event. If the Orchestration action is successfully initiated, the remediation steps are run.

Incoming events displayed on the Monitoring > Events page can be of many types. Some events are important and actionable, others are informational, and still others contribute noise. Some of the important and actionable events are recurring events with known remediation steps. Such events can be remediated quickly and easily by running Orchestration actions to reduce time, errors, and delays associated with manual methods. You can also run Orchestration actions to perform triage activities only. By default, TrueSight Presentation Server uses predefined context-based event selection to display out-of-the-box Orchestration actions for relevant events only. 

Note that Orchestration actions can be run for PATROL events and alarm events only. To be able to access the Orchestration actions, Presentation Server must be integrated with TrueSight Orchestration and a set of prerequisites must be met on both Presentation Server and TrueSight Orchestration.

If you want to remediate events for use cases other than the ones covered by the out-of-the-box Orchestration actions, you need to configure custom Orchestration actions. 

Note

On the TrueSight console, Orchestration action names are displayed in English only.



End-to-end process overview

The following image depicts the end-to-end process involved when you initiate an Orchestration action.

TSO integration e2e workflow.png


The process starts when an IT operator on the TrueSight console launches an Orchestration action for a particular event. Event data is sent to TrueSight Orchestration. Based on the use case, a triage action is triggered, which verifies the validity of the event. Note that based on the use case triage may not always be required.

Next, an incident is created. By default, BMC Service Resolution is configured to perform incident management. However, you can manually configure TrueSight Orchestration to perform incident management. To configure TrueSight Orchestration for incident management, you need to change some settings on TrueSight Orchestration, in the BMC-SA-Event_Orchestration_Configuration module configuration, under the Specifics > BMC_TrueSight configuration group. For more information, see Configuring modules in the Event Orchestration runbook.

Then, a change request is created and the workflow waits for the change to be approved. By default, change management is already enabled through TrueSight Orchestration. Note that both change and incident management may not be required based on the use case for which you want to initiate an Orchestration action.

If the remediation action is defined, the remediation action is run, which performs the corrective action on the target server where the problem has occurred. After the remediation is complete, the validation actions are run to ensure that remediation is successful. If a change request was created earlier, it is updated with the latest status and the incident is resolved. Furthermore, TrueSight Infrastructure Management detects that the condition has returned to normal and subsequently closes the event.

Each step of this orchestration process is configurable, such as whether to perform a triage only action, or a combination of triage and remediation or only remediation, and so on. TrueSight Presentation Server provides you with out-of-the box Orchestration actions to perform triage and remediation for service down use case and only triage for the host down use case. If you want to run Orchestration actions for any other use case, you need to perform a set of configurations on TrueSight Orchestration. For more information, see Configuring Orchestration actions for custom use cases.

At each stage of the process, related events are associated with the event from which the Orchestration action was run and are displayed under the Remote Action Result tab. 

Requirements for Presentation Server

To be able to initiate Orchestration actions from events, ensure that the following requirements are already met:

Requirements for TrueSight Orchestration

To be able to initiate Orchestration actions from events, ensure that the following requirements are already met:


Out-of-the-box Orchestration policies

The following table lists the supported out-of-the-box Orchestration policies. You can use these policies to initiate Orchestration actions for different types of use cases.

These policies are editable and can be customized as per your needs.

For more information about each of the use cases, see Orchestration actions.

Note

The out-of-the-box Orchestration actions are enabled for Windows and Linux operating systems only.

Also, these actions are supported in English locale settings only.

To initiate an Orchestration action from an event

An Orchestration action can be initiated for one event at a time.

  1. Go to the Monitoring > Events page.
  2. Click the action menu of the desired event and select Launch Orchestration Actions.
  3. Select an Orchestration action from the list displayed, and then click Launch.
    A status message indicating whether the action initiation was successful is displayed at the top of the page.

Examples of out-of-the-box Orchestration actions

The following examples describe the high-level process involved when you run one of the out-of-the-box Orchestration actions.

High-level process flow for the service down use case

Click to see details

Scenario: Suppose you see the following event on TrueSight console indicating that the BAO-REPO service is down on the Windows platform.

service down event.png

In this scenario, you can directly run the out-of-the-box Restart Service (or Start Service) Orchestration action and remediate the problem. 

High-level process flow: When you initiate the Restart Service Orchestration action for the BAO-REPO event, the TrueSight Orchestration Process Event workflow is run and the following steps are performed.

  1. TrueSight Orchestration performs triage to determine whether the event is valid and the BAO-REPO service is actually down.
  2. After a successful triage, TrueSight Orchestration checks if an incident is created for this event and performs one of the following actions:
    • If the incident is already created via BMC Service Resolution: TrueSight Orchestration updates the incident with the latest status.
    • If the incident is not created: If TrueSight Orchestration is configured for incident management, then TrueSight Orchestration creates an incident.
  3. TrueSight Orchestration creates a change request, the task related to the change, and associates the change with the incident. 
  4. The change request is then sent to the change approver for approval. 
  5. After the change is approved, TrueSight Orchestration starts the BAO-REPO service on the target server (remediation action).
  6. After the remediation is complete, TrueSight Orchestration validates that the BAO-REPO service is up and running.
  7. Finally, TrueSight Orchestration sends an event to TrueSight console indicating the successful remediation of the BAO-REPO service. Also, the change request is closed and the incident is resolved. 
  • Related events: At every step, related events are are logged in relation to the main event from which the Orchestration action was run. These events are displayed in the Event Details page as remote action results. To view remote action results, from the event action menu, select View Remote Action Results. Alternatively, click the event_remote_actions_results.png icon displayed in the event message or on the event toolbar.
  • Informational events: At every step, an Information event indicating the status is sent to the TrueSight console. You can view these events by selecting the Information quick filter at the top of the page.

Related events indicating the status (under the Remote Action Result tab)

service down remote actions.png.jpg

High-level process flow for the host down use case

Click to see details

Scenario: Suppose you see the following event on TrueSight console indicating that a particular machine is down in your environment.

device down event.png

In this scenario, before remediating the problem, you can run the out-of-the-box Check Host Connection Orchestration action to triage if the computer is still down.

High-level process flow: When you initiate the Check Host Connection Orchestration action for the host down event, the TrueSight Orchestration Process Event workflow is run and the following steps are performed.

  1. TrueSight Orchestration performs triage to determine whether the event is valid and the machine is actually down.
  2. After a successful triage, TrueSight Orchestration checks if an incident is created for this event and performs one of the following actions:
    • If the incident is already created via BMC Service Resolution: TrueSight Orchestration updates the incident with the latest status.
    • If the incident is not created: If TrueSight Orchestration is configured for incident management, then TrueSight Orchestration creates an incident.
  3. Finally, TrueSight Orchestration sends an event to the TrueSight console indicating that the triage is successful. 

The IT operator assigned to the event must ensure that the device is restarted and the incident is closed. After monitoring restarts on the device, the event is automatically closed.

After you run the Orchestration action, you can track the status of the action by looking at:

  • Related events: At every step, related events are are logged in relation to the main event from which the Orchestration action was run. These events are displayed in the Event Details page as remote action results. To view remote action results, from the event action menu, select View Remote Action Results. Alternatively, click the event_remote_actions_results.png icon displayed in the event message or on the event toolbar.
  • Informational events: At every step, an Information event indicating the status is sent to the TrueSight console. You can view these events by selecting the Information quick filter at the top of the page.

Related events indicating the status (under the Remote Action Result tab)

device down remote actions.png

Configuring custom Orchestration actions

If you want to initiate Orchestration actions for use cases other than the out-of-box use cases, then you need to perform additional configurations.

The following table summarizes the configuration steps required for configuring a custom Orchestration action and enabling it on relevant events. 

Example scenario for a custom use case

Suppose you see an event on the TrueSight console indicating that more than 75% memory is getting utilized on a particular computer.

You want to perform triage to see the top 10 processes that are consuming the maximum memory on the affected computer. Also, you want the custom Orchestration action to be enabled on Linux events with the memory parameter value greater than 75%. To understand the end-to-end configurations required for enabling this custom Orchestration action, see Example-of-configurations-required-for-a-custom-use-case.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*