Using Orchestration actions to enable triage and remediation of events
Supported with version 11.3.02 and later
The capabilties of integrating TrueSight Presentation Server with TrueSight Orchestration and initiating Orchestration actions from events are available only with TrueSight Presentation Server 11.3.02 and later.
TrueSight Orchestration is licensed separately and not bundled with TrueSight Operations Management. You must download and install it separately. For more information, see
Downloading the installation files for TrueSight Orchestration.
TrueSight Orchestration enables you to automate known remediation scenarios. So, when an event occurs on the TrueSight console, you can easily initiate an Orchestration action from that event. If the Orchestration action is successfully initiated, the remediation steps are run.
Incoming events displayed on the Monitoring > Events page can be of many types. Some events are important and actionable, others are informational, and still others contribute noise. Some of the important and actionable events are recurring events with known remediation steps. Such events can be remediated quickly and easily by running Orchestration actions to reduce time, errors, and delays associated with manual methods. You can also run Orchestration actions to perform triage activities only. By default, TrueSight Presentation Server uses predefined context-based event selection to display out-of-the-box Orchestration actions for relevant events only.
Note that Orchestration actions can be run for PATROL events and alarm events only. To be able to access the Orchestration actions, Presentation Server must be integrated with TrueSight Orchestration and a set of prerequisites must be met on both Presentation Server and TrueSight Orchestration.
If you want to remediate events for use cases other than the ones covered by the out-of-the-box Orchestration actions, you need to configure custom Orchestration actions.
Note
On the TrueSight console, Orchestration action names are displayed in English only.
End-to-end process overview
The following image depicts the end-to-end process involved when you initiate an Orchestration action.
The process starts when an IT operator on the TrueSight console launches an Orchestration action for a particular event. Event data is sent to TrueSight Orchestration. Based on the use case, a triage action is triggered, which verifies the validity of the event. Note that based on the use case triage may not always be required.
Next, an incident is created. By default, BMC Service Resolution is configured to perform incident management. However, you can manually configure TrueSight Orchestration to perform incident management. To configure TrueSight Orchestration for incident management, you need to change some settings on TrueSight Orchestration, in the BMC-SA-Event_Orchestration_Configuration module configuration, under the Specifics > BMC_TrueSight configuration group. For more information, see
Configuring modules in the Event Orchestration runbook.
Then, a change request is created and the workflow waits for the change to be approved. By default, change management is already enabled through TrueSight Orchestration. Note that both change and incident management may not be required based on the use case for which you want to initiate an Orchestration action.
If the remediation action is defined, the remediation action is run, which performs the corrective action on the target server where the problem has occurred. After the remediation is complete, the validation actions are run to ensure that remediation is successful. If a change request was created earlier, it is updated with the latest status and the incident is resolved. Furthermore, TrueSight Infrastructure Management detects that the condition has returned to normal and subsequently closes the event.
Each step of this orchestration process is configurable, such as whether to perform a triage only action, or a combination of triage and remediation or only remediation, and so on. TrueSight Presentation Server provides you with out-of-the box Orchestration actions to perform triage and remediation for service down use case and only triage for the host down use case. If you want to run Orchestration actions for any other use case, you need to perform a set of configurations on TrueSight Orchestration. For more information, see Configuring Orchestration actions for custom use cases.
At each stage of the process, related events are associated with the event from which the Orchestration action was run and are displayed under the Remote Action Result tab.
Requirements for Presentation Server
To be able to initiate Orchestration actions from events, ensure that the following requirements are already met:
| # | Requirement |
|---|---|
| 1 | On Administration > Components, add a TrueSight Orchestration component and a TrueSight Infrastructure Management component. For more information, see Adding and editing components. Notes:
|
| 2 | On the Administration > Components page, ensure that the TrueSight Orchestration component status is Connected. |
| 3 | Ensure that you have the permission to view and initiate Orchestration actions from events. This permission is governed by the Allow Orchestrator Actions permission on the Roles page. |
| 4 | If you plan to use BMC Service Resolution for incident management, ensure that Presentation Server 11.3.02 is integrated with BMC Service Resolution. |
Requirements for TrueSight Orchestration
To be able to initiate Orchestration actions from events, ensure that the following requirements are already met:
| # | Requirement |
|---|---|
| 1 | Ensure that TrueSight Orchestration Platform version 8.2 or later is already installed. For more information, see
Installing the TrueSight Orchestration Platform.
|
| 2 | Ensure that the following version of the Event Orchestration runbook is already installed on the TrueSight Orchestration Platform.
You can choose to configure content either during the installation or after the installation manually.
|
| 3 | Ensure that TrueSight Orchestration is already integrated with BMC Remedy IT Service Management for performing the change management process. For more information about enabling incident and change integration, see the following links based on the TrueSight Orchestration version that you are using:
|
Out-of-the-box Orchestration policies
The following table lists the supported out-of-the-box Orchestration policies. You can use these policies to initiate Orchestration actions for different types of use cases.
These policies are editable and can be customized as per your needs.
For more information about each of the use cases, see
Orchestration actions.
Note
The out-of-the-box Orchestration actions are enabled for Windows and Linux operating systems only.
Also, these actions are supported in English locale settings only.
| Orchestration Policy | Orchestration action | Description | Supported with Presentation Server |
|---|---|---|---|
BMC_TrueSight-ServiceDown-1_policy | Restart Service (in 11.3.02) Start Service (in 11.3.03) | Initiate this Orchestration action to triage and remediate service down problems. Note: The associated Orchestration action is renamed from Restart Service to Start Service in version 11.3.03 of Presentation Server and 20.19.02 of the Event Orchestration runbook. | 11.3.02, 11.3.03 |
| BMC_TrueSight-HostDown-1_policy | Check Host Connection | Initiate this Orchestration action to triage host down problems. | 11.3.02, 11.3.03 |
| Apache Web Server Service Down | Start Apache Web Server | Initiate this action to triage and start the Apache Web Server service. | 11.3.03 |
| Microsoft Exchange Service Down | Start Microsoft Exchange service | Initiate this action to triage and start the Microsoft Exchange service. | 11.3.03 |
| Microsoft SQL Service Down | Start Microsoft SQL service | Initiate this action to triage and start the Microsoft SQL service. | 11.3.03 |
| Oracle Service Down | Start Oracle service | Initiate this action to triage and start the Oracle database service. | 11.3.03 |
| Disk Space Full | Free up disk space | Initiate this action to triage and clean free up disk space. Note: You need to configure this Orchestration action to delete the specific files that are not required. | 11.3.03 |
| PATROL Agent Service Down | Start PATROL Agent | Initiate this action to triage and start the PATROL Agent service. | 11.3.03 |
| Print Spooler Service Down | Start Print Spooler | Initiate this action to triage and start the Print Spooler service. | 11.3.03 |
| Remedy Service Down | Start BMC Remedy AR System Server service | Initiate this action to triage and start the BMC Remedy AR System server service. | 11.3.03 |
| RSCD Agent Service Down | Start RSCD agent | Initiate this action to triage and start the RSCD Agent service. | 11.3.03 |
| Microsoft Skype Service Down | Start Microsoft Skype service | Initiate this action to triage and start the Microsoft Skype service. | 11.3.03 |
| SNMP Service Down | Start SNMP service | Initiate this action to triage and start the SNMP service. | 11.3.03 |
| TrueSight Server Automation Service Down | Start TrueSight Server Automation service | Initiate this action to triage and start the TrueSight Server Automation service. | 11.3.03 |
To initiate an Orchestration action from an event
An Orchestration action can be initiated for one event at a time.
- Go to the Monitoring > Events page.
- Click the action menu of the desired event and select Launch Orchestration Actions.
- Select an Orchestration action from the list displayed, and then click Launch.
A status message indicating whether the action initiation was successful is displayed at the top of the page.
Examples of out-of-the-box Orchestration actions
The following examples describe the high-level process involved when you run one of the out-of-the-box Orchestration actions.
High-level process flow for the service down use case
Scenario: Suppose you see the following event on TrueSight console indicating that the BAO-REPO service is down on the Windows platform.
In this scenario, you can directly run the out-of-the-box Restart Service (or Start Service) Orchestration action and remediate the problem.
High-level process flow: When you initiate the Restart Service Orchestration action for the BAO-REPO event, the TrueSight Orchestration Process Event workflow is run and the following steps are performed.
TrueSight Orchestration performs triage to determine whether the event is valid and the BAO-REPO service is actually down.
- After a successful triage, TrueSight Orchestration checks if an incident is created for this event and performs one of the following actions:
- If the incident is already created via BMC Service Resolution: TrueSight Orchestration updates the incident with the latest status.
- If the incident is not created: If TrueSight Orchestration is configured for incident management, then TrueSight Orchestration creates an incident.
- TrueSight Orchestration creates a change request, the task related to the change, and associates the change with the incident.
- The change request is then sent to the change approver for approval.
- After the change is approved, TrueSight Orchestration starts the BAO-REPO service on the target server (remediation action).
- After the remediation is complete, TrueSight Orchestration validates that the BAO-REPO service is up and running.
Finally, TrueSight Orchestration sends an event to TrueSight console indicating the successful remediation of the BAO-REPO service. Also, the change request is closed and the incident is resolved.
- Related events: At every step, related events are are logged in relation to the main event from which the Orchestration action was run. These events are displayed in the Event Details page as remote action results. To view remote action results, from the event action menu, select View Remote Action Results. Alternatively, click the
icon displayed in the event message or on the event toolbar. - Informational events: At every step, an Information event indicating the status is sent to the TrueSight console. You can view these events by selecting the Information quick filter at the top of the page.
Related events indicating the status (under the Remote Action Result tab)
High-level process flow for the host down use case
Scenario: Suppose you see the following event on TrueSight console indicating that a particular machine is down in your environment.
In this scenario, before remediating the problem, you can run the out-of-the-box Check Host Connection Orchestration action to triage if the computer is still down.
High-level process flow: When you initiate the Check Host Connection Orchestration action for the host down event, the TrueSight Orchestration Process Event workflow is run and the following steps are performed.
TrueSight Orchestration performs triage to determine whether the event is valid and the machine is actually down.
- After a successful triage, TrueSight Orchestration checks if an incident is created for this event and performs one of the following actions:
- If the incident is already created via BMC Service Resolution: TrueSight Orchestration updates the incident with the latest status.
- If the incident is not created: If TrueSight Orchestration is configured for incident management, then TrueSight Orchestration creates an incident.
Finally, TrueSight Orchestration sends an event to the TrueSight console indicating that the triage is successful.
The IT operator assigned to the event must ensure that the device is restarted and the incident is closed. After monitoring restarts on the device, the event is automatically closed.
After you run the Orchestration action, you can track the status of the action by looking at:
- Related events: At every step, related events are are logged in relation to the main event from which the Orchestration action was run. These events are displayed in the Event Details page as remote action results. To view remote action results, from the event action menu, select View Remote Action Results. Alternatively, click the icon displayed in the event message or on the event toolbar.
- Informational events: At every step, an Information event indicating the status is sent to the TrueSight console. You can view these events by selecting the Information quick filter at the top of the page.
Related events indicating the status (under the Remote Action Result tab)
Configuring custom Orchestration actions
If you want to initiate Orchestration actions for use cases other than the out-of-box use cases, then you need to perform additional configurations.
The following table summarizes the configuration steps required for configuring a custom Orchestration action and enabling it on relevant events.
| Step | Product involved | Task |
|---|---|---|
| 0 | Before you begin: Ensure that the requirements for TrueSight Orchestration and requirements for Presentation Server are already met. | |
| 1 | TrueSight Orchestration |
Add a custom use case to enable Orchestration actions
(Required to configure a custom Orchestration action for the custom use case.) |
| 2 | TrueSight Presentation Server | If you are using version 11.3.02 of Presentation Server: If you are using version 11.3.03 of Presentation Server:
Add an Orchestration policy to configure event selection for custom use cases
(Required only if you want to enable the custom Orchestration action on relevant events. Skipping this step will result in the custom Orchestration action getting displayed on all the events irrespective of context.) |
Example scenario for a custom use case
Suppose you see an event on the TrueSight console indicating that more than 75% memory is getting utilized on a particular computer.
You want to perform triage to see the top 10 processes that are consuming the maximum memory on the affected computer. Also, you want the custom Orchestration action to be enabled on Linux events with the memory parameter value greater than 75%. To understand the end-to-end configurations required for enabling this custom Orchestration action, see Example of configurations required for a custom use case.
Comments
hi, we saw that eg the check host OOTB orchestration action does only work if the orchestration server has English locale settings, we had different settings and then transformations and so on in the workflows fail, this seems not be mentioned in the docs
thanks
Thanks for your comment, Roland! We are checking about this issue with R&D. We'll get back to you as soon as we get an answer. In the meanwhile, you might want to contact BMC Customer Support to raise a ticket for this issue.
Thanks Roland! The doc is now updated.
Log in or register to comment.