Integrating with third-party products by using the generic collector via webhook

Related topics

Monitoring BMC Helix Intelligent Integrations

As a tenant administrator or a user with the II Admin role, use the generic collector to collect events and metrics from a webhook-based third-party product for which an out-of-the-box connector is not available. The generic collector uses a webhook mechanism to collect data.

Important

The generic collector is available only for the on-premises deployment of BMC Helix Intelligent Integrations.

You can view the collected data in various BMC Helix applications and derive the following benefits:

BMC Helix application	Type of data collected or viewed	Benefits
BMC Helix Operations Management	Events	Use a centralized event view to monitor, filter, and manage events, and perform event operations in one place. Process events to help identify actionable events quickly from a large volume of event data. For more information, see Monitoring events and reducing event noise.
BMC Helix Operations Management	Metrics	Use alarm and variate policies to detect anomalies and eliminate false positives for more accurate results while monitoring the health of your system. For more information, see Detecting anomalies by using static and dynamic thresholds.

Before you begin

Make sure you perform the following actions before you configure a connection with a third-party product:

Third-party product requirements

Make sure that the third-party product from which you want to collect data is able to send events and metrics in JSON format via a webhook. BMC Helix Intelligent Integrations receives events from devices in JSON format and applies the mapping defined in the JSLT to transform them into BMC Helix Operations Management understandable events (in JSON format).
Prepare the event JSLT and event class JSON.
For information about the guidelines, see Guidelines to prepare the event JSLT and event class JSON.
Prepare the metric JSLT.
For information about the JSLT guidelines, see Guidelines to prepare the metric JSLT.
Prepare the event class JSON if you plan to use a custom event class other than the existing classes in BMC Helix Operations Management.
Success
Best practice
We recommend that you use a custom event class for the third-party product from which you plan to collect data. The custom event class is created as the child class of the IIMonitorEvent class in BMC Helix Operations Management.

Guidelines to prepare the event JSLT and event class JSON

Use the following guidelines to prepare the event JSLT:

Iterate over input events: If the input JSON is an array of events, always wrap the transformation in a for loop, as shown in the following example:
[
for (.)
{ ... mapped object ... }
]
Generate unique identifiers: Use JSLT helper functions to ensure every outgoing event is unique, as shown in the following example:
let uuid = uuid()
Construct External CI ID: Use a fixed prefix, CI type, and UUID, as shown in the following example:
let ciExternalId = string("TSOM_" + .ciType + "-" + $uuid)
Map source fields to target fields: Prepare a table that shows the mapping, as shown below:
Source JSON Target field
.severity "severity"
.msg "msg"
.status "status"
.mc_host "source_hostname"
.class "class"
Example:

"severity": .severity,
"msg": .msg,
"status": .status,
"source_hostname": .mc_host,
"class": .class
Use extras to handle any optional or custom attributes in the source JSON that cannot be mapped to any of the fields in the Target field column, as shown in the following example:
"extras": {
"sourceEventID": string(.extras.sourceEventID),
"sourceEventType": .extras.sourceEventType
}
If you are using any optional or custom attributes, include them in the custom event class JSON as shown in an example later.
Important: If you don't include the optional or custom attributes in extras, they are not passed to the event slots in BMC Helix Operations Management.
Build composite or derived fields if required: Build fields by using the additional logic. For example, if you want to modify the event message, use the following syntax:
"msg": .msg + string(floor(random() * 100)) + "%"
Standardize source identification fields, as shown in the following example:
"_ci_id": $ciExternalId,
"source_identifier": $ciExternalId,
"source_attributes": {
"external_id": $ciExternalId
}
Define unique event identifiers, as shown in the following example:
"source_unique_event_id": "TSOM_" + $uuid
Preserve compatibility fields: Some fields are retained as-is even when not transformed, as shown in the following example:
"_identifier": .mc_host,
"ii_version": (.ii_version)

Source JSON	Target field
.severity	"severity"
.msg	"msg"
.status	"status"
.mc_host	"source_hostname"
.class	"class"

Guideline to prepare event class JSON

If you need to include mapping for any optional or custom attributes for which no mapping is available in the standard JSLT, include them in the event class JSON, as shown in the following example:

"attributes": [
    {
      "name": "sourceEventID",
      "dataType": "INTEGER"
    },
    {
      "name": "sourceEventType",
      "dataType": "STRING"
    }
  ]

The following files show a sample JSON, corresponding JSLT, and custom event class JSON. Update them as per your requirements.

Click here to see a sample event JSON.

[
    {
        "msg": "CPU usage in last 2 mins is ",
        "severity": "CRITICAL",
        "event_handle": "560",
        "mc_host": "dev-pun.abc.com",
        "date_reception": "1763024061",
        "class": "CustomEvent",
        "mc_ueid": "demo12345",
        "mc_ucid": "12345",
        "status": "OPEN",
        "ciType": "Host",
        "extras": {
            "sourceEventID": 1235,
            "sourceEventType": "customEventType"
        }
    }
]

Click here to see a sample event JSLT.

[
    for (.)
        let uuid = uuid()
        let ciExternalId = string("TSOM_"+ .ciType + "-" + $uuid)
        {
            "severity": .severity,
            "msg": .msg + string(floor(random() * 100))+"%",
            "status": .status,
            "_ci_id":$ciExternalId,
            "source_hostname":.mc_host,
            "source_identifier": $ciExternalId,
            "details":"",
            "source_attributes": {
                "external_id": $ciExternalId
            },
            "creation_time": string(round(now()) * 1000),
            "source_unique_event_id": "TSOM_" + $uuid,
            "sourceEntityType":"Generic",
            "_identifier":.mc_host,
            "ii_version": (.ii_version),
            "class":.class,
            "extras": {
                "sourceEventID": string((.extras.sourceEventID)),
                "sourceEventType": (.extras.sourceEventType)
            }
        }
]

Click here to see a sample event class JSON.

{
  "name": "CustomEvent",
  "parentClassName": "IIMonitorEvent",
  "attributes": [
    {
      "name": "sourceEventID",
      "dataType": "INTEGER"
    },
    {
      "name": "sourceEventType",
      "dataType": "STRING"
    }
  ]
}

Guidelines to prepare metric JSLT

Use the following guidelines to prepare the metric JSLT:

Click here to see a sample metric JSLT.

let convertedData =[for (.)
    [
        for (fallback(.samples,.body.samples))
            let parentBody = {for (.dmd) .key : replace(string(.value),"
,|
."," ")}
            [
                for(.dsam)
                    let entityName=.mmd.instance
                    let entityTypeId= .mmd.instance
                    {
                        "labels": {
                            "metricName": .mmd.metric,
                            "hostname": $parentBody.hostname,
                            "entityId":join(["Zabbix",":",($parentBody.hostname),":",$entityTypeId,":",$entityName],""),
                            "entityTypeId": $entityTypeId,
                            "device_id": $parentBody.hostname,
                            "instanceName":$entityName,
                            "entityName": $entityName,
                            "hostType": fallback(.mmd.hostType,"Server"),
                            "isKpi": true,
                            "unit": .mmd.unit,
                            "source": "Zabbix",
                            "external_id": join(["ZABBIX_",.mmd.category,"-",string(.mmd.ifindex)],"")
                        },
                        "samples": [
                            {
                                "value": number(.v),
                                "timestamp": number(.ts)
                            }
                        ]
                    }
            ]
    ]
]

BMC Helix Intelligent Integrations requirements

Choose the on-premises BMC Helix Intelligent Integrations deployment mode and review the corresponding port requirements. For information about various deployment modes and port requirements, see Deployment-scenarios.
Make sure the on-premises gateway can reach the third-party product on the required port.

Task 2: To configure the connection with a third-party product via webhook

To access the BMC Helix Intelligent Integrations on-premises gateway, use the following URL:
https://<hostName>:<portNumber>/swpui
On the CONNECTORS tab, click Add in the SOURCES panel.
Click the Generic Collector tile.
Specify a unique instance name.
Success
Best practice
We recommend that you specify the instance name in the following format:
<sourceType>_<sourceControllerServerName>_<InstanceQualifier>
The instance qualifier helps you to distinguish the multiple instances configured from the same source server.
For example, you can name your instances as <thirdpartyProductName>_Host_PROD, <thirdpartyProductName>_Host_TEST.
Click VALIDATE AND CREATE.
Make sure that the options for the data types for which you want to collect data are selected.
To create collector streams for the selected data types, click CREATE COLLECTORS.

To configure distributors for the selected data types, click the respective data type in the Distributors section.

Click here to view the distribution parameters for the selected data types

Parameter name	Description	Elasticsearch Events	Elasticsearch Merics
Max Batching Size	Specify the maximum number of data items to send in a single POST request to the destination API. The batch size depends on the destination’s ability to buffer the incoming data. Default value: 250	✅️	✅️
Max Batching Delay	Specify the maximum time (in seconds) to wait before building and processing a batch. Default value: 3 seconds	✅️	✅️
Base Retry Delay	Specify the initial waiting time (in seconds) before retrying to build and process a batch. The waiting time increases in the following sequence: n¹, n², n³, and so on, where n indicates the number of seconds. Default value: 2 seconds Example: The Base Retry Delay is set to 2 seconds. A retry is performed after 2, 4, 8, 16, ... seconds.	✅️	✅️
Max Intra-Retry Delay	Specify the maximum limit for the base retry delay. Default value: 60 seconds Example: Max Intra-Retry Delay is set to 60 seconds. Base Retry Delay is set to 2 seconds. Retries are performed 2, 4, 8, 16, 32,... seconds later.	✅️	✅️
Max Retry Duration	Specify the total time for retrying a delivery. For REST destinations, a delivery is a batch of data items in one POST request. Default value: 5 minutes Example: Max Retry Duration is set to 8 hours. Base Retry Delay is set to 2 seconds. Requests are sent for 2+4+8+16+32+64+132... until 8 hours in total duration is reached. After that, no subsequent attempts are made to retry the delivery. The assumption is that if there is an outage or other issue with the destination tool, recovery should take less than the value of the Max Retry Duration parameter to be completed.	✅️	✅️
Attributes To Be Dropped When Updating Events	Specify the event attributes you want to exclude from being updated in BMC Helix Operations Management when events are updated. For example, if you do not want an event's severity, source address, source category, and subcategory to be updated in BMC Helix Operations Management, you need to specify those attributes in a comma-separated format: severity,source_address,source_category,source_subcategory. Important: You can obtain the event attribute names in BMC Helix Operations Management by exporting any event data in JSON, BAROC, XML, or CSV format. From the exported file that contains all attributes of the event data, you can identify which attributes should be dropped.	✅️	❌️
Device Filters and Metrics filters Device and Metrics filters help remove unwanted data and send only the required data to BMC Helix applications. The data is filtered by using the regular expression (regex) provided for device name, monitor name, monitor type, and metrics, and is sent to BMC Helix applications.
Device Name Regex	Specify the regex for the device name. Metrics for devices whose names match the regex are sent to BMC Helix applications. Examples: To send data for devices whose names contain the string Linux, specify the regex as *.Linux.**. To send data for devices whose names contain the strings Cisco* and Linux, specify the regex as *(?=.Cisco)(?=.Linux).*. To send the data for the devices whose names start with the string Linux, specify the regex as ^Linux.. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .Linux. and ^(?!.Linux).** together. The former regex sends metrics for the devices whose names contain the string Linux, while the latter regex sends metrics for the devices whose names do not contain the string Linux.	❌️	✅️
Monitor Name Regex	Specify the regex for the monitor name. Metrics for the monitors whose names match the regex are sent to BMC Helix applications. Examples: To send data for the monitors whose names contain the string system, specify the regex as *.system.**. To send the data for monitors whose names start with the string system, specify the regex as ^system.*. To filter out the data for monitors that contain system* in their name, specify the regex as *^(?!.system).*. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .system. and ^(?!.system).** together. The former regex sends metrics for the monitors whose names contain the string system, while the latter regex sends metrics for the monitors whose names do not contain the string system.	❌️	✅️
Monitor Type Regex	Specify the regex for the monitor type. Metrics for the monitor type that matches the regex are sent to BMC Helix applications. Examples: To send data for the monitor type that contains the string health, specify the regex as *.health.**. To send data for the monitor type that starts with the string health, specify the regex as ^health.*. To filter out the monitor type containing the string health, specify the regex as ^(?!.health).. Important:* If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter ^health.* and *^(?!.health).*** together. The former regex collects metrics for the monitor type whose names contain the string health, while the latter regex collects data for the monitor types whose names do not contain the string health.	❌️	✅️
Metrics Regex	Specify the regex for the metrics. Metrics are sent to BMC Helix applications whose names match the regex. Examples: To send metrics that contain the string avg in their names, specify the regex as *.avg.**. To send metrics whose names start with the string avg, specify the regex as ^avg.*. To filter out the metrics that contain the string avg, specify the regex as ^(?!.avg).*. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter, .avg. and ^(?!.avg).** together. The former regex sends metrics whose name contains the string avg, while the latter regex sends metrics whose name does not contain the string avg.	❌️	✅️
Events Filters Events filters help remove unwanted data and send only the required events to BMC Helix applications. The data is filtered by using the regular expression (regex) provided for host, message, and detailed message and is sent to BMC Helix applications.
Host Regex	Specify the regex for the host name. Events for the hosts whose names match the regex are sent to BMC Helix applications. Examples: To send data for the hosts whose names contain the string /inventory/pricing, specify the regex as ^/inventory/pricing$. To send the data for the hosts whose names start with the string inventory, specify the regex as ^inventory.. To filter out data for the hosts whose name contains the string inventory, specify the regex as ^(?!.inventory).*. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .(inventory). and ^(?!.inventory).** together. The former regex sends events for the hosts whose names contain the string inventory, while the latter regex sends events for the hosts whose names do not contain the string inventory.	✅️	❌️
Message Regex	Specify the regex for the event message. Messages for the events that match the regex are sent to BMC Helix applications. Examples: To send events whose messages contain the string HRV alert, specify the regex as *.HRV alert**. To send events whose message starts with the string HRV alert, specify the regex as ^(HRV alert).*. To filter out the events whose message contains the string HRV alert, specify the regex as ^(?!.HRV alert).*. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .HRV alert. and ^(?!.HRV alert).** together. The former regex sends events whose message contains the string HRV alert, while the latter regex sends events whose message does not contain the string HRV alert.	✅️	❌️
Detailed Message Regex	Specify the regex for the detailed message. Detailed messages for the events that match the regex are sent to BMC Helix applications. Examples: To send events whose detailed message contains the string ci_display_name: easyTravel-k8s, specify the regex as *.ci_display_name: easyTravel-k8s.**. To send the events whose detailed message starts with the string ci_display_name: easyTravel-k8s, specify the regex as ^(ci_display_name: easyTravel-k8s).*. To filter out the events whose detailed message contains the string ci_display_name: easyTravel-k8s, specify the regex as ^(?!.ci_display_name: easyTravel-k8s).*. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .ci_display_name: easyTravel-k8s. and ^(?!.ci_display_name: easyTravel-k8s).** together. The former regex sends events whose detailed message contains the string ci_display_name: easyTravel-k8s, while the latter regex sends events whose message does not contain the string ci_display_name: easyTravel-k8s.	✅️	❌️

In the Distributors section, configure the JSLT mapping for events, metrics, and the event class JSON by performing the following steps:
1. In the JSLT mapping field, enter the required JSLT mapping.
2. In the Event Class JSON field, enter the required event class JSON and the event class policy for any custom classes.
  See the Third-party product prerequisites section for sample events and metrics JSLT mapping and sample event class JSON.
3. Save the configuration.
To create distributor streams for the selected data types, click CREATE DISTRIBUTORS.
Click SAVE STREAM.
After you save the stream, the generic collector that you created is listed on the SOURCES panel.
On the SOURCES panel, click Configure Mediator for the source connection that you created and then expand GENERIC EVENTS WEBHOOK.
Click copy to copy the auto-generated webhook collector URL and save it in a temporary file.
For example, https://hostA/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdf?token=API-KEY.
Depending on whether authentication is enabled or disabled for the on-premises gateway, perform one of the following steps:
- If you are using the on-premises gateway for which authentication is enabled, perform the following steps:
  1. Log on to BMC Helix Portal and generate an access key.
    For instructions, see Setting up access keys for programmatic access.
  2. Copy the generated access key and save it in a temporary file.
    The key is generated in the format: <accessKey>::<secretKey>::<tenantID>.
    For example, Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB::385261281
  3. Change the format of the access key to <tenantID>::<accessKey>::<secretKey>.
    For example, 385261281::Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB
  4. In a temporary file, modify the auto-generated collector URL by replacing API-KEY with the access key that you formatted in the previous step.
    For example, https://host.ab.com/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdfd?token=385261281::Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB.
  5. Configure the third-party product to send data to the on-premises gateway.
- If you are using the on-premises gateway for which authentication is disabled, perform the following steps:
  1. Save the URL in a temporary file.
  2. Remove the following string from the collector URL:?token=API-KEY
    The updated collector URL looks like the following example:
    https://hostA/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdf
  3. Configure the third-party product to send data to the on-premises gateway.
On the SOURCES panel, click Configure Mediator for the source connection that you created and then expand GENERIC METRICS WEBHOOK, and perform steps 13 and 14 for the webhook collector URL for metrics.
On the SOURCES panel, move the slider to the right to start the data stream for the generic collector.

Task 3: To configure a third-party product to send data to BMC Helix Intelligent Integrations

Log on to the third-party product.
Copy the webhook collector URLs for both events and metrics that you modified in steps 14 and 15, and add them to the third-party webhook configuration.
Log on to the BMC Helix Intelligent Integrations on-premises gateway.
On the SOURCES panel, move the slider to the right to start the data stream for the generic collector.

Task 4: To verify the connection

From BMC Helix Intelligent Integrations, on the SOURCES panel, confirm that the data streams for the connection you created are running. Data streaming is indicated by moving colored arrows.

A moving blue arrow () indicates that the event stream is running. Event data will be pushed as soon as events are available from the third-party product.
A moving red arrow () indicates that the metric stream is running. Metric data will be pushed as soon as metrics are available from the third-party product.

Task 5: To view data in BMC Helix applications

View data collected from the third-party product in multiple BMC Helix applications.

To view events in BMC Helix Operations Management

In BMC Helix Operations Management, select Monitoring and then select Events.
Filter the events by the class name that you have provided in the event class JSON.

For more information about events, see Monitoring and managing events.

To view metrics in BMC Helix Operations Management:

In BMC Helix Operations Management, select Monitoring > Devices.
Click the link for the device that has received metrics from the third-party product.
On the Monitors tab, click the monitor that contains the received metrics.
The following figure shows the graph for a sample metric on the Performance Overview tab on the Monitor Details page:

For information about metrics, see Viewing collected data.