Integrating with third-party products by using the generic connector via webhook

Related topics

Monitoring BMC Helix Intelligent Integrations

Use the generic connector in BMC Helix Intelligent Integrations to ingest events and metrics data from any third-party monitoring product. The generic connector uses a webhook mechanism to collect data.

As a tenant administrator, it's important that you can monitor the connected systems and quickly identify and resolve any issues.

Important

The generic connector is available only for the on-premises deployment of BMC Helix Intelligent Integrations.

You can view the collected data in various BMC Helix applications and derive the following benefits:

BMC Helix application	Type of data collected or viewed	Benefits
BMC Helix Operations Management	Events	Use a centralized event view to monitor, filter, and manage events, and perform event operations in one place. Process events to help identify actionable events quickly from a large volume of event data. For more information, see Monitoring events and reducing event noise.
BMC Helix Operations Management	Metrics	Use alarm and variate policies to detect anomalies and eliminate false positives for more accurate results while monitoring the health of your system. For more information, see Detecting anomalies by using static and dynamic thresholds.

As a tenant administrator, perform the following steps to configure a connection with third-party product, verify the connection, and view the collected data in various BMC Helix applications.

Supported versions

BMC Helix Intelligent Integrationssupports data collection from any third-party product that can send JSON-formatted events or metrics via webhook.

Task 1: To plan for the connection

Review the following prerequisites to help you plan and configure a connection with a third-party product.

Third-party product prerequisites

Prepare the events JSLT.
BMC Helix Intelligent Integrations receives events from devices and applies the mapping defined in the JSLT to transform them into BMC Helix Operations Management understandable events (in JSON format). For example, in the following sample JSLT, the importance field available in the third-party product is mapped to the severity slot in BMC Helix Operations Management. Therefore, the severity slot displays the value collected from the importance field.

Click here to see a sample events JSLT. Update the JSLT as per your requirements.

[
    for (.)
        let ciExternalId = string("TSOM_"+ .ciType + "-" + .mc_ucid)
        {
            "severity": .importance,
            "msg": .info,
            "status": .state,
            "_ci_id":$ciExternalId,
            "source_hostname":.mc_host,
            "source_identifier": $ciExternalId,
            "details":"",
            "source_attributes": {
                "external_id": $ciExternalId
            },
            "creation_time": string(number(.date_reception) * 1000),
            "source_unique_event_id": "TSOM_" + string(.mc_ueid),
            "sourceEntityType":"Generic",
            "_identifier":.mc_host,
            "ii_version": (.ii_version),
            "class":.class,
            "extras": {
                "sourceEventID": string((.extras.sourceEventID)),
                "sourceEventType": (.extras.sourceEventType)
            }
        }]
Prepare the metrics JSLT.
BMC Helix Intelligent Integrations receives metrics from devices and applies the mapping defined in the JSLT to transform them into BMC Helix Operations Management understandable metrics (in JSON format).

Click here to see a sample metrics JSLT mapping. Update the JSLT as per your requirements.

let convertedData =[for (.)
    [
        for (fallback(.samples,.body.samples))
            let parentBody = {for (.dmd) .key : replace(string(.value),"\\,|\\."," ")}
            [
                for(.dsam)
                    let entityName=.mmd.instance
                    let entityTypeId= .mmd.instance
                    {
                        "labels": {
                            "metricName": .mmd.metric,
                            "hostname": $parentBody.hostname,
                            "entityId":join(["Zabbix",":",($parentBody.hostname),":",$entityTypeId,":",$entityName],""),
                            "entityTypeId": $entityTypeId,
                            "device_id": $parentBody.hostname,
                            "instanceName":$entityName,
                            "entityName": $entityName,
                            "hostType": fallback(.mmd.hostType,"Server"),
                            "isKpi": true,
                            "unit": .mmd.unit,
                            "source": "Zabbix",
                            "external_id": join(["ZABBIX_",.mmd.category,"-",string(.mmd.ifindex)],"")
                        },
                        "samples": [
                            {
                                "value": number(.v),
                                "timestamp": number(.ts)
                            }
                        ]
                    }
            ]
    ]
]
flatten($convertedData)
Prepare the event class JSON if you plan to use a custom event class other than the existing classes in BMC Helix Operations Management. We recommend you to use a custom event class for the third-party product from which plan to collect data. The custom event class is created as the child class of the IIMonitorEvent class in BMC Helix Operations Management.
In addition, if you need to include mapping for any additional parameters for which there is no mapping is available in the standard JSLT, include them in the Event Class JSON section. For example, sourceEventID and sourceEventType are additional parameters in the following sample event JSON. In the JSLT, these parameters are included in the Event Class JSON section.

Click here to see a sample event class JSON. Update the event class JSON as per your requirements.

{
  "name": "CustomEvent",
  "parentClassName": "IIMonitorEvent",
  "attributes": [
    {
      "name": "sourceEventID",
      "dataType": "INTEGER"
    },
    {
      "name": "sourceEventType",
      "dataType": "STRING"
    }
  ]
}

BMC Helix Intelligent Integrations prerequisites

Choose the on-premises BMC Helix Intelligent Integrations deployment mode and review the corresponding port requirements. For information about various deployment modes and port requirements, see Deployment-scenarios.
The on-premises gateway must be able to reach the third-party product on the required port.

Task 2: To configure the connection with a third-party product via webhook

To access BMC Helix Intelligent Integrations on-premises gateway, use the following URL:
https://<hostName>:<portNumber>/swpui
On the CONNECTORS tab, click in the SOURCES panel.
Click the Generic Collector tile.
Specify a unique instance name.
Success
Best practice
We recommend that you specify the instance name in the following format:
<sourceType>_<sourceControllerServerName>_<InstanceQualifier>
The instance qualifier helps you to distinguish the multiple instances configured from the same source server.
For example, you can name your instances as <thirdpartyProductName>_Host_PROD, <thirdpartyProductName>_Host_TEST.
Click VALIDATE AND CREATE.
Make sure that the options for the datatypes for which you want to collect data are selected.
Click CREATE COLLECTORS to create the required collector streams for the selected data types.

Configure the distributors for the selected data types by clicking the respective data type in the Distributors section.
Specify the parameters for the selected data type, as explained in the following table:

Parameter name	Description	Generic Events	Generic Metrics
Max Batching Size	Specify the maximum number of data items to send in a single POST request to the destination API. The batch size depends on the destination’s ability to buffer the incoming data. Default: 250	✅️	✅️
Max Batching Delay	Specify the maximum time (in seconds) to wait before building and processing a batch. Default: 3 seconds	✅️	✅️
Base Retry Delay	Specify the initial time (in seconds) for which to wait before retrying to build and process a batch. The waiting time increases in the following sequence: n¹, n², n³, and so on, where n indicates the number of seconds. Default: 2 seconds Example: The Base Retry Delay is set to 2 seconds. A retry is performed after 2, 4, 8, 16, ... seconds.	✅️	✅️
Max Intra-Retry Delay	Specify the maximum limit for the base retry delay. Default: 60 seconds Example: Max Intra-Retry Delay is set to 60 seconds. Base Retry Delay is set to 2 seconds. Retries are performed 2, 4, 8, 16, 32,... seconds later.	✅️	✅️
Max Retry Duration	Specify the total time for retrying a delivery. For REST destinations, a delivery is a batch of data items in one POST request. Default: 5 minutes Example: Max Retry Duration is set to 8 hours. Base Retry Delay is set to 2 seconds. Requests are sent for 2+4+8+16+32+64+132... until 8 hours in total duration is reached. After that, no subsequent attempts are made to retry the delivery. The assumption here is that if there is an outage or other issue with the destination tool, recovery should take less than the value of the Max Retry Duration parameter to be completed.	✅️	✅️
Attributes To Be Dropped When Updating Events	Specify the event attributes that you do not want to be updated in BMC Helix Operations Management when events are updated. For example, if you do not want an event's severity, source address, source category, and subcategory to be updated in BMC Helix Operations Management, you need to specify those attributes in a comma-separated format: severity,source_address,source_category,source_subcategory. Important: You can obtain the event attribute names in BMC Helix Operations Management, by exporting any event data in JSON, BAROC, XML, or CSV format. The exported file contains all attributes of the event data, and from there you can identify the attributes to be dropped.	✅️	❌️
Device Filters and Metrics filters Device and Metrics filters help remove unwanted data and send only the required data to BMC Helix applications. The data is filtered by using the regular expression (regex) provided for device name, monitor name, monitor type, and metrics, and sent to BMC Helix applications.
Device Name Regex	Specify the regex for the device name. Metrics for devices whose names match the regex are sent to BMC Helix applications. Examples: To send data for devices whose names contain the string Linux, specify the regex as *.Linux.**. To send data for devices containing the strings Cisco* and Linux, specify the regex as *(?=.Cisco)(?=.Linux).*. To send the data for the devices whose names start with the string Linux, specify the regex as ^Linux.. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .Linux. and ^(?!.Linux).** together. The former regex sends metrics for the devices whose names contain the string Linux, while the latter regex sends metrics for the devices whose names do not contain the string Linux.	❌️	✅️
Monitor Name Regex	Specify the regex for the monitor name. Metrics for the monitors whose names match the regex are sent to BMC Helix applications. Examples: To send data for the monitors whose names contain the string system, specify the regex as *.system.**. To filter out the data for monitors that contain system* in their name, specify the regex as *^(?!.system).**. To send the data for monitors whose names start with the string system, specify the regex as ^system.. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .system. and ^(?!.system).** together. The former regex sends metrics for the monitors whose names contain the string system, while the latter regex sends metrics for the monitors whose names do not contain the string system.	❌️	✅️
Monitor Type Regex	Specify the regex for the monitor type. Metrics for the monitor type that matches the regex are sent to BMC Helix applications. Examples: To send data for the monitor type that contains the string health, specify the regex as *.health.**. To send data for the monitor type that starts with the string health, specify the regex as ^health.. To filter out the monitor type containing the string health, specify the regex as ^(*!!.health).. Important:* If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter, ^health.* and *^(?!.health).*** together. The former regex collects metrics for the monitor type whose names contain the string health, while the latter regex collects data for the monitor types whose names do not contain the string health.	❌️	✅️
Metrics Regex	Specify the regex for the metrics. Metrics are sent to BMC Helix applications whose names match the regex. Examples: To send metrics that contain the string avg in their names, specify the regex as *.avg.**. To filter out the metrics that contain the string avg, specify the regex as ^(?!.avg).**. To send metrics whose names start with the string avg, specify the regex as ^avg.. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter, .avg. and ^(?!.avg).** together. The former regex sends metrics whose name contains the string avg, while the latter regex sends metrics whose name does not contain the string avg.	❌️	✅️
Events Filters Events filters help remove unwanted data and send only the required events to BMC Helix applications. The data is filtered by using the regular expression (regex) provided for host, message, and detailed message, and sent to BMC Helix applications.
Host Regex	Specify the regex for the host name. Events for the hosts whose names match the regex are sent to BMC Helix applications. Examples: To send data for the host name /inventory/pricing, specify the regex as ^/inventory/pricing$. To filter out data whose host name contains the string inventory, specify the regex as *^(?!.inventory).**. To send the data for the host whose names start with the string inventory, specify the regex as ^inventory.. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .(inventory). and ^(?!.inventory).** together. The former regex sends events for the hosts whose names contain the string inventory, while the latter regex sends events for the hosts whose names do not contain the string inventory.	✅️	❌️
Message Regex	Specify the regex for the event message. Messages for the events that match the regex are sent to BMC Helix applications. Examples: To send events whose messages contain the string HRV alert, specify the regex as *.HRV alert**. To filter out the events whose message contains the string HRV alert, specify the regex as ^(?!.HRV alert).**. To send events whose message starts with the string HRV alert, specify the regex as ^(HRV alert).. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .HRV alert. and ^(?!.HRV alert).** together. The former regex sends events whose message contains the string HRV alert, while the latter regex sends events whose message does not contain the string HRV alert.	✅️	❌️
Detailed Message Regex	Specify the regex for the detailed message. Detailed messages for the events that match the regex are sent to BMC Helix applications. Examples: To send events whose detailed message contains the string ci_display_name: easyTravel-k8s, specify the regex as *.ci_display_name: easyTravel-k8s.**. To filter out the events whose detailed message contains the string ci_display_name: easyTravel-k8s, specify the regex as ^(?!.ci_display_name: easyTravel-k8s).**. To send the events whose detailed message starts with the string ci_display_name: easyTravel-k8s, specify the regex as ^(ci_display_name: easyTravel-k8s).. Important: If you are using multiple regex, make sure that the regex do not conflict. For example, do not enter .ci_display_name: easyTravel-k8s. and ^(?!.ci_display_name: easyTravel-k8s).** together. The former regex sends events whose detailed message contains the string ci_display_name: easyTravel-k8s, while the latter regex sends events whose message does not contain the string ci_display_name: easyTravel-k8s.	✅️	❌️

In the Distributors section, configure the JSLT mapping for events, metrics, and the event class JSON by performing the following steps:
1. In the JSLT mapping field, enter the required JSLT mapping.
2. In the Event Class JSON field, enter the required event class JSON and the event class policy for any custom classes.
  See the Third-party product prerequisites section for sample events and metrics JSLT mapping and sample event class JSON.
3. Save the configuration.
Click CREATE DISTRIBUTORS to create the required distributor streams for the selected data types.
Click SAVE STREAM to save the stream.
After you save the stream, the connector that you just created is listed on the SOURCES panel.
On the SOURCES panel, click Configure Mediator for the source connection that you created and then expand GENERIC EVENTS WEBHOOK.
Click copy to copy the auto-generated webhook collector URL and save it in a temporary file.
For example, https://hostA/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdf?token=API-KEY.
Depending on whether authentication is enabled or disabled for the on-premises gateway, perform one of the following steps:
- If you are using the on-premises gateway for which authentication is enabled, perform the following steps:
  1. Log on to BMC Helix Portal and generate an access key.
    For instructions, see Setting up access keys for programmatic access.
  2. Copy the generated access key and save it in a temporary file.
    The key is generated in the format: <accessKey>::<secretKey>::<tenantID>.
    For example, Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB::385261281
  3. Change the format of the access key to <tenantID>::<accessKey>::<secretKey>.
    For example, 385261281::Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB
  4. In a temporary file, modify the auto-generated collector URL by replacing API-KEY with the access key that you formatted in the previous step.
    For example, https://host.ab.com/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdfd?token=385261281::Y40OSC49QZA11Q8A1H9H6::MnVLk69TNyCEponsthHJ1Hj1uKcjTB.
  5. Configure the third-party product to send data to the on-premises gateway.
- If you are using the on-premises gateway for which authentication is disabled, perform the following steps:
  1. Save the URL in a temporary file.
  2. Remove the following string from the collector URL:?token=API-KEY
    The updated collector URL looks like the following example:
    https://hostA/hii/api/mediator/v3/push/9mn-6c97-4c2e-8pc5-12c0asdf
  3. Configure the third-party product to send data to the on-premises gateway.
On the SOURCES panel, click Configure Mediator for the source connection that you created and then expand GENERIC METRICS WEBHOOK, and perform steps 13 and 14 for the webhook collector URL for metrics.
On the SOURCES panel, move the slider to the right to start the data stream for the connector.

Task 3: To configure a third-party product to send data to BMC Helix Intelligent Integrations

Log on to the third-party product.
Copy the collector URLs for both events and metrics that you modified in steps 14 and 15, and add them to the third-party webhook configuration.
Go to step 16 to start the data stream.

Task 4: To verify the connection

From BMC Helix Intelligent Integrations, on the SOURCES panel, confirm that the data streams for the connection you created are running. Data streaming is indicated by moving colored arrows.

A moving blue arrow () indicates that the event stream is running. Event data will be pushed as soon as events are available from the third-party product.
A moving red arrow () indicates that the metric stream is running. Metric data will be pushed as soon as metrics are available from the third-party product.

Task 5: To view data in BMC Helix applications

View data collected from the third-party product in multiple BMC Helix applications.

To view events in BMC Helix Operations Management

In BMC Helix Operations Management, select Monitoring and then select Events.
Filter the events by the class name that you have provided in the event class JSON.

For more information about events, see Monitoring and managing events.

To view metrics in BMC Helix Operations Management

In BMC Helix Operations Management, select Monitoring > Devices.
Click the links for the required device.
On the Monitors tab, click the required monitor.
The Performance Overview tab shows the metrics graph.

For information about metrics, see Viewing collected data.