Creating a custom pipeline configuration file to integrate with a third-party product

After you create a sandbox environment, run the sample pipeline configuration, and get familiar with the integrating process, you can start creating custom pipeline configuration files to integrate the third-party product with BMC Helix Developer Tools. The following section considers the SolarWinds NPM product as an example, and explains how to create a custom configuration file to push events from SolarWinds NPM to BMC Helix Platform.

Before you begin

Ensure the following:

Download and install the BMC Helix Developer Tools Connector.
Create a sandbox environment by using the Custom Integration template.

To create an event pipeline configuration file

The following steps consider the Solarwinds NPM product and helps you to create an event pipeline configuration file.

Log on to the host computer on which you have installed the BMC Helix Developer ToolsConnector. For example, Test-host.
Identify the integration details. For more information, see Exploring the out-of-the-box directories and configuration files available in the sandbox environment.
Run the following command to verify the current status of the Connector:
tail -f /opt/bmc/connectors/<connector-name>/logs/fluent.log
The following output is displayed, which conveys that the Connector is up and running and logging the heartbeat periodically.
Go to the /opt/bmc/connectors/<connector name>/custom directory. Example connector name is Custom_integration_demo.
Create a text file named solarwinds_event_mapping.json file.
Add the mapping details as shown in the following example:
#The solarwinds_event_mapping.json file maps solarwinds events data to the BMC Helix Platform compatible format
#by using the following mapping structure:
{
"EventMappingDetails": [
{
   "inputkey": "Event",
   "outputkey": "class",
   "type": "constant"
},
{
   "inputkey": "LocalEventTime",
   "outputkey": "creation_time",
   "type": "time",
   "inputformat": "%Y-%m-%dT%H:%M:%S",
   "outputformat": "EPOCH"
},
{
   "inputkey": "EventID",
   "outputkey": "_identifier",
   "type": "assignment"
},
{
   "inputkey": "DNS",
   "outputkey": "source_hostname",
   "type": "assignment"
},
{
   "inputkey": "Message",
   "outputkey": "msg",
   "type": "assignment"
},
{
   "inputkey": "status",
   "outputkey": "status",
   "type": "lookup",
   "lookupMap": [{
     "0": "OPEN",
     "1": "ACK"
   }]
}]
}
Save the file.
Go to the /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline directory.
Locate the generic_event_pipeline.conf file.
Create a copy of this file and rename it to represent the custom third-party integration that you plan to create. For example, rename it to solarwinds_event_pipeline.conf.
Using a text editor, open the solarwinds_event_pipeline.conf file.
Uncomment the <source>, <filter>, and <match> directives, and provide the details as shown in the following example:
<source>
  @type bmc_ade_http_pull
  tag refreshEvents
  url https://host.test.com:17778/SolarWinds/InformationService/v3/Json/Query?query=SELECT top 5 e.EventID, tolocal(e.EventTime) as LocalEventTime, e.NetObjectID, n.DNS, n.IP, e.EventType, et.Name as EventTypeName, et.Icon, e.Message, tostring(e.Acknowledged) as status, e.InstanceType, e.DisplayName, e.Description FROM Orion.Events e , Orion.EventTypes et, Orion.Nodes n where e.EventType = et.EventType and e.NetObjectID = n.NodeID and e.EventTime > ADDSECOND(-(((CURRENT_TIME/1000)-(LAST_FETCH_TIME/1000))*1),GETUTCDATE())
  Path to remember the state: config_file_path /fluentd/etc/custom/state_config.json
  interval 20s
  verify_ssl false
  format json
  user user1
  password test123
</source>

# Flatten the JSON structure
<filter refreshEvents>
  @type json_transform
  transform_script flatten
</filter>

<filter refreshEvents>
  @type record_modifier
  whitelist_keys message.results
</filter>

# Transform the flattened data to the BMC Helix Platform compatible format
<filter refreshEvents>
  @type bmc_ade_transformer
  mapping_file /fluentd/etc/custom/solarwinds_event_mapping.json
  mapping_json_key EventMappingDetails
  input_result message.results
</filter>

# Print the transformed data on the screen
<match refreshEvents>
  @type stdout
</match>

# Send the transformed data to BMC Helix Platform
<match refreshEvents>
  @type bmc_ade_http
  endpoint_url    https://test.helixplatform.com/events-service/api/v1.0/events
  #endpoint_url   ${{CONNECTOR_ENV.ADE_BASE_URL}}/events-service/api/v1.0/events
  ssl_no_verify   true # default: false
  use_ssl         true
  http_method     post    # default: post
  serializer      json   # default: form
  rate_limit_msec 0    # default: 0 = no rate limiting
  raise_on_error true # default: true
  recoverable_status_codes 503, 400, 200 # default: 503
  format           json
</match>
Save the file.
Click the action menu of the sandbox integration that you have created, and select Edit.
Select Events from the Entity Name drop-down list, and enter the solarwinds_event_pipeline.conf file name in the Pipeline File text box.
Click Update.
If the pipeline configuration is applied successfully, you will start seeing the following output:
- Do the following steps to view the count of ingested events:
  1. Click Configured Integrations.
  2. Locate the configured integration tile that you created by using the Custom Integration template. For example, Custom_Integration_test. You will be able to see the count of ingested events on the configured tile as shown in the following example:
- Log on to the BMC Helix Operations Management console, and go to Monitoring > Events to view the ingested events.

Where to go from here

After you have create your own custom pipeline configuration file to ingest data from a product that you plan to integrate, parameterize the file to enable end users to provide details.

Creating a custom pipeline configuration file to integrate with a third-party product

Before you begin

To create an event pipeline configuration file

Where to go from here

On this page