Connectors, templates, plug-ins, and pipelines

This topic explains a connector and its directory structure, Custom Integration templates, plug-ins, and pipelines.

BMC Helix Developer Tools connector

A BMC Helix Developer Tools connector is a container based on Fluentd that can host multiple integrations and facilitates data collection and consumption from the integrated products. For example, if you want to get TrueSight Operations Management events into BMC Helix Operations Management so that you can monitor these events in BMC Helix Operations Management, you can configure the TrueSight Operations Management integration easily by using a BMC Helix Developer Tools connector.

Fluentd is an open-source data collector that lets you unify data collection and consumption so that you can understand and analyze third-party data. BMC Helix Developer Tools uses Fluentd plug-ins to integrate with third-party products. Fluentd offers the following features:

Unified logging with JSON
Pluggable architecture
Use of minimal resources

For more information about Fluentd features, see Fluentd features.

A connector acts as a bridge between your application and the third-party product. The connector runs in a customer data center and helps in managing and configuring integrations with various third-party products.

Tenant administrators can install and run a connector by downloading the installation script from the BMC Helix Developer Tools console.

BMC Helix Developer Toolsconnectors support the following key functions:

Send a heartbeat request periodically to BMC Helix Platform.
Receive commands from BMC Helix Platform through a heartbeat response.
Run the commands received from BMC Helix Platform and send the acknowledgment (success/failure) to BMC Helix Platform.
Provide a runtime environment for the execution of integration pipelines and store the application logs.

Connector directory structure

The following diagram lists the important directories and their content of BMC Helix Developer Tools connectors.

Custom Integration templates in BMC Helix Developer Tools

BMC Helix Developer Toolsprovides Custom Integration templates to facilitate custom third-party integrations. As an integration developer, create a sandbox environment by using the Custom Integration template. Using this sandbox environment, you can:

Create a pipeline configuration file for the custom third-party integration
Parameterize the pipeline configuration file
Test the custom pipeline configuration
Deploy the custom third-party integration

Plug-ins

A plug-in is a piece of software that adds new functions to a host program without altering it. Fluentd uses plug-ins to process events. The following plug-in types are available in Fluentd:

Input
Parser
Output
Formatter
Storage
Service discovery
Buffer

For more information, see Fluentd plug-ins.

BMC Helix Developer Toolsuses a few custom-created plug-ins along with the Fluentd plug-ins to seamlessly integrate with BMC Helix Platform REST APIs. For more information, see Custom-plug-ins-used-in-BMC-Helix-Developer-Tools.

Pipelines

Pipelines contain directives for enabling the data to be ingested, transformed, and correlated into a model for performing ML-based analytics. The following diagram represents a typical pipeline used in a BMC Helix Developer Tools connector.

Example pipeline to collect and transform events

Step 1: Get events from SolarWinds NPM

The following example represents a sample input Fluentd plug-in that is used to get data from a third-party product. In this example, we have used SolarWinds NPM as the third-party product.

<source>
  @type http_pull
  tag refreshEvents
  url https://host.test.com:17778/SolarWinds/InformationService/v3/Json/Query?query=SELECT top 5 e.EventID, tolocal(e.EventTime) as LocalEventTime, e.NetObjectID, n.DNS, n.IP, e.EventType, et.Name as EventTypeName, et.Icon, e.Message, tostring(e.Acknowledged) as status, e.InstanceType, e.DisplayName, e.Description FROM Orion.Events e , Orion.EventTypes et, Orion.Nodes n where e.EventType = et.EventType and e.NetObjectID = n.NodeID order by EventTime DESC
  interval 20s
  verify_ssl false
  format json
  user user1
  password test123
</source>

Step 2: Transform events to the BMC Helix Platform compatible format

The following example represents a sample data transform BMC plug-in that is used to convert data from a third-party product to the BMC Helix Platform compatible format:

<filter refreshEvents>
  @type bmc_ade_transformer
  mapping_file /fluentd/etc/custom/event_slot_mapping.json
  mapping_json_key EventMappingDetails
  input_result message.results
</filter>

For more information, see bmc-ade-transformer-plug-in.

Step 3: Send the transformed events to BMC Helix Platform

The following example represents a sample output BMC plug-in that is used to send data to BMC Helix Platform.

<match refreshEvents>
  @type bmc_ade_http
  endpoint_url    https://<BMC Helix Portal URL>/events-service/api/v1.0/events
  ssl_no_verify   true # default: false
  use_ssl         true
  http_method     post    # default: post
  serializer      json   # default: form
  rate_limit_msec 0    # default: 0 = no rate limiting
  raise_on_error true # default: true
  recoverable_status_codes 503, 400, 200 # default: 503
  format           json
</match>

For more information, see bmc-ade-http-plug-in.

Where to go from here

Identify the product that you want to integrate and explore its details.