Getting started with an example pipeline

After you create a sandbox integration environment by using the Custom Integration template, you can run the example pipeline that is available out-of-the-box to get started quickly. The sandbox integration environment has the example_pipeline directory as shown in the following image:

The example_pipeline directory has the nagios_events_pipeline.conf event pipeline configuration file. When you run this pipeline configuration file, events data is read from the nagios_input.json file, formatted as per the mapping details defined in the nagios_event_mapping.json file, and the transformed events data is sent to BMC Helix Platform. 

The following section helps you to run the nagios_events_pipeline.conf file by using the Custom Integration template. 

Before you begin

• Download and install the connector.
• Create a sandbox integration environment by using the Custom Integration template.

To run the example pipeline file

1. Log on to the host computer on which you have installed the BMC Helix Developer Tools connector.
2. Go to the /opt/bmc/connectors/<connector-name>/data/<integration ID>/pipeline/example_pipeline/config directory.
   An example connector name is Custom_integration_demo. To learn how to identify the integration id, host name, and connector name of a sandbox integration, see Exploring-the-default-sandbox-directory-structure-and-configuration-files.

3. Run the following command to verify the current status of the connector: 

   tail -f /opt/bmc/connectors/<connector-name>/logs/fluent.log

   The following output is displayed, which conveys that the connector is up and running and logging the heartbeat periodically. 
   

4. Select the Configured Integrations tab.
5. Click the action menu of the sandbox integration that you have created, and select Edit. 
   
6. Select Events from the Entity Name list, and enter the nagios_events_pipeline.conf file name in the Entity Configuration File text box. 
   
7. Click Configure. 
8. After the pipeline configuration is applied successfully, perform the following steps to see the output.
   • To view the count of ingested events:
     1. Go to the Configured Integrations tab on the BMC Helix Developer Tools console.
     2. Locate the configured integration tile that you created by using the Custom Integration template. You should see the count of ingested events on the configured tile, as shown in the following example:
        
   • Log on to the BMC Helix Operations Management console, and select Monitoring > Events to view the ingested events.
      

Pipeline customization examples

To understand the pipeline configuration files better and to learn its functionality, explore these configuration files in detail, modify, and then run them. 

Example customization for the nagios_input.json file

The  nagios_input.json file has a few input directives, such as HOSTNAME from where to fetch the input data and a few other parameters. To experiment, you can modify any of these input parameters and save the file as explained in the following steps:

1. Go to the /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline/example_pipeline/config/ directory.

2. Using a text editor, open the nagios_input.json file.
   By default, the nagios_input.json file has the following content.

   {"HOSTNAME" : "www.bmc.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK - Packet loss = 0%, RTA = 222.15 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=221.057999ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.bmc.com0"}
   
   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}
   
   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}

3. Modify the content.
   In the following sample modification, the PING OK label is changed to PING OK Demo.

   {"HOSTNAME" : "www.bmc.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK Demo - Packet loss = 0%, RTA = 222.15 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=221.057999ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.bmc.com0"}
   
   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK Demo - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}
   
   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK Demo - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}
4. Save the file.

Example customization for the nagios_event_mapping.json file

The  nagios_event_mapping.json  file has defined the mapping details between input and output. To explore, you can modify any of these mapping details or add new mapping details and save the file as explained in the following steps:

1. Go to the /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline/example_pipeline/config/ directory.

2. Using a text editor, open the nagios_event_mapping.json file.
   By default, the nagios_event_mapping.json file has the following content: 

   {
           "NagiosMappingDetails": [
                   {
                           "inputkey": "vl-aus-domdv019.abc.com",
                           "outputkey": "source_identifier",
                           "type": "constant"
                   },
                                   {
                           "inputkey": "OPEN",
                           "outputkey": "status",
                           "type": "constant"
                   },
                   {
                           "inputkey": "HOSTOUTPUT",
                           "outputkey": "msg",
                           "type": "assignment"
                   },
                   {
                           "inputkey": "SERVICESTATE",
                           "outputkey": "severity",
                           "type": "assignment"
                   }
           ]
    }

3. Modify or add the mapping details. 
   In the sample modification that follows, an additional mapping for the SERVICEDESC input key is added:

   {
   "NagiosMappingDetails": [
   {
   "inputkey": "vl-aus-domdv019.abc.com",
   "outputkey": "source_identifier",
   "type": "constant"
   },
   {
   "inputkey": "OPEN",
   "outputkey": "status",
   "type": "constant"
   },
   {
   "inputkey": "HOSTOUTPUT",
   "outputkey": "msg",
   "type": "assignment"
   },
   {

   "inputkey": "SERVICESTATE",
   "outputkey": "severity",
   "type": "assignment"
   },

   { 
   "inputkey": "SERVICEDESC", 
   "outputkey": "DemoDescription", 
   "type": "assignment" 
   } 
   ]
   }

4. Save the file.

To reload the configurations to reflect the changes

1. Go to the /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline/example_pipeline/ directory and run the following command: 

   Syntax

   docker exec -it <Container ID> bash reload.sh

   Example

   docker exec -it deed817ccb72 bash reload.sh

   How do I find the Container ID?

   Run the following command to get the container ID:

   docker ps | grep <connector-name>

   The command displays the output in the following format. Note that the CONTAINER ID from the output.

   CONTAINER ID   IMAGE          COMMAND      CREATED        STATUS       PORTS                NAMES deed817ccb72   57e8da5cb8ac   "/run.sh"    5 hours ago    Up 2 hours   5140/tcp,24224/tcp   generic_connector_test

Understand the nagios_events_pipeline.conf file structure

The nagios_events_pipeline.conf file has the configurations to run the events pipeline, which will get the data from the specified input resource, modify the data according to the mapping configurations, and send the data to BMC Helix Platform.

The following code block displays the nagios_events_pipeline.conf file content. There are three important directives in this file: source, filter, and match.

The following section explains the different directives and plug-ins used in this configuration file and its functionality:

Input directive — <source> </source>

The source directive is used to specify the source details such as the type of the plugin used, path, and user credentials. In this example, the path of the nagios_input.json and nagios_input.json.pos files are specified. 

Filter directive — <filter> </filter>

The filter directive specifies the sample data transformation plug-in that is used to convert data from a third-party product to the BMC Helix Platform  compatible format. In this example, the nagios_event_mapping.json file is specified as the mapping file.

Output directive — <match> </match>

The match directive is used to push the transformed data intoBMC Helix Platform. 

For more information about the parameters used in the plug-in, see bmc-ade-http-plug-in.

Where to go from here

After you explore the example pipeline configuration files, create a new custom pipeline configuration file to ingest data from a product that you plan to integrate.