Getting started with an example pipeline

After you create a sandbox integration environment by using the Custom Integration template, you can run the example pipeline that is available out-of-the-box to get started quickly. The sandbox integration environment has the example_pipeline directory as shown in the following image:

The example_pipeline directory has the nagios_events_pipeline.conf event pipeline configuration file. When you run this pipeline configuration file, events data is read from the nagios_input.json file, formatted as per the mapping details defined in the nagios_event_mapping.json file, and the transformed events data is sent toBMC Helix Platform. 

The following section helps you to run the nagios_events_pipeline.conf file by using the Custom Integration template. 

Before you begin

Ensure the following:

1. Download and install the  Connector.
2. Create a sandbox integration environment by using the Custom Integration template.

To run the example pipeline file

1. Log on to the host computer on which you have installed the BMC Helix Developer Tools Connector. For example, Test_host.
2. Go to the /opt/bmc/connectors/<connector-name>/data/<integration ID>/pipeline/example_pipeline/config directory. Example connector name is Custom_integration_demo. To know how to identify the integration id, host name, connector name details of the sandbox integration, see Oct_2021.

3. Run the following command to verify the current status of the Connector: 

   tail -f /opt/bmc/connectors/<connector-name>/logs/fluent.log

   The following output is displayed, which conveys that the Connector is up and running and logging the heartbeat periodically. 
   

4. Select the Configured Integrations tab.
5. Click the action menu of the sandbox integration that you have created, and select Edit. 
   
6. Select Events from the Entity Name drop-down list, and enter the nagios_events_pipeline.conf file name in the Entity Configuration File text box. 
   
7. Click Update. 
8. If the pipeline configuration is applied successfully, you will start seeing the following output:
   • Do the following steps to view the count of ingested events:
     1. Go to the Configured Integrations tab on the BMC Helix Developer Tools console.
     2. Locate the configured integration tile that you created by using the Custom Integration template. For example, Custom_Integration_Demo. You will be able to see the count of ingested events on the configured tile as shown in the following example:
        
   • Log on to the BMC Helix Operations Management console, and go to  Monitoring > Events to view the ingested events.
     
     

Pipeline customization examples

To understand the pipeline configuration files better and to learn its functionality, explore these configuration files in detail, modify them, and run. 

Example customization for the nagios_input.json file

The nagios_input.json file has a few input directives, such as the hostname from where to fetch the input data and a few other parameters. To experiment, you can modify any of these input parameters and save the file as explained in the following steps:

1. Go to the /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline/example_pipeline/config/ directory.
2. Using a text editor, open the nagios_input.json file.

3. By default, the nagios_input.json file has the following content.

   {"HOSTNAME" : "www.bmc.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK - Packet loss = 0%, RTA = 222.15 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=221.057999ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.bmc.com0"}
   
   
   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}
   
   
   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}

4. Modify the content. A sample modification is shown in the following example. Here the PING OK label is changed to PING OK Demo: 

   Sample modification

   {"HOSTNAME" : "www.bmc.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK Demo - Packet loss = 0%, RTA = 222.15 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=221.057999ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.bmc.com0"}
   
   

   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK Demo - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}
   
   

   {"HOSTNAME" : "www.timesofindia.indiatimes.com","Custom_key_PP" : "PING_status=1","HOSTOUTPUT":"PING OK Demo - Packet loss = 0%, RTA = 230.57 ms","SERVICEDESC" : "PING","SERVICESTATE" : "WARNING","SERVICEPERFDATA" : "rta=230.264008ms;100.000000;500.000000;0.000000 pl=0%;20;60;0","SERVICESTATETYPE" : "","SERVICEEXECUTIONTIME" : "","SERVICELATENCY" : "","SERVICEOUTPUT" : "","SERVICEPERFDATA":"www.timesofindia.indiatimes.com0"}

5. Save the file.

Example customization for the nagios_event_mapping.json file

The nagios_event_mapping.json file has defined the mapping details between input and output. To explore, you can modify any of these mapping details or add new mapping details and save the file as explained in the following steps:

1. Go to the /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline/example_pipeline/config/ directory.
2. Using a text editor, open the nagios_event_mapping.json file.

3. By default, the nagios_event_mapping.json file has the following content: 

   {
          "NagiosMappingDetails": [
                  {
                          "inputkey": "vl-aus-domdv019.bmc.com",
                          "outputkey": "source_identifier",
                          "type": "constant"
                  },
                                  {
                          "inputkey": "OPEN",
                          "outputkey": "status",
                          "type": "constant"
                  },
                  {
                          "inputkey": "HOSTOUTPUT",
                          "outputkey": "msg",
                          "type": "assignment"
                  },
                  {
                          "inputkey": "SERVICESTATE",
                          "outputkey": "severity",
                          "type": "assignment"
                  }
           ]
   }

4. Modify or add the mapping details. A sample modification is shown in the following example. Here, an additional mapping for the SERVICEDESC input key is added: 

   {
    "NagiosMappingDetails": [
    {
    "inputkey": "vl-aus-domdv019.bmc.com",
    "outputkey": "source_identifier",
    "type": "constant"
    },
    {
    "inputkey": "OPEN",
    "outputkey": "status",
    "type": "constant"
    },
    {
    "inputkey": "HOSTOUTPUT",
    "outputkey": "msg",
    "type": "assignment"
    },
    {

   "inputkey": "SERVICESTATE",
    "outputkey": "severity",
    "type": "assignment"
    },

   {
    "inputkey": "SERVICEDESC",
    "outputkey": "DemoDescription",
    "type": "assignment"
    }
    ]
    }

5. Save the file.

To reload the configurations to reflect the changes

Do the following steps to reload the configurations to reflect the configuration file customizations.

1. Go to the /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline/example_pipeline/ directory and run the following command: 

   #Syntax
   docker exec -it <Container ID> bash reload.sh
   
   #Example
   docker exec -it deed817ccb72 bash reload.sh
   #In the preceding example, deed817ccb72 is the container id.

   How do I find the Container ID?

   Run the following command to get the container ID:

   docker ps | grep <connector-name>

   The command displays the output in the following format. Note the CONTAINER ID from the output.

   CONTAINER ID   IMAGE          COMMAND      CREATED        STATUS       PORTS                NAMES deed817ccb72   57e8da5cb8ac   "/run.sh"    5 hours ago    Up 2 hours   5140/tcp,24224/tcp   generic_connector_test

Understand the nagios_events_pipeline.conf file structure

The nagios_events_pipeline.conf file has the configurations to run the events pipeline, which will get the data from the specified input resource, modify the data as per the mapping configurations, and send the data to 

Some content is unavailable due to permissions.

The following code block displays the nagios_events_pipeline.conf file content. There are three important directives in this file:

The following section explains the different directives and plug-ins used in this configuration file and its functionality:

Input directive — <source> </source>

The source directive is used to specify the source details such as the type of the plugin used, path, and user credentials. In this example, the path of the nagios_input.json and nagios_input.json.pos files are specified. 

Filter directive — <filter> </filter>

The filter directive specifies the sample data transformation plug-in that is used to convert data from a third-party product to the 

Some content is unavailable due to permissions.

Output directive — <match> </match>

The match directive is used to push the transformed data into

Some content is unavailable due to permissions.

For more information, see bmc-ade-http plug-in.

Where to go from here

After you have explored the example pipeline configuration files, create a new custom pipeline configuration file to ingest data from a product that you plan to integrate.