Exploring the default sandbox directory structure and configuration files

After you have created a sandbox environment by using the Custom Integration template, explore the default directories and pipeline configuration files that are available in the sandbox. Understand how these pipeline configuration files are designed.

Before you begin

To identify the sandbox integration details

The following section helps you determine the details of the sandbox integration that you created by using the Custom Integration template.

On BMC Helix Portal, click
BMC Helix Developer Tools
.
Click Configured integrations.
Locate the integration that you configured by using the Custom Integration template; for example, Custom_Integration_Demo.
Click the action menu, and select Copy ID to copy the integration ID of the sandbox integration.
Make a note of this integration ID, which will be used in the next section.
Click the info icon on the integration tile, and note the connector name and host name on which the connector is running.
In the following example, the host name is Test-host, and the connector name is Custom_integration_demo.

To explore the out-of-the-box directories and pipeline configuration files available in the sandbox environment

Log in to the host computer whose host name and connector name you noted down in the previous section.
Go to the /opt/bmc/connectors/<connector name> directory.
For example, /opt/bmc/connectors/Custom_integration_demo
The /opt/bmc/connectors/<connector name> directory contains the following directories and configuration files out-of-the-box. You need these files and directories while creating a custom third-party integration.

For more information about the directory structure, see Connectors-templates-plug-ins-and-pipelines.

To learn about the design of the pipeline configuration files

The /opt/bmc/connectors/<connector name>/data/<integration ID>/pipeline directory contains the following out-of-the-box pipeline configuration files. You can create a copy of these files and modify according to your requirement:

generic_entity_pipeline.conf
generic_event_pipeline.conf
generic_metric_pipeline.conf

To understand these configuration files, you need to learn how a pipeline is designed in this context and what are its significant elements. Consider the generic_event_pipeline.conf file for analysis. This pipeline configuration file contains directives for enabling the data to be ingested, transformed, and correlated into a model for performing ML-based analytics. The following diagram represents the pipeline design used in the generic_event_pipeline.conf file.

The following code block displays the default content of the generic_event_pipeline.conf file:

#Fluentd Input Plugin:
#Input plugins extend Fluentd to retrieve and pull records from external sources. An input plugin typically creates a thread, socket, and listening socket. It can also be written to periodically pull data from the data sources.
#
#Example: Fluentd http_pull plugin.
#<source>
# @type http_pull
# tag Generic_demo_test_GENERICEVENTS
# url http://targetURL
# interval 20s
# verify_ssl false
# format json
# user user1
# password test123
#</source>
#For more Input plugins, check URL - https://www.fluentd.org/plugins/all#input-output
#
#Fluentd Filter Plugin:
#Filter plugins enable Fluentd to modify records stream. Filtering out records by grepping the value of one or more fields. Enriching records by adding new custom fields. Deleting certain fields for privacy and compliance. It can also change format align with output required
#<filter Generic_demo_test_GENERICEVENTS>
#</filter>
#For more Filter plugins, check URL - https://www.fluentd.org/plugins/all#filter
#
#To process records to ADE event rest endpoint, records must be in the below format after all conversions are performed.
#[{"class":"GenericEvent","eventType":"RESOURCE_CONTENTION","_ci_id":"HOST-9785505BCFCF9B50","source_identifier":"Generic-a1cff084-fcbc-4ed3-a238-7747fbb8b14f-HOST-9785505BCFCF9B50","source_hostname":"zwl-np-tspsl-1.onbmc.com","entityName":"zwl-np-tspsl-1.onbmc.com","sourceEventId":"6451525205044286870","msg":"RESOURCE_CONTENTION -- zwl-np-tspsl-1.onbmc.com","status":"OPEN","_identifier":"6451525205044286870_1619090277176","impactLevel":"INFRASTRUCTURE","source":"Custom","creation_time":1619090277176,"severityLevel":"RESOURCE_CONTENTION","annotationDescription":"CPU Saturation observed in the Host PG 002","correlationId":"b2f06778562f4689","severity":"MINOR"}]
#Please check the link for better understanding the API - https://docs.bmc.com/docs/HelixOperationsManagement/managing-events-with-rest-apis-955525274.html
#
#Before sending records to ADE event rest endpoint, check the above final format of records using the below filter plugin.
#Final records along with tag in the fluent.log file.
#After following filter executed it will log the final record along with tag Generic_demo_test_GENERICEVENTS in fluent log its format should match to example above.

<filter Generic_demo_test_GENERICEVENTS>
@type stdout
</filter>

<match Generic_demo_test_GENERICEVENTS>
       @type bmc_ade_http
       endpoint_url    <BMC Helix Portal URL>/events-service/api/v1.0/events
       ssl_no_verify   true # default: false
       use_ssl         true
       http_method     post    # default: post
       serializer      json   # default: form
       rate_limit_msec 0    # default: 0 = no rate limiting
       raise_on_error true # default: true
       recoverable_status_codes 503, 400 # default: 503
       format           json
           audit_payload {"labels":{"metricName":"Events","hostname":"${{CONNECTOR_ENV.HOST_NAME}}","entityId":"${{CONNECTOR_ENV.AGENT_ID}}","entityType":"Connector","entityName":"${{CONNECTOR_ENV.AGENT_NAME}}","IntegrationID":"ae44843b-4bdc-4835-89f7-b5d134c60d8c","IntegrationName":"Generic_demo_test","unit":"Nos.","source":"Generic"},"samples":[{"value":0,"timestamp":0}]}
</match>

The following section explains the different directives and plug-ins used in this configuration file and its functionality:

Input directive — <source> </source>

The source directive is used to specify the source details such as the type of the plugin used, url, and user credentials.

For more information about the http_pull plug-in, see Custom-plug-ins-used-in-BMC-Helix-Developer-Tools.

Filter directive — <filter> </filter>

The filter directive specifies the sample data transformation plug-in that is used to convert data from a third-party product to the BMC Helix Platform compatible format:

#Fluentd Filter Plugin:
#Filter plugins enable Fluentd to modify records stream. Filtering out records by grepping the value of one or more fields. Enriching records by adding new custom fields. Deleting certain fields for privacy and compliance. It can also change format align with output required
#<filter Generic_demo_test_GENERICEVENTS>
#</filter>
#For more Filter plugins, check URL - https://www.fluentd.org/plugins/all#filter
#
#To process records to ADE event rest endpoint, records must be in the below format after all conversions are performed.
#[{"class":"GenericEvent","eventType":"RESOURCE_CONTENTION","_ci_id":"HOST-9785505BCFCF9B50","source_identifier":"Generic-a1cff084-fcbc-4ed3-a238-7747fbb8b14f-HOST-9785505BCFCF9B50","source_hostname":"zwl-np-tspsl-1.onbmc.com","entityName":"zwl-np-tspsl-1.onbmc.com","sourceEventId":"6451525205044286870","msg":"RESOURCE_CONTENTION -- zwl-np-tspsl-1.onbmc.com","status":"OPEN","_identifier":"6451525205044286870_1619090277176","impactLevel":"INFRASTRUCTURE","source":"Custom","creation_time":1619090277176,"severityLevel":"RESOURCE_CONTENTION","annotationDescription":"CPU Saturation observed in the Host PG 002","correlationId":"b2f06778562f4689","severity":"MINOR"}]
#Please check the link for better understanding the API - https://docs.bmc.com/docs/HelixOperationsManagement/managing-events-with-rest-apis-955525274.html
#
#Before sending records to ADE event rest endpoint, check the above final format of records using the below filter plugin.
#Final records along with tag in the fluent.log file.
#After following filter executed it will log the final record along with tag Generic_demo_test_GENERICEVENTS in fluent log its format should match to example above.

<filter Generic_demo_test_GENERICEVENTS>
@type stdout
</filter>

Output directive — <match> </match>

The match directive is used to push the transformed data into BMC Helix Platform.

<match Generic_demo_test_GENERICEVENTS>
       @type bmc_ade_http
       endpoint_url    <BMC Helix Portal URL>/events-service/api/v1.0/events
       ssl_no_verify   true # default: false
       use_ssl         true
       http_method     post    # default: post
       serializer      json   # default: form
       rate_limit_msec 0    # default: 0 = no rate limiting
       raise_on_error true # default: true
       recoverable_status_codes 503, 400 # default: 503
       format           json
           audit_payload {"labels":{"metricName":"Events","hostname":"${{CONNECTOR_ENV.HOST_NAME}}","entityId":"${{CONNECTOR_ENV.AGENT_ID}}","entityType":"Connector","entityName":"${{CONNECTOR_ENV.AGENT_NAME}}","IntegrationID":"ae44843b-4bdc-4835-89f7-b5d134c60d8c","IntegrationName":"Generic_demo_test","unit":"Nos.","source":"Generic"},"samples":[{"value":0,"timestamp":0}]}
</match>

For more information about the parameters used in the plug-in, see bmc-ade-http-plug-in.

Where to go from here

After you have explored the directory structure and the out-of-the-box configuration files available with the sandbox integration environment and understood the design of the pipeline configuration files, run an example pipeline available in the sandbox integration environment.