Ingesting data from OpenTelemetry

Before operators or site reliability engineers (SREs) can use telemetry data to correlate application issues with infrastructure problems, the data needs to be ingested into various BMC Helix applications, such as BMC Helix Operations Management, BMC Helix AIOps, and BMC Helix Dashboards.

Perform the following tasks to ingest data from OpenTelemetry and send it to BMC Helix applications:

1. Instrument your application.
2. Install the OpenTelemetry Collector.
3. Configure the OpenTelemetry Collector. 
4. Verify the application topology elements in BMC Helix Discovery.
5. Enable an out-of-the-box service blueprint or create your own service blueprint.
6. Create a service model in BMC Helix AIOps by using the service blueprint.
7. View ingested traces and span events in BMC Helix Dashboards.
8. View ingested RED metrics in BMC Helix Dashboards. 
9. View ingested application-specific metrics in BMC Helix Dashboards.

Before you begin

OpenTelemetry uses OpenTelemetry Protocol (OTLP) to transmit telemetry data between components. Before ingesting data into BMC Helix applications, make sure that your network and firewalls allow connectivity to the following OTLP receiver ports on the OpenTelemetry Collector host:

Task 1: Instrument your application

Make sure that the application from which you want to receive telemetry data via OpenTelemetry has been instrumented. After an application is instrumented, it is able to emit traces and metrics. By using OpenTelemetry, you can instrument your application in the following ways:

• Code-based solutions: Get telemetry data from an application by using the OpenTelemetry APIs and SDKs, available for most languages.
• Zero-code solutions: Get telemetry data from libraries you use and the environment your application runs in.

For more information about using these methods to instrument your applications, see Instrumentation in the OpenTelemetry documentation.

For example, to instrument a Python application by using the zero-code solution method, perform the following tasks:

1. Install the appropriate packages, and initialize and bootstrap the instrumentation:
   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
2. (Optional) Set additional metadata for an entity for its easy identification. For example, for a Kubernetes resource, you might want to set it as shown in the following example:
   export OTEL_RESOURCE_ATTRIBUTES="service.name=<serviceName>,k8s.cluster.name=<clusterName>,k8s.namespace.name=<namespaceName>"
3. Set the OTLP endpoint and protocol:export OTEL_EXPORTER_OTLP_ENDPOINT="http://<collectorHost>:4317"
   export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

   Replace <collectorHost> with the Collector host name or IP address. 

4. Set the service name:
   export OTEL_SERVICE_NAME="<serviceName>"
   If the service name is also provided in OTEL_RESOURCE_ATTRIBUTES, the value set for OTEL_SERVICE_NAME takes precedence. 
5. (Optional) Set Python’s standard logging module for instrumentation:
   export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
6. Run the application:
   opentelemetry-instrument python <applicationName>.py

For more information about instrumenting Python applications, see Python zero-code instrumentation in the OpenTelemetry documentation.

Task 2: To install the OpenTelemetry Collector

The OpenTelemetry Collector supports the following deployment modes:

• Agent: In this mode, the collector runs with the application or on the same host as the application and sends data directly to BMC Helix applications. 
• Gateway: In this mode, applications or other collectors send telemetry data to a single OTLP endpoint. This endpoint is provided by one or more collector instances running as a standalone service. 

For more information about the deployment modes, see Deploy the Collector in the OpenTelemetry documentation.

You can install the collector on various platforms. For instructions on installing the collector, see the following topics in the OpenTelemetry documentation:

• Install the Collector with Docker
• Install the Collector with Kubernetes
• Install the Collector from a binary on Linux, macOS, and Windows

Task 3: To configure the OpenTelemetry Collector

The location and name of the entity that contains the collector configuration depend on the method used for installing the collector. Typically, in a standalone installation, the config.yaml file contains the collector configuration. The file is located in the /etc/<otelInstalldirectory> directory, where <otelInstalldirectory> can be otelcol, otelcol-contrib, or another value, depending on the collector version or the collector distribution you are using. In a cluster environment, for example, Kubernetes, the collector configuration is stored in a ConfigMap.

To enable the collector to export data to the BMC Helix applications, configure the following components of the collector:

• Receivers: Receive telemetry data from the instrumented applications by using various protocols. Configure the receivers required for your telemetry data in the receivers section of the configuration file or ConfigMap.
• (Optional) Processors: Transform, filter, and enrich telemetry data collected by the receiver before sending it to the exporters. Configure the processors required for your telemetry data in the processors section of the configuration file or  ConfigMap.
• Exporters: Send telemetry data to one or more observability destinations or backends. Configure the exporters required for your telemetry data in the exporters section of the configuration file or  ConfigMap.

After you configure a component, you need to enable it by using pipelines in the service section. A pipeline defines the components that work together to process telemetry data. For more information about the collector components and the collector configuration, see Configuration in the OpenTelemetry documentation.

To configure the collector

1. Open the collector configuration file or ConfigMap.
   Here is a configuration file that you can use as a template. The file is configured to collect traces and metrics from an application and export them to BMC Helix applications.

   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:<portNumber>
         http:
           endpoint: 0.0.0.0:<portNumber>
   
   processors:
     batch:
   
   exporters:
     # Console exporters for debugging
     logging:
       loglevel: info
    
     # OTLP exporters - update this to have the bmc helix exporter defined here
     otlphttp/bmchelix:
       endpoint: <tenantURL>
       headers:
           X-Source:<originTelemetryData>   
           X-Api-Key:<apiKey>        
       sending_queue:
           enabled: true
           num_consumers: 100
           queue_size: 10000
    
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [logging]
      
       metrics:
         receivers: [otlp]
         processors: [batch]
         exporters: [logging, otlphttp/bmchelix]
2. In the file, configure the properties in the following sections:
   • receivers: Update endpoint with the collector port number.
     By default, OTLP uses port 4317 for gRPC and 4318 for HTTP.
   • (Optional) processors: Update the attributes for processors. 
     For the list of processors recommended by OpenTelemetry, see Recommended Processors in the OpenTelemetry documentation.
   • exporters: Update the following properties:
     • endpoint – BMC Helix tenant URL. For example, https://otel-itom.onbmc.com.
     • X-Source – A string identifier indicating the source of the telemetry data.
       For example, if your application, ITSM Services, is running on a Kubernetes namespace, enter itsm-kublet.

     • X-Api-Key – API key required for authentication between the collector and BMC Helix applications in the following format:
       <tenant ID>::<access key>::<secret key>.
       For example, 1369852381::ATCRDJ4529QMBNCT7ZDES4079::bmwwPvuQyak3CgCIDV4WQEor1DT6R8hU8PiQPuulrR
       You can generate the tenant ID, access key, and secret key from BMC Helix Portal. For more information, see Setting up access keys for programmatic access.

   • service: Update the section with the names of the receivers, processors, and exporters for the type of data (traces and metrics) you want to collect and export. Entries in this section enable the components in the collector based on the configuration found in the receivers,  processors, and exporters sections.

Sample configuration file for an application deployed in a Kubernetes cluster

Here is a sample configuration file that collects traces and metrics from an application deployed in a Kubernetes cluster and exports them to BMC Helix applications. In addition, the Kubernetes Attribute processor (indicated by k8attributes in the file) automatically discovers Kubernetes pods, extracts their metadata, and adds it to spans and metrics as resource attributes. The processor correlates your application’s traces and metrics with your Kubernetes telemetry data, such as pod metrics and traces. For more information about the processor, see Important Components for Kubernetes in the OpenTelemetry documentation.

Task 4: To verify the application topology elements

Verify that the OpenTelemetry application topology elements are ingested into BMC Helix Discovery. 

Before you begin

Make sure that you have downloaded the latest Technology Knowledge Update (TKU) required to collect data from OpenTelemetry. To obtain the latest TKU, see Schedule and Roadmap. For download instructions, see Downloading TKU and Extended Data Packs from BMC Helix EPD site.

To verify the application topology elements in BMC Helix Discovery

1. Log on to BMC Helix Discovery.
2. Select Explore > Data.
3. In the Miscellaneous section, click External Elements.
4. Verify that the entries of type such as OTel Service, OTel Service Namespace, and OTel Host appear on the External Element List page.
   

Task 5: To enable a service blueprint or create a service blueprint

To create a service model from a service blueprint in Task 6, you can either use the out-of-the-box Default Blueprint for OTel Service blueprint or create a blueprint manually.

To enable the out-of-the-box blueprint

1. Log on to BMC Helix AIOps.
2. Select Configurations > Manage Service Blueprints.
3. On the Service Blueprints page, from the action menu, select Import BMC default Blueprints.
4. Click Show Disabled. 
5. Search for the Default Blueprint for OTel Service blueprint.
6. Select Action > Enable to enable the blueprint.

To create the service blueprint manually

1. Log on to BMC Helix AIOps.
2. Select Configurations > Manage Service Blueprints.
3. On the Service Blueprints page, click Create Service Blueprint.
4. Specify the name, description, and provider (domain) details for the blueprint.
   For example, OTel Service Blueprint, Trace extraction, and .itbiz.com.
5. Click Next Step.
6. On the Edit Blueprint Rules tab, add CI Kinds and establish relationships between them:
   1. In the Available CI Kinds section, type External Element and select it from the list. 
   2. Add two more CI Kinds of type External Element.  
   3. Add one element of CI Kind of type Host.
   4. Establish a relationship between the selected CI Kinds, as shown in the following figure:
      
   5. Define the CI rule for the start node.
      1. Click the start node.
         On the Define CI Rule page, the default filter criterion shows the Name attribute of a CI added as a variable. When creating a service by using this blueprint, you need to provide this attribute as an input. 
      2. From Actions, select Edit. 
      3. On the Define Filter page, from Actions, select Edit. 
      4. In the Define user prompt message text box, update the message to Enter OTel Service Namespace.
      5. Click Add New Filter Criteria.
      6. In the Select attribute list, select Type.
      7. In the Select value(s) from existing data list, select OTel Service Namespace. 
      8. Click Save and Close.
         
            
   6. Define the link rule.
      1. Click the link between the start node and the second node.
      2. Select the Filter by Kind option.
      3. From the list, select Dependency.
         
      4. Click Save and Close.
      5. Click Save.
   7. Enable the blueprint:
      1. On the Service Blueprints page, click Show disabled to view the list of all disabled service blueprints.
      2. Search for the OTel Service Blueprint in the list. 
      3. Select Action > Enable for the blueprint to enable the blueprint to be used as part of a service model.  

Task 6: To create a service model

Create a service model by using a service blueprint or static content. For information about creating a service model by using the static content, seeConfiguring alarm policies.

To create a service model by using a service blueprint

1. Log on to BMC Helix AIOps.
2. Click Services and then click Create New Service.
3. In the Define Service pane, specify the service name, for example, ITSM Service.
4. (Optional) Assign a kind to the service.
   For more information about service kinds, see Endpoints in the REST API.
5. (Optional) Select service criticality from the list to mark the importance of the business service.
6. Click Add Dynamic content.
7. From the blueprints list, select Default Blueprint for OTel Service (out-of-the-box blueprint) or OTel Service Blueprint that you have created in Task 5. 
8. Select the required OpenTelemetry namespace. 
9. Click Save.
10. Click Save and Close.
11. On the Services page, search for the service, and then click the service.
    The CI Topology tab shows the topology of the business service for the application instrumented by OpenTelemetry. Various application topology elements, such as services and namespaces, are represented as CIs in the service model.
    
12. Add health indicators to various OTel Service CIs. For example, add Error as the health indicator to the product-reviews OTel Service CI.
    1. Click Edit Service.
    2. Click Health Indicators.
    3. In the Define Health Indicators pane, click Add Health Indicator.
    4. Select Metrics Applicable for this Service Content.
    5. In the CI Name filter, enter product-reviews. 
    6. Click product-reviews and then click Service.
    7. In the Metric Name filter, enter Error.
    8. Click Error (count).
    9. From the Monitored HostName filter, select the option ending with product-reviews. 
       
    10. Click Add Metric.  
        The metric you added is displayed in the Added Metrics section.

    11. In the Added Metrics section, click Create Alarm Policy to add the alarm policy for the listed metric in BMC Helix Operations Management. 
        For example, create a policy to generate an alarm event as soon as the Error (count) is above 2 for 1 min, as shown in the following figure:
        
        For more information about creating an alarm policy, see Configuring alarm policies.

    12. Click Enable Policy.

    13. Click Save in BMC Helix Operations Management.

13. Click Save and Close in BMC Helix AIOps.
    The metrics you have added appear in the Define Health Indicators pane as part of the first health indicator group.

14. Click Save and Close again.
    The count of health indicator groups is shown as part of the service. 

Task 7: To view traces and span events in BMC Helix Dashboards

View application traces and associated span events in the OTel Raw Trace Data dashboard.

1. Log on to BMC Helix Dashboards.
2. From the navigation menu , click Dashboards.
3. Search for the Helix OpenTelemetry Dashboards folder and select it.
4. Click OTel Raw Trace Data.
   The dashboard displays the ingested traces and span events (prefixed by Logs in the following figure).
   

Task 8: To view RED metrics in BMC Helix Dashboards

The Rate, Error, and Duration (RED) metrics (for example, Rate Trend and Error Trend) for an instrumented application are generated by traces. You can view these metrics in the OTel Service Overview dashboard.

1. Log on to BMC Helix Dashboards.
2. From the navigation menu, click Dashboards.
3. Search for the Helix OpenTelemetry Dashboards folder and select it.
4. Click OTel Service Overview.
   The Business Service list on the dashboard shows the business services for which OpenTelemetry is enabled.
5. From the Business Service list, select a business service (for example, OTel-Demo).
6. From the OTel Namespace list, select a namespace (for example, demo-latest).
7. From the OTel Service list, select an application service (for example, product-reviews).
   The dashboard shows the RED metrics (for example, Rate Trend and Error Trend) for the product-reviews service.
   

Task 9: To view application-specific metrics in BMC Helix Dashboards

The application-specific metrics are not available in an out-of-the-box dashboard. You can create your own dashboard and add these metrics to the dashboard. For more information about creating your own dashboards, see Creating a dashboard.

1. Log on to BMC Helix Dashboards.
2. Do one of the following:
   • On the Home page, click and select New Dashboard.
   • From the navigation menu, click Dashboards. Then click New > New Dashboard.
3. On the New dashboard page, click Add visualization.
   Alternatively, you can add an existing panel from the library panels. For more information, see Library panels.
4. In the Query section, select Metric from the Query type list.
5. Make sure that Monitor is selected from the Source list.
6. Click Metric Browser.
7. In the Select labels to search in list, click ServiceNamespace. 
8. Scroll down and from the ServiceNamespace list, select the service namespaces that contain the services for which you want to collect metrics.
9. In the Select labels to search in list, click ServiceName. 
10. Scroll down and from the ServiceName list, select the service (for example, api-gateway) for which you want to collect metrics.
11. From the Select a metric list, select the metric (for example, query.length) you want to view.
12. Click Use Query.
    The visualization shows the graph for the query.length metric: 
    
13. Add a name for the panel. 
14. Save the dashboard.

Where to go from here

Monitoring services and identifying application issues