Ingesting data from OpenTelemetry

Before operators or site reliability engineers (SREs) can use telemetry data to correlate application issues with infrastructure problems, the data needs to be ingested into various BMC Helix applications, such as BMC Helix Operations Management, BMC Helix AIOps, and BMC Helix Dashboards.

Perform the following tasks to ingest data from OpenTelemetry and send it to BMC Helix applications: 

1. Instrument your application.
2. Install the OpenTelemetry Collector.
3. Configure the OpenTelemetry Collector and create a service model in BMC Helix AIOps. 
4. Verify the application topology elements in BMC Helix Discovery.
5. Add health indicators to the service model
6. View ingested traces and span events in BMC Helix Dashboards.
7. View ingested RED metrics in BMC Helix Dashboards. 
8. View ingested application-specific metrics in BMC Helix Dashboards.

Before you begin

OpenTelemetry uses OpenTelemetry Protocol (OTLP) to transmit telemetry data between components. Before ingesting data into BMC Helix applications, make sure that your network and firewalls allow connectivity to the following OTLP receiver ports on the OpenTelemetry Collector host:

Task 1: Instrument your application

Make sure that the application from which you want to receive telemetry data via OpenTelemetry has been instrumented. After an application is instrumented, it is able to emit traces and metrics. By using OpenTelemetry, you can instrument your application in the following ways:

• Code-based solutions: Get telemetry data from an application by using the OpenTelemetry APIs and SDKs, available for most languages.
• Zero-code solutions: Get telemetry data from libraries you use and the environment your application runs in.

For more information about using these methods to instrument your applications, see Instrumentation in the OpenTelemetry documentation.

For example, to instrument a Python application by using the zero-code solution method, perform the following tasks:

1. Install the appropriate packages, and initialize and bootstrap the instrumentation:
   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install
2. (Optional) Set additional metadata for an entity for its easy identification. For example, for a Kubernetes resource, you might want to set it as shown in the following example:
   export OTEL_RESOURCE_ATTRIBUTES="service.name=<serviceName>,k8s.cluster.name=<clusterName>,k8s.namespace.name=<namespaceName>"
3. Set the OTLP endpoint and protocol:export OTEL_EXPORTER_OTLP_ENDPOINT="http://<collectorHost>:4317"
   export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

   Replace <collectorHost> with the Collector host name or IP address. 

4. Set the service name:
   export OTEL_SERVICE_NAME="<serviceName>"
   If the service name is also provided in OTEL_RESOURCE_ATTRIBUTES, the value set for OTEL_SERVICE_NAME takes precedence. 
5. (Optional) Set Python’s standard logging module for instrumentation:
   export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
6. Run the application:
   opentelemetry-instrument python <applicationName>.py

For more information about instrumenting Python applications, see Python zero-code instrumentation in the OpenTelemetry documentation.

Task 2: To install the OpenTelemetry Collector

The OpenTelemetry Collector supports the following deployment modes:

• Agent: In this mode, the collector runs with the application or on the same host as the application and sends data directly to BMC Helix applications. 
• Gateway: In this mode, applications or other collectors send telemetry data to a single OTLP endpoint. This endpoint is provided by one or more collector instances running as a standalone service.  

For more information about deployment modes, see Deploy the Collector in the OpenTelemetry documentation.

You can install the collector on various platforms. For instructions on installing the collector, see the following topics in the OpenTelemetry documentation:

• Install the Collector with Docker
• Install the Collector with Kubernetes
• Install the Collector from a binary on Linux, macOS, and Windows

Task 3: To configure the OpenTelemetry Collector and create a service model in BMC Helix AIOps

The location and name of the entity that contains the collector configuration depend on the method used for installing the collector. Typically, in a standalone installation, the config.yaml file contains the collector configuration. The file is located in the /etc/<otelInstalldirectory> directory, where <otelInstalldirectory> can be otelcol, otelcol-contrib, or another value, depending on the collector version or the collector distribution you are using. In a cluster environment, for example, Kubernetes, the collector configuration is stored in a ConfigMap.

The collector configuration consists of the following components:

• Receivers: Receive telemetry data from the instrumented applications by using various protocols. The receivers section of the configuration file or ConfigMap contains information about the receivers required for your telemetry data.

• (Optional) Processors: Transform, filter, and enrich telemetry data collected by the receiver before sending it to the exporters. The processors section of the configuration file or  ConfigMap contains information about the processors required for your telemetry data.

  Warning

  Important

  BMC Helix AIOps does not support the use of the Transform processor when exporting telemetry data to BMC Helix applications. Advanced transformations performed by the Transform processor might significantly impact Collector performance.

• Exporters: Send telemetry data to one or more observability destinations or backends. The exporters section of the configuration file or ConfigMap contains information about the exporters required for your telemetry data. 

The service section in the configuration file or ConfigMap contains pipelines to enable receivers, processors, and exporters. A pipeline defines the components that work together to process telemetry data. Unless these components are added to the service section, they are not enabled. For more information about the collector components and the collector configuration, see Configuration in the OpenTelemetry documentation.

Before your begin

To create a service model that represents the application topology, either enable the out-of-the-box blueprint or create a blueprint manually.

To enable the out-of-the-box blueprint

1. Log on to BMC Helix AIOps.
2. Select Configurations > Manage Service Blueprints.
3. On the Service Blueprints page, from the action menu, select Import BMC default Blueprints.
4. Click Show Disabled. 
5. Search for the Default Blueprint for OTel Service blueprint.
6. Select Action > Enable to enable the blueprint.

To create the service blueprint manually

1. Log on to BMC Helix AIOps.
2. Select Configurations > Manage Service Blueprints.
3. On the Service Blueprints page, click Create Service Blueprint.
4. Specify the name, description, and provider (domain) details for the blueprint.
   For example, OTel Service Blueprint, Trace extraction, and .itbiz.com.
5. Click Next Step.
6. On the Edit Blueprint Rules tab, add CI Kinds and establish relationships between them:
   1. In the Available CI Kinds section, type External Element and select it from the list. 
   2. Add two more CI Kinds of type External Element.  
   3. Add one element of CI Kind of type Host.
   4. Establish a relationship between the selected CI Kinds, as shown in the following figure:
      
   5. Define the CI rule for the start node.
      1. Click the start node.
         On the Define CI Rule page, the default filter criterion shows the Name attribute of a CI added as a variable. When creating a service by using this blueprint, you need to provide this attribute as an input. 
      2. From Actions, select Edit. 
      3. On the Define Filter page, from Actions, select Edit. 
      4. In the Define user prompt message text box, update the message to Enter OTel Service Namespace.
      5. Click Add New Filter Criteria.
      6. In the Select attribute list, select Type.
      7. In the Select value(s) from existing data list, select OTel Service Namespace. 
      8. Click Save and Close.
         
            
   6. Define the link rule.
      1. Click the link between the start node and the second node.
      2. Select the Filter by Kind option.
      3. From the list, select Dependency.
         
      4. Click Save and Close.
      5. Click Save.
   7. Enable the blueprint:
      1. On the Service Blueprints page, click Show disabled to view the list of all disabled service blueprints.
      2. Search for the OTel Service Blueprint in the list. 
      3. Select Action > Enable to enable the blueprint for use in a service model.  

To configure the collector and create the service model

1. Log on to BMC Helix AIOps. 
2. Select Configurations > Manage OpenTelemetry.
3. Enter the X-Source name, a string identifier indicating the source of the telemetry data.
   For example, if your application, ITSM Services, is running on a Kubernetes namespace, enter itsm-kublet.
   A service will be created in BMC Helix AIOps with this name. 
4. Click Generate Configuration File.
   The collector configuration appears.
5. Copy the default collector configuration to a temporary file.

6. Review the configuration and update it if needed.
   The default collector configuration contains the following sections and parameters that you can update according to your requirements:

   Section/ Parameter	Description
   receivers > endpoint parameter	Specifies the collector port number. By default, OTLP uses port 4317 for gRPC and 4318 for HTTP. Update the port numbers if needed.
   processors section (Optional)	Specifies the processors to be used by the collector. Add processors and update their attributes if required.  For the list of recommended processors, see Recommended Processors in the OpenTelemetry documentation.
   exporters > endpoint parameter	Specifies the BMC Helix tenant URL.  Don't update this parameter.
   exporters > X-Source parameter	Specifies the string identifier indicating the source of the telemetry data that you have provided while generating the collector configuration. Don't update this parameter.
   exporters > X-Api-Key parameter	Specifies the API key required for authentication between the collector and BMC Helix applications in the following format: <tenant ID>::<access key>::<secret key>. For example, 1369852381::ATCRDJ4529QMBNCT7ZDES4079::bmwwPvuQyak3CgCIDV4WQEor1DT6R8hU8PiQPuulrR You can generate the tenant ID, access key, and secret key from BMC Helix Portal. For more information, see Setting up access keys for programmatic access. Don't update this parameter.
   service section	Specifies the names of the receivers, processors, and exporters for the type of data (traces and metrics) you want to collect and export. Entries in this section enable the components in the collector based on the configuration found in the receivers, processors, and exporters sections. Update this section if you have added any receivers, exporters, or processors.

7. Update your existing collector configuration file or ConfigMap with the reviewed collector configuration. 
   For a sample collector configuration, see Sample configuration file for an application deployed in a Kubernetes cluster.
8. Click Next.
9. Select the blueprint that you want to use to create the service model.
10. Click Finish.
    The service model is generated with the application name as the parent service name.

11. On the Services page, search for the service, and then click the service.
    The CI Topology tab shows the topology of the business service for the application instrumented by OpenTelemetry. Various application topology elements, such as services and namespaces, are represented as CIs in the service model.
    

Sample configuration file for an application deployed in a Kubernetes cluster

Here is a sample configuration file that collects traces and metrics from an application deployed in a Kubernetes cluster and exports them to BMC Helix applications. In addition, the Kubernetes Attribute processor (indicated by k8attributes in the file) automatically discovers Kubernetes pods, extracts their metadata, and adds it to spans and metrics as resource attributes. The processor correlates your application’s traces and metrics with your Kubernetes telemetry data, such as pod metrics and traces. For more information about the processor, see Important Components for Kubernetes in the OpenTelemetry documentation.

Task 4: To verify the application topology elements

Verify that the OpenTelemetry application topology elements are ingested into BMC Helix Discovery. 

Before you begin

Make sure that you have downloaded the latest Technology Knowledge Update (TKU) required to collect data from OpenTelemetry. To obtain the latest TKU, see Schedule and Roadmap. For download instructions, see Downloading TKU and Extended Data Packs from BMC Helix EPD site.

To verify the application topology elements in BMC Helix Discovery

1. Log on to BMC Helix Discovery.
2. Select Explore > Data.
3. In the Miscellaneous section, click External Elements.
4. Verify that the entries of type such as OTel Service, OTel Service Namespace, and OTel Host appear on the External Element List page.
   

Task 5: Add health indicators to the service model

Add health indicators to various OTel service CIs. For example, add Error as the health indicator to the product-reviews OTel Service CI.

1. Click Edit Service.
2. Click Health Indicators.
3. In the Define Health Indicators pane, click Add Health Indicator.
4. Select Metrics Applicable for this Service Content.
5. In the CI Name filter, enter product-reviews. 
6. Click product-reviews and then click Service.
7. In the Metric Name filter, enter Error.
8. Click Error (count).
9. From the Monitored HostName filter, select the option ending with product-reviews. 
   
10. Click Add Metric.
    The metric you added is displayed in the Added Metrics section.

11. In the Added Metrics section, click Create Alarm Policy to add the alarm policy for the listed metric in BMC Helix Operations Management. 
    For example, create a policy to generate an alarm event as soon as the Error (count) is above 2 for 1 min, as shown in the following figure:
    
    For more information about creating an alarm policy, see Configuring alarm policies.

12. Click Enable Policy.
13. Click Save in BMC Helix Operations Management.
14. Click Save and Close in BMC Helix AIOps.
    The metrics you have added appear in the Define Health Indicators pane as part of the first health indicator group.

15. Click Save and Close again.
    The count of health indicator groups is shown as part of the service. 

Task 6: To view traces and span events in BMC Helix Dashboards

View application traces and associated span events in the OTel Raw Trace Data dashboard.

1. Log on to BMC Helix Dashboards.
2. From the navigation menu , click Dashboards.
3. Search for the Helix OpenTelemetry Dashboards folder and select it.
4. Click OTel Raw Trace Data.
   The dashboard displays the ingested traces and span events (prefixed by Logs in the following figure).
   

Task 7: To view RED metrics in BMC Helix Dashboards

The Rate, Error, and Duration (RED) metrics (for example, Rate Trend and Error Trend) for an instrumented application are generated by traces. You can view these metrics in the OTel Service Overview dashboard.

1. Log on to BMC Helix Dashboards.
2. From the navigation menu, click Dashboards.
3. Search for the Helix OpenTelemetry Dashboards folder and select it.
4. Click OTel Service Overview.
   The Business Service list on the dashboard shows the business services for which OpenTelemetry is enabled.
5. From the Business Service list, select a business service (for example, OTel-Demo).
6. From the OTel Namespace list, select a namespace (for example, demo-latest).
7. From the OTel Service list, select an application service (for example, product-reviews).
   The dashboard shows the RED metrics (for example, Rate Trend and Error Trend) for the product-reviews service.
   

Task 8: To view application-specific metrics in BMC Helix Dashboards

The application-specific metrics are not available in an out-of-the-box dashboard. You can create your own dashboard and add these metrics to the dashboard. For more information about creating your own dashboards, see Creating a dashboard.

1. Log on to BMC Helix Dashboards.
2. Do one of the following:
   • On the Home page, click and select New Dashboard.
   • From the navigation menu, click Dashboards. Then click New > New Dashboard.
3. On the New dashboard page, click Add visualization.
   Alternatively, you can add an existing panel from the library panels. For more information, see Library panels.
4. In the Query section, select Metric from the Query type list.
5. Make sure that Monitor is selected from the Source list.
6. Click Metric Browser.
7. In the Select labels to search in list, click ServiceNamespace. 
8. Scroll down and from the ServiceNamespace list, select the service namespaces that contain the services for which you want to collect metrics.
9. In the Select labels to search in list, click ServiceName. 
10. Scroll down and from the ServiceName list, select the service (for example, api-gateway) for which you want to collect metrics.
11. From the Select a metric list, select the metric (for example, query.length) you want to view.
12. Click Use Query.
    The visualization shows the graph for the query.length metric: 
    
13. Add a name for the panel. 
14. Save the dashboard.

Where to go from here

Monitoring services and identifying application issues