Enabling BMC Helix applications to collect service traces from OpenTelemetry

BMC Helix AIOps connects with OpenTelemetry (also known as OTel) to help site reliability engineers (SREs) solve application errors. OpenTelemetry is a vendor-neutral, open-source observability framework to instrument, generate, collect, and export telemetry data such as traces, metrics, and logs. A trace describes the journey of a request through a distributed system. By viewing a trace, you can track the complete execution path of the request and identify which part of the application is causing issues such as errors and latency concerns.

Related topics

Identifying-bottlenecks-by-using-OpenTelemetry

A trace contains RED (Request, Errors, and Duration) metrics, which you can use to generate events in BMC Helix Operations Management by defining alarm generation criteria. For example, you can configure a policy to generate an event when the duration of any API call is more than 5 ms.

In BMC Helix AIOps, you create a business service model that represents the topology of the application. The events generated impact the health of the business service​​​, where you can identify the impacted application service. From the impacted application service, you can launch dashboards to analyze traces and identify the exact operation that is causing the issue.

To collect telemetry data from applications and send them to BMC Helix applications through OpenTelemetry, you can use the OpenTelemetry Collector or the OpenTelemetry SDKs.

Traces exported by OpenTelemetry are sent to BMC Helix applications through the BMC Helix OpenTelemetry service. To enable this service, contact BMC Support. 

Supported languages

BMC Helix AIOpssupports monitoring of all the applications that are developed in the languages supported by OpenTelemetry. For the list of languages that are supported by OpenTelemetry, see the OpenTelemetry documentation.

Supported OpenTelemetry Collector versions

BMC Helix AIOpssupports the following versions of OpenTelemetry Collector:

  • 0.105.0
  • 0.99.0
  • 0.54.4
  • 0.58.0

Supported metrics

In addition to the RED metrics, BMC Helix AIOps can ingest the following metric types from OpenTelemetry:

  • Counter: Represents a value that accumulates or increases over time, for example, the number of disk reads.
  • Gauge: Represents a current value at the time it is read, for example, the count of active connections on an application.
  • Histogram: Represents a client-side aggregation of values, for example, request latency.

You can create dashboards in BMC Helix Dashboards by using these metrics and monitor them as needed. In addition, you can create policies in BMC Helix Operations Management to generate an alarm when the value of a metric breaches a threshold.

To enable BMC Helix applications to collect service traces from OpenTelemetry

  1. Configure the OpenTelemetry Collector or use OpenTelemetry SDK. 
    For information about SDKs, see the OpenTelemetry documentation.
  2. Create an alarm policy in BMC Helix Operations Management.
  3. Verify the service topology in BMC Helix Discovery.
  4. (Optional) Use the out-of-the-box Default Blueprint for OTel Service blueprint (versions 25.1.01 and later) or create your own blueprint to create a service model in BMC Helix AIOps.
  5. Configure a service model in BMC Helix AIOps, by using a service blueprint or static content, and then monitor the business service.

Task 1: To configure the OpenTelemetry Collector

You can configure the OpenTelemetry Collector to suit your observability requirements. To enable the collector to export data to the BMC Helix applications, configure the following components of the collector:

  • Receiver: Receives traces data from instrumented applications by using various protocols.
  • (Optional) Processor: Takes the data collected by the receiver and modifies or transforms it before sending it to the exporter. 
  • Exporter: Exports traces data to one or more destinations.

For more information about the collector components, see the OpenTelemetry documentation.

Before you begin

Before you configure the collector, make sure that the application from which you want to receive traces via OpenTelemetry has been instrumented, which means code from the application's components is able to emit traces. For more information, see the OpenTelemetry documentation.

To configure the collector

You can deploy the collector on a variety of operating systems and architectures. For information, see the OpenTelemetry documentation.

Location and name of the file that contains the collector configuration depends on the installation method of the collector. For example, in a standalone environment, typically, the config.yaml file contains the collector configuration. In a Kubernetes environment, collector configuration is stored in a ConfigMap. For more information, see OpenTelemetry documentation.

  1. Open the collector configuration file (for example, config.yaml or ConfigMap).
    The following example shows a configuration file that you can use to export traces to BMC Helix applications.

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:<portNumber>
          http:
            endpoint: 0.0.0.0:<portNumber>

    processors:
      batch:
      memory_limiter:
         check_interval: 5s
         limit_percentage: 80
         spike_limit_percentange: 25

    exporters:
      # Console exporters for debugging
      logging:
        loglevel: info
     
      # OTLP exporters - update this to have the bmc helix exporter defined here
      otlp/bmchelix:
      endpoint: <tenantURL>
        headers:
            X-Api-Key:<apiKey>
        sending_queue:
            enabled: true
            num_consumers: 100
            queue_size: 10000
     
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging, otlp/bmchelix]
       
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging, otlp/bmchelix]
       
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging, otlp/bmchelix]
  2. In the file, configure the properties in the following sections:
    • receivers: Update endpoint with the OpenTelemetry Collector port number.
    • (optional) processors: Update the attributes for various processors. 
       
    • exporters: Update the following properties:
      • endpoint – BMC Helix tenant URL. For example, https://htel-http-trace-service.abc.com.

      • X-Api-Key – API key required for authentication between the collector and BMC Helix applications in the following format: <tenant ID>::<access key>::<secret key>::<source>

        • <tenant ID>, <access key>, <secret key>: Log on to BMC Helix Portal and generate the tenant ID, access key, and secret key. For more information, see Setting up access keys for programmatic access.  

        • <source>: A string identifier indicating the source of traces. For example, if your application, ITSM Services is running on a Kubernetes namespace, enter itsm-kublet.

        For example, 1369852381::ATCRDJ4529QMBNCT7ZDES4079::bmwwPvuQyak3CgCIDV4WQEor1DT6R8hU8PiQPuulrR::itsm-kublet.

    • service: Update the section with the names of the receivers, processors, and exporters for the type of data (traces, metrics, or logs) you want to collect and export. Entries in this section enable the components in the collector based on the configuration found in the receivers,  processors, and exporters sections.

Task 2: To create an alarm policy

Create an alarm policy in BMC Helix Operations Management that generates events when the value of a trace or operation metric is greater or less than a specified value. 

  1. Log on to BMC Helix Operations Management.
  2. Select Configuration > Alarm Policies.
  3. Click Create.
  4. Add a unique precedence number to the policy.
    You can add a custom value in this field, or use the arrows to increase or decrease the value.

  5. In the Alarm Generation Conditions section, enter the following information:

    Configuration

    Description

    Attribute to monitor

    Select the operation-level or trace-level metric for which you want to add this policy, for example, Apdex.

    Instance details

    Select All Instances or Multiple Instances as needed.

    Threshold conditions

    Select the threshold value, violation duration, and details about when the generated alarm must be closed eventually. Also, specify if the event must be closed immediately after it is generated or after the metric reaches a normal state and a duration after the violation time period has lapsed. You can specify that the event must not be closed. Alarm events that are not closed remain open until they are closed manually, the policy is deleted, or the PATROL Agent associated with the alarm is deleted.

    By specifying conditions, you can get notified of event alerts when a metric violates the baseline value. To change any of the values, click them.

    For example, if the metric value is below 0.98 score for more than one min, then generate major alarm and do not close the alarm.

    The following figure shows a sample alarm generation condition:
    OTel_AlarmGenerationCondition_24102.png

  6. Optional) Select Enable Policy.
    You can enable or disable the policy any time from the Alarm Policies page. For more information, seeConfiguring alarm policies.

  7. Save the policy.

Task 3: To verify the service topology

Verify that the OpenTelemetry application topology is ingested into BMC Helix Discovery. 

Before you begin

Make sure that you have obtained the latest The Pattern Language (TPL). For details, contact BMC Support.

To verify the service topology in BMC Helix Discovery

  1. Log on to BMC Helix AIOps.
  2. Select Explore > Data.
  3. In the Miscellaneous section, click External Elements.
  4. Verify that the entries of type OTel Span, OTel Host, OTel Service, and OTel Service Namespace appear on the External Element List page.
    OTel_Discovery_24102.png

(Optional) Task 4: To create a service blueprint

To create a service model from a service blueprint in Task 5, you can either use the out-of-the-box Default Blueprint for OTel Service blueprint (versions 25.1.01 and later) or create a blueprint manually. 

To use the out-of-the-box blueprint

  1. Log on to BMC Helix AIOps.
  2. Select Configurations > Manage Service Blueprints.
  3. On the Service Blueprints page, from the action menuaction_menu.png, select Import BMC default Blueprints.
  4. Click Show Disabled. 
  5. Search for the Default Blueprint for OTel Service blueprint.
  6. Select Action > Enable to enable the blueprint.

To create the service blueprint manually

  1. Log on to BMC Helix AIOps.
  2. Select Configurations > Manage Service Blueprints.
  3. On the Service Blueprints page, click Create Service Blueprint.
  4. Specify the name, description, and provider (domain) details for the blueprint.
    For example, OTel Service Blueprint, Trace extraction, and .itbiz.com.
  5. Click Next Step.
  6. On the Edit Blueprint Rules tab, add CI Kinds and establish relationships between them:
    1. In the Available CI Kinds section, type External Element and select it from the list. 
    2. Add two more CI Kinds of type External Element.  
    3. Add one element of CI Kind of type Host.
    4. Establish relationship between the selected CI Kinds, as shown in the following figure:
      OTelBlueprint_Define_24102.png
    5. Define the CI rule for the start node.
      1. Click the start node.
        On the Define CI Rule page, the default filter criterion shows the Name attribute of a CI added as a variable. When creating a service by using this blueprint, you need to provide this attribute as an input. 
      2. From Actions, select Edit. 
      3. On the Define Filter page, from Actions, select Edit. 
      4. In the Define user prompt message text box, update the message to Enter OTel Service Namespace.
      5. Click Add New Filter Criteria.
      6. In the Select attribute list, select Type.
      7. In the Select value(s) from existing data list, select OTel Service Namespace. 
      8. Click Save and Close.
        DefineCIRule.png
           
    6. Define the link rule.
      1. Click the link between the start node and the second node.
      2. Select the Filter by Kind option.
      3. From the list, select Dependency.
        DefineLinkRule.png
      4. Click Save and Close.
      5. Click Save.
    7. Enable the blueprint:
      1. On the Service Blueprints page, click Show disabled to view the list of all disabled service blueprints.
      2. Search for the OTel Service Blueprint in the list. 
      3. Select Action > Enable for the blueprint to enable the blueprint to be used as part of a service model.  

Task 5: To create a service model and monitor the business service

Create a service model by using a service blueprint or static content. For information about creating a service model by using the static content, see Creating service models.

Important

To create a service model automatically in BMC Helix AIOps, leverage the POST method for the /models endpoint provided by BMC Helix Discovery APIs. For more information about the endpoint, see Endpoints in the REST API.

To create a service model by using a service blueprint

  1. Log on to BMC Helix AIOps.
  2. Click Services and then click Create New Service.
  3. In the Define Service pane, specify the service name, for example, ITSM Service.
  4. (Optional) Assign a kind to the service.
    For more information about service kinds, see Business services, technical services, and business applications.
  5. (Optional) Select service criticality from the list to mark the importance of the business service.
  6. Click Add Dynamic content.
  7. From the blueprints list, select Default Blueprint for OTel Service (out-of-the-box blueprint) or OTel Service Blueprint that you have created in Task 4. 
  8. Select the required OpenTelemetry namespace. 
  9. Click Save.
  10. Click Save and Close.
  11. On the Services page, search for the service and then click the service.
    The CI Topology tab shows the topology of the business service for the application instrumented by OpenTelemetry.
    Otel_SampleServiceTopology.png

After the service model is created, you can start monitoring the application services through the service model. For more information, see Monitoring-service-health.

Where to go from here

View and analyze traces by using the following dashboards:

 

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*