Enabling BMC Helix applications to collect service traces from OpenTelemetry

BMC Helix AIOps connects with OpenTelemetry (also known as OTel) to help site reliability engineers (SREs) solve application errors. OpenTelemetry is a vendor-neutral, open-source observability framework to instrument, generate, collect, and export telemetry data such as traces. A trace describes the journey of a request through a distributed system. By viewing a trace, you can track the complete execution path of request and identify which part of the application is causing issues such as errors and latency concerns. 

Related topics

Identifying-bottlenecks-in-the-system-by-leveraging-OpenTelemetry-tracing


Traces exported by the OpenTelemetry Collector are sent to BMC Helix applications through the BMC Helix OpenTelemetry service. To enable this service, contact BMC Support. 

A trace contains RED (Request, Errors, and Duration) metrics, which you can use to generate events in BMC Helix Operations Management by defining alarm generation criteria. For example, you can configure a policy to generate an event when the duration of any API call is more than 5 ms. These events impact a service health in BMC Helix AIOps, where you can identify the impacted application service. You can launch dashboards from BMC Helix AIOps to analyze traces and identify the exact operation that is causing the issue.

To enable BMC Helix applications to collect service traces from OpenTelemetry

Perform the following tasks to enable BMC Helix applications to collect service traces from OpenTelemetry:

  1. Configure the OpenTelemetry Collector.
  2. Create an alarm policy in BMC Helix Operations Management.
  3. Verify the service topology in BMC Helix Discovery.
  4. Create a blueprint and service model in BMC Helix AIOps and then monitor the business service. 

Task 1: To configure the OpenTelemetry Collector

You can configure the OpenTelemetry Collector to suit your observability requirements. To enable the collector to export data to the BMC Helix applications, configure the following components of the collector:

  • Receiver: Receives traces data from instrumented applications by using various protocols.
  • Exporter: Exports traces data to one or more destinations.

For more information about the collector components, see the Open Telemetry documentation.

Before you begin

Before you configure the collector, ensure that the application from which you want to receive traces via OpenTelemetry has been instrumented, which means code from the application's components is able to emit traces. For more information, see Open Telemetry documentation..

To configure the collector

You can deploy the collector on a variety of operating systems and architectures. For information, see the Open Telemetry documentation..

Location and name of the file containing the collector configuration depends on the installation method of the collector. For example, in a standalone environment, typically, the config.yaml file contains the collector configuration. In a Kubernetes environment, collector configuration is stored in a ConfigMap. For more information, see Open Telemetry documentation..

  1. Open config.yaml or ConfigMap.
    The following example shows a configuration file that you can use to export traces to BMC Helix applications.

    receivers:
        otlp:
            protocols:
              http:
                endpoint: 0.0.0.0:<portNumber>
    exporters:
        otlphttp:
        endpoint: https://htel-http-<K8sNamespace>.<K8sClusterDomain>
        headers:
            X-Api-Key:<apiKey>
        sending_queue:
            enabled: true
            num_consumers: 100
            queue_size: 10000
    service:
        pipelines:
            traces:
              exporters:
              - otlphttp
              receivers:
              - otlp

  2. In the file, configure the properties in the following sections:
    • receivers: Update endpoint with the OpenTelemetry Collector port number.
    • exporters: Update the following properties:
      • endpointURL – BMC Helix tenant URL. For example, https://htel-trace-service.qa1.svc.cluster.local.

      • X-Api-Key – API key required for authentication between the collector and BMC Helix applications in the following format: <tenant ID>::<access key>::<secret key>::<source>

        • <tenant ID>, <access key>, <secret key>: Log on to BMC Helix Portal and generate the tenant ID, access key, and secret key. For more information, see Setting up access keys for programmatic access..  

        • <source>: A string identifier indicating the source of traces. For example, if your application, ITSM Services is running on a Kubernetes namespace, enter itsm-kublet.

        For example, 1369852381::ATCRDJ4529QMBNCT7ZDES4079::bmwwPvuQyak3CgCIDV4WQEor1DT6R8hU8PiQPuulrR::itsm-kublet.

    • service: Update the section with the names of the receivers and exporters. Entries in this section enable the components in the collector based on the configuration found in the receivers and exporters sections.

Task 2: To create an alarm policy in BMC Helix Operations Management

Create an alarm policy that generates events when the value of a trace or operation metric is greater or less than a specified value. 

  1. Log on to BMC Helix Operations Management.
  2. On the Configuration > Alarm Policies page, click Create.
  3. Add a unique precedence number to the policy.
    You can add a custom value in this field, or use the arrows to increase or decrease the value.

  4. In the Alarm Generation Conditions section, enter the following information:

    Configuration

    Description

    Attribute to monitor

    Select the operation-level or trace-level metric for which you want to add this policy, for example, Apdex.

    Instance details

    Select All Instances or Multiple Instances as needed.

    Threshold conditions

    Select the threshold value, violation duration, and details about when the generated alarm must be closed eventually. Also, specify if the event must be closed immediately after it is generated or after the metric reaches a normal state and a duration after the violation time period has lapsed. You can specify that the event must not be closed. Alarm events that are not closed remain open until they are closed manually, the policy is deleted, or the PATROL Agent associated with the alarm is deleted.

    By specifying conditions, you can get notified of event alerts when a metric violates the baseline value. To change any of the values, click them.

    For example, if the metric value is below 0.98 score for more than one min, then generate major alarm and do not close the alarm.

    The following image shows a sample alarm generation condition:
    OTel_AlarmGenerationCondition_24102.png

  5. Optional) Select Enable Policy.
    You can enable or disable the policy any time from the Alarm Policies page. For more information, seeConfiguring alarm policies.

  6. Save the policy.

Task 3: To verify the service topology in BMC Helix Discovery

Verify that the OpenTelemetry application topology is ingested into BMC Helix AIOps. 

Before you begin

Ensure that you have obtained the latest The Pattern Language (TPL). For details, contact BMC Customer Support.

To verify the service topology in BMC Helix Discovery

  1. Log on to BMC Helix AIOps.
  2. Click Explore > Data.
  3. In the Miscellaneous section, click External Elements.
  4. Verify that the entries of type OTel Span, OTel Host, OTel Service, and OTel Service Namespace appear on the External Element List page.
    OTel_Discovery_24102.png

Task 4: To create a blueprint and service model in BMC Helix AIOps and monitor the business service

  1. Log on to BMC Helix AIOps.
  2. (Optional) Create the service blueprint:
    1. Select Configurations > Manage Service Blueprints.
    2. On the Service Blueprints page, click Create Service Blueprint.
    3. Specify the name, description, and provider (domain) details for the blueprint.
      For example, OTel Service Blueprint, Trace extraction, and .itbiz.com.
    4. Click Next Step.
    5. On the Edit Blueprint Rules tab, add CI Kinds and establish relationships between them:
      1. In the Available CI Kinds section, type External Element and select it from the list. 
      2. Add two more CI Kinds of type External Element.  
      3. Add one element of CI Kind of type Host.
      4. Establish relationship between the selected CI Kinds, as shown in the following image:
        OTelBlueprint_Define_24102.png
      5. Define the CI rule for the start node.
        1. Click the start node.
          On the Define CI Rule page, the default filter criterion shows the Name attribute of a CI added as a variable. When creating a service by using this blueprint, you need to provide this attribute as an input. 
        2. From Actions, select Edit. 
        3. On the Define Filter page, from Actions, select Edit. 
        4. In the Define user prompt message text box, update the message to Enter OTel Service Namespace.
        5. Click Add New Filter Criteria.
        6. In the Select attribute list, select Type.
        7. In the Select value(s) from existing data list, select OTel Service Namespace. 
        8. Click Save and Close.
          DefineCIRule.png
             
      6. Define the link rule.
        1. Click the link between the start node and the second node.
        2. Select the Filter by Kind option.
        3. From the list, select Dependency.
          DefineLinkRule.png
        4. Click Save and Close.

      7. Click Save.
      8. Enable the blueprint:
        1. From the Service Blueprints page, choose a blueprint from the list and select Action >Enable to enable the blueprint to be used as part of a service model.  
        2. Click Hide disabled to view the list of all enabled service blueprints.
        3. Check for the OTel Service Blueprint in the list. 
  3. Create the service model:
    1. Click Services and then click Create New Service.
    2. In the Define Service pane, specify the service name, for example, ITSM Service.
    3. (Optional) Assign a kind to the service.
      For more information about service kinds, see Business services, technical services, and business applications.
    4. (Optional) Select service criticality from the list to mark the importance of the business service.
    5. Click Add Dynamic content.
    6. From the blueprints list, select OTel Service Blueprint.
    7. Select the required OpenTelemetry namespace. 
    8. Click Save.
    9. Click Save and Close.
  4. On the Services page, search for the service and then click the service.
    The CI Topology tab shows the topology of the business service for the application instrumented by OpenTelemetry.
    Otel_SampleServiceTopology.png

After the service model is created, you can start monitoring the application services through the service model. 

Where to go from here

View and analyze traces by using the following dashboards:


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*