Configuring the log collection and alert policies


The Log Analytics connector for Kubernetes (connector for Kubernetes) collects filtered logs from the on-premise BMC Helix IT Operations Managementand sends them to BMC Helix Log Analytics on the Monitoring tool. You can view the logs on BMC Helix Operations Management.

Use the instructions in this topic to configure the connector to collect logs.

To configure BMC Helix Log Analytics connector to collect Kubernetes and Openshift logs

  1. Log on to BMC Helix Log Analytics.
  2. On your terminal, create a Data.csv file with the following contents:

    bmc_connector_name,UseCase
    env3-openshift-connector,SelfMonitoring
  3. Create Enrichment Sources:
    1. From the Enrichment menu, select Enrichment Sources, and then click Create.
      LAK8s14.png
      ​​​​​​On the Create Enrichment Source page:
    2. From the Type list, select CSV.
    3. In the Name field, enter Set UseCase.
    4. In the Description field, enter a description.
      LAK8s15.png
    5. Click Save.
    6. Edit the Set UseCase enrichment source and perform the following steps:
      1. In the CSV File field, import the Data.csv file that you created.
      2. From the Source Fields list, select bmc_connector_name.
      3. In the Enrichment Fields, add UseCase.
      4. Select the Enable Enrichment Source check box.
      5. Click Save.
        LAK8s16.png
  4. Create a filtering rule:

    Important

    The filtering rules define the log statements that will be sent to BMC Helix Log Analytics. They do not define the alert criteria.

    1. From the Collection menu, select Filtering Rules.
    2. Click Create.
    3. On the Create Filtering Rule page, in the Rule Information area, add a rule name (Agent filter for Kubernetes connector) and description.
      From the Collection Type list, select Kubernetes.

      Important

      Select Kubernetes even if you are configuring the connector to collect Openshift logs.

    4. In the Rule Configuration area, perform the following tasks:
      1. From the Type list, select Agent.
      2. From the Log Filter list, select Grep.
      3. From the Directive list, select Regex.
      4. From the Condition list, select AND.
      5. From the Key list, select log.
      6. In the pattern field, enter one of the following patterns:
        .*ERROR.*|.*NOT_ENOUGH_REPLICAS.*|.*Exception.*|.*invalid permission.*|.*TSMicroserviceNotAvailableException.*|.*org.redisson.client.RedisException.*
      7. Click Save.
        LAK8s22.png
  5. From the Collection menu, select Kubernetes.
    LAK8s1.png

    Important

    Select Kubernetes even if you are configuring the connector to collect logs in an OpenShift cluster.

     

  6. On the Add Kubernetes Connector page, in the Connector Information area, perform the following steps:
    1. In the Connector Name field, enter a unique name for the connector.
    2. In the Connector Type field, select Helm.
      LAK8s2.png
  7. In the Agent Configuration area, click Configure.
    If needed, you can use java multiline parser to configure the log agent.
    LAK8s3.png
  8. In the Customize Agent Logs Data dialog box, perform the following steps:
    1. From the Container Platform Format list, select a container platform format.
      Docker and CRIO are the supported formats.
    2. Click Save.
  9. From the Filtering Rule list, select the filtering rule that you created.
  10. In the Aggregator Configuration area, click Configure.
    LAK8s5.png
  11. In the Customize Aggregator Logs Data pane, perform the following steps:
    1. (Optional) In the Tags field, enter the tags to identify logs from a cluster or node.
    2. In the PVCStorageClass field, enter a valid StorageClass name to automatically create the PersistentVolume (PV) and PersistentVolumeClaim (PVC) storage requests.
      Run the following command to obtain the storage class details:

      kubectl get sc
    3. In the Watched Namespaces field, enter the BMC Helix ITOM namespace.
    4. Click Save.
      LAK8s6A.png
  12. In the Download and Configure area, click Download to download the Helm package.
  13. Create the bmc-k8s-logs namespace by running the following command:

    kubectl create ns bmc-k8s-logs
  14. Create the image pull secret by running the following command:

    kubectl create secret docker-registry <image-pull-secret-name> --docker-server=containers.bmc.com --docker-username=<username> --docker-password=<password> --docker-email=<email-    id> -n bmc-k8s-logs
  15. In the Download and Configure area, in the Image pull secrets for Docker Registry field, enter the image-pull-secret-name that you created in the previous step.
    Image-pull_secret.png
  16. Click the Create & Download button to create the integration and download the BMC Kubernetes Helm configuration file.
  17. In the values.yaml file, replace the repository name as shown in the following table:

    Entry in the values.yaml file...

    ...to be changed to

    fluentbit:
      crdsEnable: true
      name: fluent-bit
      enable: true
      image:attach:xwiki:IT-Operations-Management.On-Premises-Deployment.BMC-Helix-IT-Operations-Management-Deployment.itomdeploy252.Administering.Setting-up-and-configure-the-monitoring-solution.Configuring-the-log-collection-and-alert-policies.WebHome@filename
        repository: "containers.bmc.com/bmc"

    fluentbit:
      crdsEnable: true
      name: fluent-bit
      enable: true
      image:attach:xwiki:IT-Operations-Management.On-Premises-Deployment.BMC-Helix-IT-Operations-Management-Deployment.itomdeploy252.Administering.Setting-up-and-configure-the-monitoring-solution.Configuring-the-log-collection-and-alert-policies.WebHome@filename
        repository: "containers.bmc.com/bmc/log-fluentbit"

    fluentd:
     crdsEnable: true
     enable: true
     name: fluentd
     mode: "collector"
     image:attach:xwiki:IT-Operations-Management.On-Premises-Deployment.BMC-Helix-IT-Operations-Management-Deployment.itomdeploy252.Administering.Setting-up-and-configure-the-monitoring-solution.Configuring-the-log-collection-and-alert-policies.WebHome@filename
       repository: "containers.bmc.com/bmc"

    fluentd:
     crdsEnable: true
     enable: true
     name: fluentd
     mode: "collector"
     image:attach:xwiki:IT-Operations-Management.On-Premises-Deployment.BMC-Helix-IT-Operations-Management-Deployment.itomdeploy252.Administering.Setting-up-and-configure-the-monitoring-solution.Configuring-the-log-collection-and-alert-policies.WebHome@filename
       repository: "containers.bmc.com/bmc/log-fluentd"

    operator:
      initcontainer:
        repository: "containers.bmc.com/bmc"
      container:
        repository: "containers.bmc.com/bmc"

    operator:
      initcontainer:
        repository: "containers.bmc.com/bmc/log-fluentoperator"
      container:
        repository: "containers.bmc.com/bmc/log-fluentoperator"

  18. (For Openshift only) If you are configuring the connector for Kubernetes in the Openshift environment, perform the following steps:
    1. Add the following service accounts to the privileged Security Context Constraints (SCC):
      1. oc adm policy add-scc-to-user privileged -z fluent-bit -n bmc-k8s-logs
      2. oc adm policy add-scc-to-user privileged -z fluentd -n bmc-k8s-logs
      3. oc adm policy add-scc-to-user privileged -z fluent-operator -n bmc-k8s-logs
    2. In the values.yaml file, go to the Fluentbit section, and add the following value under the securityContext entry:
      privileged: true
      Use the following images for reference:
      Before adding the securityContext entry:

      LAK8s7.png
  19. Copy the downloaded Helm package and the configuration YAML file on the Kubernetes controller host of the cluster.
  20. Perform the following steps to deploy the Helm package:
    1. Navigate to the folder where the connector helm package is extracted and copy the downloaded configuration YAML file.
    2. Install the Helm package by running the following command:

      helm install fluent-operator . --create-namespace -n bmc-k8s-logs -f values.yaml
    3. Verify that the pods are running in the bmc-k8s-logs namespace by running the following command:

      kubectl get po -n bmc-k8s-logs
    4. If the agents or aggregators are not running, run the following command:

      helm upgrade fluent-operator . -n bmc-k8s-logs -f values.yaml
  21. Verify that the connector is configured correctly by running the following commands:

    kubectl get po -n bmc-k8s-logs

    Important

    If you do not see the pods created in the bmc-k8s-logs namespace, run the Helm upgrade again.

  22. To view the connector that you created, from the Collection list, select Connectors.
    ViewConnector.png

 

To create Alert policies in BMC Helix Log Analytics

Perform the following steps to generate events in BMC Helix Log Analytics when the log lines parsed in the BMC Helix Log Analytics meet the alerting criteria.

  1. From the Alerts menu, select Alert Policies.
    LAK8s8.png
  2. Click Create.
  3. On the Create Alert Policy page, perform the following steps:
    1. In the Policy Information area, add an alert name and a description:
      LAK8s9.png
    2. In the Policy Selection Criteria area, perform the following steps:
      1. Define the conditions to trigger the policy. See the image.
      2. In the Group by field, specify the group as log_source_host.
        LAK8s10A.png
      3. From the Create alert on the basis of list, select Static Thresholds
      4. In the Alert Condition area, select the conditions shown in the image to generate alerts:
        LAK8s10B.png
    3. In the Alert Parameters area, perform the following steps:
      1. In the Hostname field, specify $.log_source_host.
      2. In the Message field, enter a message.
      3. Under Additional Details, from the Select list, select details, and then type $.message
        LAK8s11.png
    4. In the User Group area, select a user group and then select the Enable Alert Policy check box.
      LAK8s12.png
  4. Click Save.

To create an Enrichment policy

Perform the following steps to add additional information to the log document:

  1. From the Enrichment menu, select Enrichment Policies
    LAK8s13.png
  2. Click Create.
  3. On the Create Enrichment Policy page, in the Policy Information area, add the following details:
    1. In the name field, add Set the UseCase field.
    2. In the Description field, add a description.
      LAK8s17.png
  4. In the Policy Selection Criteria area, set the trigger condition by using the connector for Kubernetes you created earlier.
    An example is shown in the following image:

    LAK8s18.png
  5. In the Enrichment Source area, click Add Enrichment, and then add the following details:
    1. From the Select Enrichment Type list, select CSV.
    2. From the Select Enrichment Source list, select Set UseCase.
      LAK8s20.png
    3. Click Done.
  6. In the Enrichment Source area, select the Enable Enrichment Policy check box.
    LAK8s19.png
  7. Click Save.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*