Preparing to collect logs from external log sources

BMC Helix Logging uses Elasticsearch, Fluent Bit, and Kibana (EFK) to collect, store, and view logs:

  • Elasticsearch: A search engine also used to store and analyze logs.
  • Fluent Bit: Aggregates the application logs collected by the log shipper and sends them to Elasticsearch.
  • Kibana: A web user interface for data visualization and a log exploration tool.

As a system administrator, you must deploy and configure BMC Helix Logging to collect logs, and then configure the Kibana to view logs. 

Important

BMC Helix Continuous Optimization does not support BMC Helix Logging.

In addition, you must integrate BMC Helix ITOM with external logging systems such as Splunk, and external Elasticsearch, Fluentbit, and Kibana (EFK) stacks to collect logs.

Related topics

Viewing-logs-on-Kibana

Troubleshooting-BMC-Helix-Logging-issues

Before you begin

  • BMC Helix IT Operations Management version 25.1.00 or later must be installed.

  • Make sure that the following ports are available in cluster nodes for Elasticsearch and Kibana services: 

    Important

    The traffic to Kibana ports is routed from the Load balancer on port 443 to the Ingress controller. As the Ingress controller decides the internal routing of the traffic, the Kibana ports should be open in the Kubernetes cluster.

To prepare to collect logs

  1. Download the bmc-helix-logging-25.1.00-45.tar file from EPD.
    EPD-Logging.png

  2. Extract the tar file to the utilities folder.

  3. Perform the following prerequisites that are relevant to your deployment.

    Deployment

    Procedure

    Kubernetes

    1. Use the default namespace or create a namespace in Kubernetes: Example: bmc-helix-logging.
    2. Navigate to helix-on-prem-deployment-manager/utilities/bmc-helix-logging/efk/fluent-bit/ and replace the efk-fluent-bit-clusterrole.yaml namespace with the namespace that created in step a.

    3. Use the cluster admin permission and apply the privileged policy to the new namespace.

      kubectl label namespace <namespace-name> pod-security.kubernetes.io/enforce=privileged
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/enforce-version=latest
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/audit=privileged
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/audit-version=latest
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/warn=privileged
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/warn-version=latest
    4. To use a different namespace, go to bmc-helix-logging.config and set the variable BMC_HELIX_LOGGING_NAMESPACE to a new namespace.

    OpenShift

    1. Use the default namespace or create a namespace in Openshift: Example: bmc-helix-logging.
    2. Using admin privileges, navigate to helix-on-prem-deployment-manager/utilities/bmc-helix-logging/efk/fluent-bit/ and run the following commands:
      • oc apply -f efk-fluent-bit-scc.yaml to replace the efk-fluent-bit-scc.yaml namespace.
      • oc apply -f efk-fluent-bit-clusterrole.yaml to replace the efk-fluent-bit-clusterrole.yaml namespace.
      • oc adm policy add-scc-to-user efk-fluent-bit -z efk-fluent-bit -n <namespace> to add adm policy scc to the service account in the namespace.
    2. Add bmc-helix-logging GID in bmc-helix-logging.config. Example:
      • LOGGING_RUN_AS_USER=1000750000
      • LOGGING_RUN_AS_GROUP=1000750000
      • LOGGING_FS_GROUP=1000750000

    Splunk

    • You have the Spunk deployed in your cluster.
    • Splunk is running on a different namespace in the cluster.
    • (Optional) If you are not using Splunk for logs collection, you must have the Fluentbit or Fluentd running in your cluster.
  4. Go to helix-on-prem-deployment-manager/utilities/bmc-helix-logging/bmc-helix-logging.config and set the following values:
    • To use IPv4, set ESKIBANA_SERVER_HOST ="0.0.0.0"
    • To use IPv6, set ESKIBANA_SERVER_HOST ="::"
  5. Run the ./bmc-helix-logging-deployer.sh script.
    The BMC Helix Logging deployer deploys EFK in the bmc-helix-logging namespace.

  6. Perform the relevant post deployment steps:

    Action

    Desctiption

    To access the Kibana URL

    1. Open the bmc-helix-logging.config file.
    2. Find the KIBANA_LB_HOST parameter.
    3. Specify a URL to create a Kibana load balancer host. 
      The BMC Helix Logging Ingress uses the value of this parameter.
      Example: KIBANA_LB_HOST= kibana-private-poc.mydomain.com.
    4. Viewing-logs-on-Kibana.

    For Splunk 


    1. Add the Splunk plugin in Fluentbit configmap.

       Kubectl edit cm fluent-bit -n <namespace> 
    2. Remove the output plugins other than Splunk from the Fluentbit configmap.

    3. Update the parameters:

      [OUTPUT]\n Name splunk\n Match kube.*\n Host <SPLUNK_HOST>\n Port <SPLUNK_PORT>\n Splunk_Token <SPLUNK_TOKEN>\n TLS On\n TLS.Verify Off\n

      Splunk host is the service name of Splunk and the namespace.
      <svc name>.<Splunk deployed namespace>Example: splunk-enterprise.splunk.svc.cluster.local

      [OUTPUT]
             Name splunk
             Match kube.*
             Host splunk-enterprise.splunk.svc.cluster.local
             Port <port>
             Splunk_Token <token>
             TLS On
             TLS.verify Off

    4. Restart the Fluentbit pods.
    5. Go to Splunk to see the streaming logs.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*