Skip to Content
Information

This site will undergo a brief period of maintenance on Friday, 16 October at 2 AM Central/12:30 PM IST. During a 30 minute window, site availability will be intermittent.

Installing BMC Helix Logging

As a system administrator, you can deploy and configure BMC Helix Logging to collect logs, and then configure the Kibana to view logs.

BMC Helix Logging uses Elasticsearch, Fluent Bit, and Kibana (EFK) to collect, store, and view logs:

  • Elasticsearch: A search engine which is also used to store and analyze logs.
  • Fluent Bit: A data agent aggregates the application logs collected by the log shipper and sends them to Elasticsearch.
  • Kibana: A log exploration tool, which has a web interface for data visualization.

 

Related topics

Installing-BMC-Helix-Platform-Common-services-25-3-00

Preparing to collect logs from external log sources

Viewing and collecting critical logs by using the Kibana dashboard

Before you begin

  • Make sure that BMC Helix Platform Common Services 25.3.00 is installed.

  • Make sure that the following ports are available for Elasticsearch and Kibana services: 

    Service

    Ports

    ElasticSearch

    9200, 9300

    Kibana

    5601

     

  • Make sure that you have a separate namespace to install BMC Helix Logging.
    See System requirements.
  • Download the bmc-helix-logging-25.3.00.tgz file from EPD by using the BMC Helix Platform CommonServices Logging Bundle option.
    See Downloading the installation files.

To install BMC Helix Logging

  1. Extract the bmc-helix-logging-25.3.00.tar file to the helix-on-prem-deployment-manager/utilities folder in the working directory.
  2. Perform the following procedures that are relevant to your deployment.

    Deployment

    Procedure

    Kubernetes

    1. Use the default namespace or create a namespace in Kubernetes; for example, bmc-helix-logging.
    2. Navigate to helix-on-prem-deployment-manager/utilities/bmc-helix-logging/efk/fluent-bit/ and replace the efk-fluent-bit-clusterrole.yaml namespace with the namespace that was created in step a.
    3. Run the following command:
      kubectl apply -f efk-fluent-bit-clusterrole.yaml

    4. Use the cluster admin permission and apply the privileged policy to the new namespace.

      kubectl label namespace <namespace-name> pod-security.kubernetes.io/enforce=privileged
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/enforce-version=latest
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/audit=privileged
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/audit-version=latest
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/warn=privileged
      kubectl label namespace <namespace-name> pod-security.kubernetes.io/warn-version=latest
    5. To use a different namespace, navigate to bmc-helix-logging.config and set the variable BMC_HELIX_LOGGING_NAMESPACE to a new namespace.

    OpenShift

    1. Use the default namespace or create a namespace in Openshift; for example, bmc-helix-logging.
    2. Make sure that you have the cluster administrator permissions.

    3. Navigate to helix-on-prem-deployment-manager/utilities/bmc-helix-logging/efk/fluent-bit/ and run the following commands:

      • To replace the efk-fluent-bit-scc.yaml namespace, run the oc apply -f efk-fluent-bit-scc.yaml command.
      • To replace the efk-fluent-bit-clusterrole.yaml namespace, run the oc apply -f efk-fluent-bit-clusterrole.yaml command.
      • To add adm policy scc to the service account in the namespace, run the oc adm policy add-scc-to-user efk-fluent-bit -z efk-fluent-bit -n <namespace> command.
    4. Add bmc-helix-logging UID in bmc-helix-logging.config; for example,
      • LOGGING_RUN_AS_USER=1000750000
      • LOGGING_RUN_AS_GROUP=1000750000
      • LOGGING_FS_GROUP=1000750000
  3. Navigate to the helix-on-prem-deployment-manager/utilities/bmc-helix-logging/bmc-helix-logging.config file and set the following values:

    Parameter

    Description

    KIBANA_LB_HOST 

    Specify a URL to create a Kibana load balancer host. 
    The BMC Helix Logging Ingress uses the value of this parameter.
    Example: KIBANA_LB_HOST= kibana-private-poc.mydomain.com

    ESLOG_MASTER_STORAGE_CLASS 

    Specify a storage class for Elasticsearch master nodes.
    Usually, a single storage class by using block storage is configured for all the infra services.

    Example: ESLOG_MASTER_STORAGE_CLASS=acme-block-storage

    ESLOG_DATA_STORAGE_CLASS 

    Specify a storage class for Elasticsearch data nodes.
    Usually, a single storage class by using block storage is configured for all the infra services.

    Example: ESLOG_DATA_STORAGE_CLASS=acme-block-storage

    ESKIBANA_SERVER_HOST

    • To allow connections to Kibana service from any IPv4 address, set ESKIBANA_SERVER_HOST ="0.0.0.0"
    • To allow connections to Kibana service from any IPv6 address, set ESKIBANA_SERVER_HOST ="::"
  4. Run the bmc-helix-logging-deployer.sh script by using the following command:
    ./bmc-helix-logging-deployer.sh fresh
    The BMC Helix Logging EFK is deployed in the BMC Helix Logging namespace.
     

To set the ILM policy for log retention

If you enable BMC Helix Logging, you must set the Index Lifecycle Management (ILM) policy for log retention in Kibana UI.

Warning

Important

The steps provided here are an example of setting the ILM policy to retain the logs for 10 days. You must make appropriate changes to the HelixLogDeletePolicy.json and helixlogindextemplate.json files to retain logs for a desired number of days.

To set up an ILM policy to retain the logs for 10 days, perform the following steps:

  1. To create an ILM policy, perform the following steps:

    1. Download the HelixLogDeletePolicy.jsonHelixLogDeletePolicy.json file and copy its contents.

      Success

      To retain the logs for a desired number of days, open the HelixLogDeletePolicy.json file, under the delete section, change the value of the min_age parameter.

    2. Go to helix-on-prem-deployment-manager/utilities/bmc-helix-logging/bmc-helix-logging.config.
    3. Obtain the URL to access the Kibana UI. 
      This is the value that you had set for the parameter KIBANA_LB_HOST.
      For example, https://kibana.apexglobal.bmc.com. 
      See Configuration-file-settings.
    4. Click 23.4_ILM.png, and select Management > Dev Tools.
      23.4_ILM1.png
    5. If the Console tab has any data, delete it.
    6. Paste the contents of the HelixLogDeletePolicy.json file in the Console tab and click 23.4_ILM2.png. 

      23.4_ILM3A.png

      You will get the following message and a policy is created with the name HelixLogDelete10DaysLog-Policy:
      23.4_ILM_AclnowledgeTrue.png
  2. To apply the ILM policy to all indices that are already created, perform the following steps:
    1. Click 23.4_ILM.png, and select Management > Stack Management.
    2. Select Data > Index Management
    3. Select an index; for example, logstash-2024.01.04.
    4. From the Manage Index list, select Add lifecycle policy.
      23.4_ILM_IndexManagement2.png
    5. In the Add lifecycle policy to "<Index name>" box, from the Lifecycle policy list, select the policy that you created, and click Add policy.
      23.4_ILM-ManageIndex3.png
      The policy is applied only to the index that you had selected; for example, logstash-2024.01.04.
    6. Repeat the steps to apply the ILM policy to all indices.
  3. To verify that the policy is applied an index, perform the following steps:
    1. Click 23.4_ILM.png, and select Management > Stack Management.
    2. Select Data > Index Management
    3. Click an index to view the summary.
    4. In the Index lifecycle management area, click Show definition to verify that the policy is applied.
      23.4_ILM-ManageIndex4Confirmation1.png
  4. To create a ILM policy template, perform the following steps: 
    To apply the ILM policy to all new indices that might get created, you must create the ILM policy template.

    1. Download the HelixLogIndexTemplate.jsonfile and copy its contents.

      Success

      In the HelixLogDeletePolicy.json file, if you changed the policy name, make sure to update the same policy name in the HelixLogIndexTemplate.json file. 

      In the HelixLogIndexTemplate.json file, the name of the template should always be in lowercase; for example, helixlogindextemplate.

    2. Click 23.4_ILM.pngand select Management > Devtools.
    3. In the Console tab, paste the contents of the HelixLogIndexTemplate.json file and click 23.4_ILM2.png. 
      You will get the following message:
      23.4_ILM_AclnowledgeTrue.png
      The index template HelixLogIndexTemplate gets created with the policy name HelixLogDelete10DaysLog-Policy.
      The new template will apply to all new indices. 
  5. To verify the ILM policy template created, perform the following steps:
    1. Click 23.4_ILM.png, and select Management > Stack Management.
    2. Select Data > Index Management
    3. Go to the Index Templates tab and verify that the template is created.
      23.4_ILM-FinalTemplateConfirmation.png

For more information, see ILM: Manage the index lifecycle.

Where to go from here

Setting up the installation environment

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*

BMC Helix Service Management Deployment 25.3.01