Generic - General Text Log Parser

This topic describes how to use the General Text Log Parser ETL. Some important properties of this parser are:

  • Input: It is more flexible in comparison with compatible CSV data format parsers.
  • Mandate on regular expressions: You need to specify regular expressions needed to read and understand the file format.
  • Additional features: It counts rows that match a regular expression.
  • Supported dataset: WKLDAT (vertical format).
  • Aggregation
    • To aggregate parsing results using one of the available statistics: Use the following script:

      extract.parser.wkldregex.[CKB:num].aggrtype = [sum | CKB:count | max | min | average]
    • Aggregation field: Use the following script:

      extract.parser.wkldregex.[CKB:num].value = %GROUPn

      where [CKB:num] is the definition number.
  • lastcounter parameter: The lastcounter parameter cannot be used with the Generic Text Log Parser. The parser parses all the files that match the available pattern, whereas extractors extract data from the database, and the lastcounter parameter holds the last sample extracted.

Integration steps

To integrate BMC Helix Continuous Optimization with the general text log parser:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, click Add > Add ETL under the Last run tab.

  3. In the Add ETL page, set values for the following properties under each expandable tab.

    Note

    Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

    Basic properties

    Note

    To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

  4. Click Save.
    You return to the Last run tab under the ETL tasks page.
  5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click run_etl.png to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    • OK: The ETL executed without any error in simulation mode.
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  6. Switch the ETL to production mode: To do this, perform the following task:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
    2. In the Run configurations table in the ETL details page, click edit icon.png to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  7. Locate the ETL in the ETL tasks table and click run_etl.png to Run it, or schedule an ETL run.
    After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the BMC Helix Continuous Optimization database.

Aggregation

Add the following two configuration parameters:

  • extract.parser.wkldregex.[CKB:num].aggrtype = [sum | CKB:count | max | min | average] : Lets you aggregate parsing results using one of the available statistics.
  • extract.parser.wkldregex.[CKB:num].value = %GROUPn : The aggregation field.

where [CKB:num] is the definition number.

lastcounter parameter

The lastcounter parameter cannot be used with the Generic Text Log Parser. The parser parses all the files that match the available pattern, whereas extractors extract data from the database, and the lastcounter parameter holds the last sample extracted.

Related topics

Using-ETL-datasets

Generic-CSV-file-parser

Developing-custom-ETLs

Dataset-reference-for-ETL-tasks

Horizontal and Vertical datasets

Viewing-datasets-and-metrics-by-dataset-and-ETL-module

This topic describes how to use the General Text Log Parser ETL. Some important properties of this parser are:

  • Input: It is more flexible in comparison with compatible CSV data format parsers.
  • Mandate on regular expressions: You need to specify regular expressions needed to read and understand the file format.
  • Additional features: It counts rows that match a regular expression.
  • Supported dataset: WKLDAT (vertical format).
  • Aggregation
    • To aggregate parsing results using one of the available statistics: Use the following script:

      extract.parser.wkldregex.[CKB:num].aggrtype = [sum | CKB:count | max | min | average]
    • Aggregation field: Use the following script:

      extract.parser.wkldregex.[CKB:num].value = %GROUPn

      where [CKB:num] is the definition number.
  • lastcounter parameter: The lastcounter parameter cannot be used with the Generic Text Log Parser. The parser parses all the files that match the available pattern, whereas extractors extract data from the database, and the lastcounter parameter holds the last sample extracted.

Integration steps

To integrate BMC Helix Continuous Optimization with the general text log parser:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, click Add > Add ETL under the Last run tab.

  3. In the Add ETL page, set values for the following properties under each expandable tab.

    Note

    Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

    Basic properties

    Note

    To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

  4. Click Save.
    You return to the Last run tab under the ETL tasks page.
  5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click run_etl.png to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    • OK: The ETL executed without any error in simulation mode.
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  6. Switch the ETL to production mode: To do this, perform the following task:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
    2. In the Run configurations table in the ETL details page, click edit icon.png to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  7. Locate the ETL in the ETL tasks table and click run_etl.png to Run it, or schedule an ETL run.
    After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the BMC Helix Continuous Optimization database.

Aggregation

Add the following two configuration parameters:

  • extract.parser.wkldregex.[CKB:num].aggrtype = [sum | CKB:count | max | min | average] : Lets you aggregate parsing results using one of the available statistics.
  • extract.parser.wkldregex.[CKB:num].value = %GROUPn : The aggregation field.

where [CKB:num] is the definition number.

lastcounter parameter

The lastcounter parameter cannot be used with the Generic Text Log Parser. The parser parses all the files that match the available pattern, whereas extractors extract data from the database, and the lastcounter parameter holds the last sample extracted.

Related topics

Using-ETL-datasets

Generic-CSV-file-parser

Developing-custom-ETLs

Dataset-reference-for-ETL-tasks

Horizontal and Vertical datasets

Viewing-datasets-and-metrics-by-dataset-and-ETL-module