Configuring Splunk Generic Extractor

The "Moviri– Splunk Generic Extractor" connector aims at importing almost any Business Driver or System metric contained in your Splunk instance that are not specifically mapped by the other Splunk connectors provided by Moviri.
It works in a similar fashion to the built-in 'Generic' connectors available in BMC Helix Continuous Optimization and its main use case is the analysis from a capacity management perspective of custom metrics, for example:

• Baselining
• Historical analysis, seasonality identification
• Trending and forecasting
• Correlation with other metrics (specially infrastructure utilization) already present in BMC Helix Continuous Optimization (or in their turn imported from Splunk) to enable capacity modeling and what-if scenarios

In order to do so it must be provided with:

• A Splunk search query for retrieving results
• How the results of the query map into BMC Helix Continuous Optimization data model

At each execution, the connector

• Executes the provided search query (this step can be skipped for Splunk saved search)
• Retrieves the result set of the query
• Transforms the result set according to the specified mapping, producing BMC Helix Continuous Optimization Datasets
• Load the Datasets into CO
• Stores the most recent timestamp to be used as lower time boundary in the next execution

How to specify the search query for retrieving results

Two options are available for specifying the search query:

1. a Splunk "saved search": i.e. a ready to use search saved on Splunk instance which can retain historical results of previous executions for a settable retention period and/or be scheduled to be periodically executed. Additionally, saved searches can be "accelerated" for faster execution. "Moviri Integration for BMC Helix Continuous Optimization – Splunk (Generic)" can exploit Splunk saved searches functionality to
   • define queries on the Splunk side, thus avoiding the management and maintenance of search query syntax on the BMC Helix Continuous Optimization side
   • rely on already available results produced by previous on-demand or scheduled executions, and only optionally execute the saved search if need arises
   • take advantage of saved searches acceleration
2. a search query text: the search is saved in the connector configuration properties and it is passed at each connector execution to the Splunk instance

 
In both cases search results must meet the following requirements:

• Results must come in a tabular format comprising multiple fields
• Each record of the results table must refer to the same aggregated period of time (e.g. one record for each hour, one record for each day…)
• A timestamp field must be present in order to identify the beginning of the time period (e.g. "2013-05-24 13:00:00 for hourly granularity, "2013-06-03 00:00:00" for daily granularity). Splunk has an implicit "_time" field for each event it processes, so the natural choice is to use it as the timestamp.
• At least one value field must be present containing the data series to be transferred

In the table below a minimal result set example conforming to the above exposed requirements is presented: the daily number of logins on a system.

How to select the appropriate dataset

When creating an ETL task that uses the 'Moviri – Splunk Generic Extractor', the first configuration step is associating one or more datasets to the ETL task.

• Select 'WKLDAT' if the task is going to import 'Business Drivers'
• Select 'SYSDAT' if the task is going to import 'System' data
• Select 'APPDAT' if the task is going to import Application configuration This case is not going to be covered in this document; please refer to support for more information if required

How to map search query results to BMC Helix Continuous Optimization data model

The most important configuration of the connector is the definition of how to map each time series, extracted from Splunk, to

• Entities (either Business Drivers or Systems);
• Metrics (also referred to as resources);
• Metric Subobjects (also referred to as subresources).

These mapping are very similar to the ones required by the built-in Generic - Database extractor:

• Entities:
  • For Business Drivers (WKLDAT dataset) this is equivalent to DS_WKLDNM column, representing "Business driver lookup identifier"
  • For Systems (SYSDAT dataset) this is the equivalent of the DS_SYSNM column, representing the "System lookup identifier"
• Metrics (also referred to as resources)
  • Equivalent of the OBJNM column
• Metric Subobjects (also referred to as subresources)
  • Equivalent of the SUBOBJNM column
  • required when the metric is not of type 'GLOBAL', for example for every metric that has a sub-category dimension

For any time series the connector allows the following mapping options for the three above mentioned dimensions:

• Entities
  • Use the name of the Splunk search query column containing the series values
  • Input a fixed string
  • Use values from another Splunk search query column to identify entity name
• Metric (or resource)
  • Select among some proposed generic metrics (applicable to Business Drivers entities)
  • Input a specific metric (Advanced)
• Metric Subobject (or subresource), when metric is not of type GLOBAL
  • Input a fixed string
  • Use values from another Splunk search query column to identify subobject

Some examples are reported in the remainder of this paragraph to facilitate the application of the above mentioned principles: each example includes the Splunk search query result set, the Splunk-to-BMC Helix Continuous Optimization mappings, the resulting BMC Helix Continuous Optimization series and a screenshot of relevant configuration properties from the "Splunk – Query and Mapping" ETL task configuration tab.

EXAMPLE 1: number of daily logins on two services

In this example, the 'Saved Search' query on Splunk returns on two different columns the number of daily logins on two services ('svcA' and 'svcB') and we need to import these metrics in CO, as two separate business drivers.

• Select 'WKLDAT' as dataset, as this data is related to Business Drivers
• As 'Entity', we specify to use the name of the column (so that business drivers 'svcA' and 'svcB' will be created)
• As 'Metric', the number of daily logins can be mapped to "a count of events over time" and thus fits the description of the metric.
• As 'Metric subresource', the selected metric does not require a 'subobject', being a global metric and so we are not required to specify it.

This is the final mapping:

EXAMPLE 2: number of daily logins on monitored services

In this example, the query on Splunk returns the number of daily logins on two services ('svcA' and 'svcB') specifying the specific service in the 'Services' column and we need to import these metrics in CO, as two separate business drivers.

• Select 'WKLDAT' as dataset, as this data is related to Business Drivers
• As 'Entity', we specify to use the values in the 'Services' column (so that business drivers 'svcA' and 'svcB' will be created)
• As 'Metric', the number of daily logins can be mapped to "a count of events over time" and thus fits the description of the metric.
• As 'Metric subresource', the selected metric does not require a 'subobject', being a global metric and so we are not required to specify it.

This is the final mapping:

Note that in this case as new Services will appear in query results new Entities (Business Drivers) will be created in CO.

 
EXAMPLE 3: data volumes managed by a Datawarehouse system, split by its sub-procedures (steps)

This is the mapping:

Note that in this case as new Steps will appear in query results new subresources will be attached to existing entities.

 
EXAMPLE 4: cpu utilization by host

This is the mapping:

Note that in this case as new hosts will appear in query results new BMC Helix Continuous Optimization Entities will be created.

Full list of configuration properties

The following are the specific settings valid for connector "Moviri – Splunk Generic Extractor", they are presented in the "Splunk – Query and Mapping" configuration panel.