Extensible module for SCOM Integrator

Extensible module

This module extends the functionalities of the existing SCOM integration allowing the Capacity Optimization administrator to import any SCOM counter.

The procedure has 5 steps

1. Analysis: Match and Gap
2. Design: Configuration file creation
3. Build: Upload of the configuration file
4. Build: Setup of the ETL task
5. Validate: Testing of the ETL configuration and activation

Analysis: Match and Gap

When a Microsoft SCOM counter is imported into TrueSight Capacity Optimization, the ETL must ‘match’ it to a metric defined in the Capacity Optimization catalogue of metrics.

For example:

• SCOM Object Memory, SCOM Counter ‘Available Mbytes Total’ is the Capacity Optimization metric ‘MEM_FREE’
• SCOM Object System, SCOM Counter ‘% Processor Time’ is the Capacity Optimization metric ‘CPU_UTIL’

The ‘Match and Gap’ phase defines this mapping between metrics in SCOM and metrics in TrueSight CO: you can find which metrics are already mapped by the ETL in the ‘Microsoft SCOM User’s guide’ documentation.

If you are using the extensible module, you are importing metrics that are not supported by the connector: this is the step where you have to define the mapping: ‘Match and Gap’ means

• a ‘match’ is when you can find that counter X in SCOM has an equivalent metric Y in Capacity Optimization
• a ‘gap’ is when you can’t find an appropriate metric Y in Capacity Optimization and you need to define your own custom metric from Administration > Datawarehouse > Datasets & metrics > “Add metric” (for more information on how to do this check Data Warehouse )

To summarize, when you identify in the SCOM Console a counter not covered by the ETL and you want to import it in Truesight CO, you have to:

1. Find the metric in SCOM, and extracts its identifier fields (discovery)
2. Define the mapping with the TrueSight Capacity Optimization metrics catalogue (match and gap process)

This first activity differs for Performance and Configuration metrics.

Performance metrics

The first step is collecting SCOM counters in terms of:

1. Management pack name
2. Object name
3. Counter name
4. If the metric represents the total or the instance value

Looking at the screenshot below taken from a SCOM Console chart you could gather the four metrics identifier

Starting from the SCOM console identifier, you can ease the process by running a few SQL queries to help you extract these values from the SCOM DWH (the same database used by the ETL) that are listed in SQL queries to execute on SCOM DWH

The queries Q1, Q2, Q3, Q4, Q5 can be found here: SQL queries

a)      Q1 lists all the installed Management Packs with their description.

b)      Q2 lists all Objects and Counters given a chosen management pack (to insert in the specified placeholder)

c)       Q3 lists all systems and instances with the timestamp of the most recent data given a chosen management pack, object and counter (to insert in the specified placeholders)

Using these queries you should identify the triplets (management pack, object, counter) you want to import, Q3 is useful to understand as well:

a)      The System name is the “Path” column, if path is null, will be used “DisplayName”

b)      The amount of system imported by counting the different system names

c)       If the metric is currently collected by looking at MaxTS

d)      If the metric is “total” (instancename column have “_Total” or null value) or “by” (instancename column have a specific subresource value)

Configuration metrics

The first step is choosing counters from SCOM in terms of:

1. Management pack name
2. Property name
3. Managed entity type system name

SQL queries to help you extract these values from the SCOM DWH (the same database used by the ETL) are in SQL queries to execute on SCOM DWH

a)      Q1 lists all the installed management packs with their description.

b)      Q4 lists all Properties given a chosen management pack (to insert in the specified placeholder)

c)       Q5 lists all systems, entity type and values with validity period given a chosen management pack and property name (to insert in the specified placeholders)

Using these queries you identify the value triplets (management pack, property, entity type) you want to import, Q5 is useful to understand as well:

a)      The System name is the “Path” column, if path is null, will be used “DisplayName”

b)      The amount of system imported by counting the different system names

c)       If the metric is recently changed looking at From-To TS

Mapping in BMC TrueSight Capacity Optimization (common to both performance and configuration)

The second step is to map the SCOM triplets with TrueSight Capacity Optimization metrics, in the TrueSight Capacity Optimization console.

The page “Administration >DATA WAREHOUSE >Datasets & metrics” shows all existing metrics and it allows the user to create new metrics in case of a gap.

The field you need to define the mapping is the column “Metric name”

The last step is to verify if the SCOM to TrueSight Capacity Optimization metric mapping needs a conversion expression (e.g. SCOM express time in minutes, TrueSight Capacity Optimization in seconds) and write the expression using basic math operators (+,-,*,/) and “source_value” to express the SCOM input value.

Here some examples of how to convert metrics:

a)      To convert a time in minutes to seconds: source_value*60

b)      To convert memory utilization % to free memory %: 1-source_value (please consider % metrics expressed in [0..1] interval)

c)       To convert bits into bytes: source_value/8

Design: Configuration file creation

The next step is to store the mapping of metrics created in the previous step in a CSV configuration file (one row for each metric to map) having the following format

a)      Each row contains one mapping

b)      Comments are preceded by “#”

c)       The first column identify the metric type: performance (value=PERF) or configuration (value=CONF)

The performance metrics table is composed of ordered columns separated by semicolon:

1. Metric type must contains the value “PERF”
2. Management pack name: the SCOM management pack identified in the Match and gap phase
3. Object: the SCOM Object identified in the Match and gap phase
4. Counter: the SCOM Counter identified in the Match and gap phase
5. By/Tot, if the metric is “total” insert “tot”, if the metric has subresources insert “by”
6. Import: TRUE to import the metric in CO, FALSE to not import it
7. BCO Metric: the field “Metric name” identified in the TrueSight Capacity Optimization dataset and metric page in the Match and gap phase
8. Transformation operation: the conversion expression to apply to SCOM metric values before import them in TrueSight Capacity Optimization, if no expression is needed, write “1”

The configuration metrics table is composed of ordered columns separated by semicolon:

1. Metric type must contains the value “CONF”
2. Management pack name: the SCOM management pack identified in the Match and gap phase
3. PropertySystemName: the SCOM property identified in the Match and gap phase
4. ManagedEntityTypeSystemName: the SCOM entity type identified in the Match and gap phase
5. BCO Metric: the field “Metric name” identified in the TrueSight Capacity Optimization dataset and metric page in the Match and gap phase
6. Transformation operation: the conversion expression to apply to SCOM metric values before import them in TrueSight Capacity Optimization, if no expression is needed, write “1”

Examples of a mapping configuration file

# PERF metrics

# Metric type;Management pack name;Object;Counter;by/tot;import;BCO Metric;Transf_operation

PERF;Microsoft.Windows.Server.2012.Monitoring;logicaldisk;%idletime;by;true;BYLDISK_UTIL;(1-(source_value*0.01))

PERF;Microsoft.Windows.Server.2012.Monitoring;memory;Percent Memory Used;total;true;MEM_UTIL;0.01

PERF;Microsoft.Windows.Server.2012.R2.Monitoring;memory;Percent Memory Used;total;true;MEM_UTIL;0.01

PERF;Microsoft.SystemCenter.DataWarehouse.Internal;OpsMgr DW Synchronization Module;Batches/sec;by;true;BYSET_EVENT_RATE;1
 PERF;Microsoft.Windows.Server.2008.Monitoring;memory;Pool Nonpaged Bytes;total;true;MEM_POOL_NONPAGED_BYTES_C;1

# CONF metrics

# metrictype; Management pack name;PropertySystemName;ManagedEntityTypeSystemName; BCO Metric;Transf_operation

CONF;Microsoft.Windows.Library;PhysicalMemory;Microsoft.Windows.OperatingSystem;TOTAL_REAL_MEM;1024

CONF;Microsoft.Windows.Library;DNSName;Microsoft.Windows.Computer;DNS_NAME_C;1

Build: Upload of the configuration file

When the mapping phase is complete, the file needs to be uploaded on TrueSight Capacity Optimization EE (suggested directory <CO_HOME>/repository/imprepository/SCOM).

The file can be transferred via usual unix file transfer mechanism or leveraging the ‘Upload file’ mechanism available from ETL & System Tasks > ETL Tasks

Build: Setup of the ETL task

At this point you can configure an instance of an ETL Task following the same set instructions available in the ETL user’s guide, the only differences being

• As “Management pack to import” select Extensible
• As “Extensible module file location”, enter the full path and name of the mapping file

Validate: Testing of the ETL configuration and activation

The last step before completing the process is a test: run the etl in simulation mode and look at the log, that can gives you the following error messages for each metric defined:

If no warnings appear, you should see a message like the following

When the ETL terminates correctly, it is possible to download the dataset to check:

• system name
• metric name
• value

if all fields are in line with the expectations, it’s possible to:

• turn off the simulation mode
• run again the etl
• wait until the end of the task and the NRTWH processing
• check the data in the systems using TrueSight Capacity Optimization Console

Tips and tricks

• In the setup/test phases, limit the metrics extraction to few sample, by choosing a recent default lastcounter or limit the number of max days to extract. Then when the data has been validated it’s possible to perform a history recovery.
• If available, use the DEV/TEST environment for both SCOM and TrueSight Capacity Optimization platform to perform the match and gap and test/validation phases
• For the ‘Match and Gap’ step, you can define your own custom metrics in TrueSight Capacity Optimization but keep it as your last option.
• For configuring the ETL task, check the entity filters being used. If the extensible module imports the same entities defined the entity filters, the denylist and allowlist also applies. Make sure the entity filter does not affect the imported entities.