Maintenance outage for upgrade on Sunday, September 22

This site, docs.bmc.com, will be inaccessible for two hours starting at 8 AM CDT, Sunday, September 22, for a platform upgrade.

    Page tree
    Skip to end of metadata
    Go to start of metadata

    Extensible module

    This module extends the functionalities of the existing SCOM integration allowing the Capacity Optimization administrator to import any SCOM counter.

    The procedure has 5 steps

    1. Analysis: Match and Gap
    2. Design: Configuration file creation
    3. Build: Upload of the configuration file
    4. Build: Setup of the ETL task
    5. Validate: Testing of the ETL configuration and activation

    Analysis: Match and Gap

    When a Microsoft SCOM counter is imported into TrueSight CO, the ETL must ‘match’ it to a metric defined in the Capacity Optimization catalogue of metrics.

    For example:

    • SCOM Object Memory, SCOM Counter ‘Available Mbytes Total’ is the Capacity Optimization metric ‘MEM_FREE’
    • SCOM Object System, SCOM Counter ‘% Processor Time’ is the Capacity Optimization metric ‘CPU_UTIL’

    The ‘Match and Gap’ phase defines this mapping between metrics in SCOM and metrics in TrueSight CO: you can find which metrics are already mapped by the ETL in the ‘Microsoft SCOM User’s guide’ documentation.

    If you are using the extensible module, you are importing metrics that are not supported by the connector: this is the step where you have to define the mapping: ‘Match and Gap’ means

    • a ‘match’ is when you can find that counter X in SCOM has an equivalent metric Y in Capacity Optimization
    • a ‘gap’ is when you can’t find an appropriate metric Y in Capacity Optimization and you need to define your own custom metric from Administration > Datawarehouse > Datasets & metrics > “Add metric” (for more information on how to do this check Data Warehouse )

    To summarize, when you identify in the SCOM Console a counter not covered by the ETL and you want to import it in Truesight CO, you have to:

    1. Find the metric in SCOM, and extracts its identifier fields (discovery)
    2. Define the mapping with the BMC TrueSight Capacity Optimization metrics catalogue (match and gap process)

    This first activity differs for Performance and Configuration metrics.

    Performance metrics

    The first step is collecting SCOM counters in terms of:

    1. Management pack name
    2. Object name
    3. Counter name
    4. If the metric represents the total or the instance value

    Looking at the screenshot below taken from a SCOM Console chart you could gather the four metrics identifier

     

    Starting from the SCOM console identifier, you can ease the process by running a few SQL queries to help you extract these values from the SCOM DWH (the same database used by the ETL) that are listed in SQL queries to execute on SCOM DWH

    a)      Q1 lists all the installed Management Packs with their description.

    b)      Q2 lists all Objects and Counters given a chosen management pack (to insert in the specified placeholder)

    c)       Q3 lists all systems and instances with the timestamp of the most recent data given a chosen management pack, object and counter (to insert in the specified placeholders)

    Using these queries you should identify the triplets (management pack, object, counter) you want to import, Q3 is useful to understand as well:

    a)      The System name is the “Path” column, if path is null, will be used “DisplayName”

    b)      The amount of system imported by counting the different system names

    c)       If the metric is currently collected by looking at MaxTS

    d)      If the metric is “total” (instancename column have “_Total” or null value) or “by” (instancename column have a specific subresource value)

     

    Configuration metrics

    The first step is choosing counters from SCOM in terms of:

    1. Management pack name
    2. Property name
    3. Managed entity type system name

    SQL queries to help you extract these values from the SCOM DWH (the same database used by the ETL) are in SQL queries to execute on SCOM DWH

    a)      Q1 lists all the installed management packs with their description.

    b)      Q4 lists all Properties given a chosen management pack (to insert in the specified placeholder)

    c)       Q5 lists all systems, entity type and values with validity period given a chosen management pack and property name (to insert in the specified placeholders)

    Using these queries you identify the value triplets (management pack, property, entity type) you want to import, Q5 is useful to understand as well:

    a)      The System name is the “Path” column, if path is null, will be used “DisplayName”

    b)      The amount of system imported by counting the different system names

    c)       If the metric is recently changed looking at From-To TS

    Mapping in BMC TrueSight Capacity Optimization (common to both performance and configuration)

    The second step is to map the SCOM triplets with BMC TrueSight Capacity Optimization metrics, in the BMC TrueSight Capacity Optimization console.

    The page “Administration >DATA WAREHOUSE >Datasets & metrics” shows all existing metrics and it allows the user to create new metrics in case of a gap.

    The field you need to define the mapping is the column “Metric name”

     

    The last step is to verify if the SCOM to BMC TrueSight Capacity Optimization metric mapping needs a conversion expression (e.g. SCOM express time in minutes, BMC TrueSight Capacity Optimization in seconds) and write the expression using basic math operators (+,-,*,/) and “source_value” to express the SCOM input value.

    Here some examples of how to convert metrics:

    a)      To convert a time in minutes to seconds: source_value*60

    b)      To convert memory utilization % to free memory %: 1-source_value (please consider % metrics expressed in [0..1] interval)

    c)       To convert bits into bytes: source_value/8

    Design: Configuration file creation

    The next step is to store the mapping of metrics created in the previous step in a CSV configuration file (one row for each metric to map) having the following format

    a)      Each row contains one mapping

    b)      Comments are preceded by “#”

    c)       The first column identify the metric type: performance (value=PERF) or configuration (value=CONF)

     

    The performance metrics table is composed of ordered columns separated by semicolon:

    1. Metric type must contains the value “PERF”
    2. Management pack name: the SCOM management pack identified in the Match and gap phase
    3. Object: the SCOM Object identified in the Match and gap phase
    4. Counter: the SCOM Counter identified in the Match and gap phase
    5. By/Tot, if the metric is “total” insert “tot”, if the metric has subresources insert “by”
    6. Import: TRUE to import the metric in CO, FALSE to not import it
    7. BCO Metric: the field “Metric name” identified in the BMC TrueSight Capacity Optimization Dataset and metric page in the Match and gap phase
    8. Transformation operation: the conversion expression to apply to SCOM metric values before import them in TrueSight CO, if no expression is needed, write “1”

    The configuration metrics table is composed of ordered columns separated by semicolon:

    1. Metric type must contains the value “CONF”
    2. Management pack name: the SCOM management pack identified in the Match and gap phase
    3. PropertySystemName: the SCOM property identified in the Match and gap phase
    4. ManagedEntityTypeSystemName: the SCOM entity type identified in the Match and gap phase
    5. BCO Metric: the field “Metric name” identified in the BMC TrueSight Capacity Optimization Dataset and metric page in the Match and gap phase
    6. Transformation operation: the conversion expression to apply to SCOM metric values before import them in TrueSight CO, if no expression is needed, write “1”

    Examples of a mapping configuration file

    # PERF metrics
    # Metric type;Management pack name;Object;Counter;by/tot;import;BCO Metric;Transf_operation
    PERF;Microsoft.Windows.Server.2012.Monitoring;logicaldisk;%idletime;by;true;BYLDISK_UTIL;(1-(source_value*0.01))
    PERF;Microsoft.Windows.Server.2012.Monitoring;memory;Percent Memory Used;total;true;MEM_UTIL;0.01
    PERF;Microsoft.Windows.Server.2012.R2.Monitoring;memory;Percent Memory Used;total;true;MEM_UTIL;0.01
    PERF;Microsoft.SystemCenter.DataWarehouse.Internal;OpsMgr DW Synchronization Module;Batches/sec;by;true;BYSET_EVENT_RATE;1
    PERF;Microsoft.Windows.Server.2008.Monitoring;memory;Pool Nonpaged Bytes;total;true;MEM_POOL_NONPAGED_BYTES_C;1
    # CONF metrics
    # metrictype; Management pack name;PropertySystemName;ManagedEntityTypeSystemName; BCO Metric;Transf_operation
    CONF;Microsoft.Windows.Library;PhysicalMemory;Microsoft.Windows.OperatingSystem;TOTAL_REAL_MEM;1024
    CONF;Microsoft.Windows.Library;DNSName;Microsoft.Windows.Computer;DNS_NAME_C;1

    Build: Upload of the configuration file

    When the mapping phase is complete, the file needs to be uploaded on BMC TrueSight Capacity Optimization EE (suggested directory <CO_HOME>/repository/imprepository/SCOM).

    The file can be transferred via usual unix file transfer mechanism or leveraging the ‘Upload file’ mechanism available from ETL & System Tasks > ETL Tasks

    Build: Setup of the ETL task

    At this point you can configure an instance of an ETL Task following the same set instructions available in the ETL user’s guide, the only differences being

    • As “Management pack to import” select Extensible
    • As “Extensible module file location”, enter the full path and name of the mapping file

    Validate: Testing of the ETL configuration and activation

    The last step before completing the process is a test: run the etl in simulation mode and look at the log, that can gives you the following error messages for each metric defined:

    Error message

    Resolution

    Metric skipped due to malformed definition - check first column (PERF/CONF) or separator (;)

    Check if all fields are set and if the first column contains PERF or CONF value

    Metric skipped due to missing column

    Check the presence of all the expected columns

    Metric skipped due to malformed definition - fifth column (TOTAL/BY)

    Check the value in the fifth column, it must contains TOTAL or BY value

    Metric skipped due to malformed definition - sixth column (TRUE/FALSE)

    Check the value in the fifth column, it must contains TRUE or FALSE value

    Metric skipped due to inexistent BCO metric

    The BMC TrueSight Capacity Optimization metric set in the sixth column is not defined, check if exists from the console (Administration >DATA WAREHOUSE >Datasets & metrics) or if it is written correctly

    Metric skipped malformed conversion expression

    Check that the conversion expression match the syntax described in the previous paragraph

     

    If no warnings appear, you should see a message like the following

    When the ETL terminates correctly, it is possible to download the dataset to check:

    • system name
    • metric name
    • value

    if all fields are in line with the expectations, it’s possible to:

    • turn off the simulation mode
    • run again the etl
    • wait until the end of the task and the NRTWH processing
    • check the data in the systems using BMC TrueSight Capacity Optimization Console

    Tips and tricks

    • In the setup/test phases, limit the metrics extraction to few sample, by choosing a recent default lastcounter or limit the number of max days to extract. Then when the data has been validated it’s possible to perform a history recovery.
    • If available, use the DEV/TEST environment for both SCOM and BMC TrueSight Capacity Optimization platform to perform the match and gap and test/validation phases
    • For the ‘Match and Gap’ step, you can define your own custom metrics in BMC TrueSight Capacity Optimization but keep it as your last option.

    • No labels