Moviri - Microsoft SCOM Extractor

"Moviri Integrator for BMC Helix Continuous Optimization  – Microsoft SCOM" allows collecting information from the Microsoft System Center Operation Manager tool. It is able to extract and summarize resources performance data, configuration data and relationships among IT entities.

• Requirements
  • Supported versions of data source software
  • Supported configurations of data source software
  • Datasource Check and Configuration
  • Preparing to connect to the data source software
  • Connect SCVMM to SCOM
  • Install Moviri VM Detector management pack in SCOM (optional)
  • SCOM Data Availability Check
• Connector configuration
  • Connector configuration attributes
  • Management pack and metric classes valid combinations
  • Extensible module – Customized metric support
  • Filtering systems with whitelist and blacklist
  • Database Connection parameters configuration
  • Filter virtual machines using Moviri VM Detector management pack
  • Historical data extraction
  • Supported Platforms
• Hyper-V View
• Troubleshooting
  • Entity Naming Conventions
• Lookup Fields
• Object Relationships
  • Assignment of imported entities to a specified domain
  • Windows cluster, Windows computer and cluster disks
  • SQL Server Instances and Databases
  • Hyper-V Clusters, Hosts, Datastore and Virtual Machines
• SQL queries to execute on SCOM DWH

Requirements

Supported versions of data source software

• Microsoft SCOM 2007 R2 (The VM Detector is not compatible with this version of SCOM)
• Microsoft SCOM 2012
• Microsoft SCOM 2012 SP1
• Microsoft SCOM 2012 R2
• Microsoft SCOM 2016
• Microsoft SCOM 2019

Supported configurations of data source software

Moviri Integration for Microsoft SCOM supports SCOM instances mandatorily configured with data warehouse component.
The supported DBMS versions, for the data warehouse DB, are Microsoft SQL Server 2000, 2005, 2008, 2012, 2014, 2016, 2019.

Supported Management packs

In the following table are shown the Management Packs supported by "Moviri Integrator for BMC Helix Continuous Optimization - Microsoft SCOM".

Virtual Machine Manager

In order to extract virtualization metrics, the connector requires:

1. The Microsoft System Center Virtual Machine Manager (SCVMM from now on) management tool, which is in charge of collecting data for the Microsoft virtual environment
2. A working connection among Microsoft SCOM and System Center Virtual Machine Manager
3. The SCOM Virtual Machine Manager Management Packs

The following matrix presents the supported versions of the above-mentioned components.

Datasource Check and Configuration

Preparing to connect to the data source software

As a first step, please ensure that the correct version of SCOM is installed. You can perform this check from the "About" section of the SCOM Console (as shown in Figure 2).

 
Figure 9: Version Check in MS SCOM Console

In order for SCOM to store historical metrics, it needs to be configured to use data warehouse database.
Please also ensure that the data warehouse DBMS version is compliant with the requirement described in Section 2.5.
The connector needs a Login that has at least read rights on the data warehouse DB. SQL Server allows two authentication types:

• SQL Server Authentication
• Windows Authentication

If your SQL Server authentication mode is set to "Mixed mode" it is suggested to use a user with SQL Server Authentication for the connector, however both Authentication types are supported.

Connect SCVMM to SCOM

A working network connection between VMM and SCOM is required to have SCVMM collected metrics available into the SCOM datawarehouse.

SCVMM 2012 to SCOM 2012 and SCVMM 2016 to SCOM 2016

The connect SCOM 2012 to SCVMM 2012 / SCOM 2016 to SCVMM 2016 you need to access the SCVMM Settings panel and click on "Settings" (Red square), then on "System Center settings" (blue square) as shown in the following figure

Now you have to click on "System Center Operations Manager", access to properties (by right click), then insert SCOM management server hostname and credentials, as shown in the following figure:

VMM 2008 to SCOM 2007 R2

The same steps are valid also for the connection between SCOM 2007 and VMM2008 and shown in the following figure:

Install Moviri VM Detector management pack in SCOM (optional)

Standard windows management pack do not expose any property to detect if a system is a VM or a physical server, in order to get that information “Moviri VM Detector” management pack enable SCOM to collect the VM technology using a WMI query on all agent monitored systems.

To activate the “Moviri VM Detector” on SCOM

1. Extract the MoviriVMDetector.xml file from the ETL package
2. Import MoviriVMDetector.xml in SCOM (Administration -> Management Packs -> Import Management packs)
   
   
3. Wait for the data collection in SCOM DWH (up to 1 day)

SCOM Data Availability Check

In order to make ensure data is available in SCOM DWH, please make sure that the SCOM Monitoring views do present the relevant metrics by selecting them and setting a proper chart time frame. Please repeat this step for each desired management pack and entity type you are interested in.
As an example, the following figure shows the Host Performance view of the SCOM 2012 Virtual Machine Manager Management Pack.

Connector configuration

Connector configuration attributes

This section describes the available configuration properties for the integrator.

(*)The property "Relocate all series to time zone" has to be filled with the name of the time zone in IANA (Internet Assigned Numbers Authority, see http://www.iana.org/time-zones/) format. Some time zone names examples are the following: Europe/Rome, Europe/London, America/New_York, Australia/Melbourne.
(**)"Compute size and occupation metrics even for aggregated data": some of the metrics the connector extracts are not directly available in the SCOM data warehouse database. They are computed using other metrics – they are called derived metrics (for more details, see section 6.2.3) . It could happen that the base metrics (upon which the derived are computed) are collected from the SCOM Agent at different time instants. In this case, if the connector is extracting data at raw level, the related derived metric will be not computed.

If the connector is extracting data at hour level, there might be a chance that the related derived metric could be wrong, in case the SCOM Agent sample them at different instants. For those reasons, it is possible to choose whether enable or not the computation of the derived metrics, setting properly the "Compute size and occupation metrics even for aggregated data" property.

Management pack and metric classes valid combinations

To give a detailed view on metrics and management pack, the property “Metric classes to import” is mapped on the following Management packs. 

Extensible module – Customized metric support

Moviri integrator for Capacity Optimization supports the most common Management Packs, the extensible module is a framework that to enable the extension of the metric supported in any Management packs.

To enable the extensible module it’s necessary to check “extensible module” in the property Management pack to import and to set the location of the mapping file in Extensible module file location.

To write correctly a new mapping file please follow the procedure shown in Extensible-module-for-SCOM-Integrator.

Filtering systems with whitelist and blacklist

In order to restrict SCOM ETL extraction only to chosen entities, it is possible to activate filtering by entity type based on combination of whitelist and blacklist. If whitelist and blacklist are empty, no filter will be applied.
The two properties accept list of expressions semicolon separated, and expression can be:

• Flat Strings (e.g. websrv105;appsrv20)
• SQL like operator expressions: accept wildcard "%" (matches any group of characters) and "" (matches any single{_} character)

Figure below shows a configuration example: filtering is active on physical servers, databases and virtual machines.

Physical servers:

Included: all servers matching "websrv%" or "appsrv%"
Excluded: "websrv25"

Databases:

Included: all DB instances
Excluded: all DB instances matching "dbprod%"

Virtual machines:

Included: all VM matching "vm1"
Excluded: no VM excluded

Database Connection parameters configuration

Moviri ETL allows to connect to SCOM DWH using SQL Server Authentication or Windows Authentication.Below are shown how to configure the database connection attributes in "Connection parameters" tab
SQL Server Authentication

• Database type: "Microsoft SQL Server"
• Database user: <SCOM_DWH_username>
• Database password: <SCOM_DWH_password>
• JDBC Driver: com.microsoft.sqlserver.jdbc.SQLServerDriver
• Host Name: <SCOM_DWH_hostname>
• Port: <SCOM_DWH_port>
• Database Name: <SCOM_DWH_DB_name>

Windows Authentication

• Database type: "Other Database"
• Database user: <Domain_username>
• Database password: <Domain_password>
• JDBC Driver: net.sourceforge.jtds.jdbc.Driver
• JDBC Url: jdbc:jtds:sqlserver:// <SCOM_DWH_hostname>:<SCOM_DWHport>/<SCOM_DWH_DB_name>;domain=<Domain_name>

Filter virtual machines using Moviri VM Detector management pack

If Moviri VM Detector Management Pack has been installed on SCOM, detected VMs can be excluded from the data import. In order to do so:

1. Enable the filtering in ETL configuration in “SCOM – Data Selection” panel
   
2. Run the ETL (first in simulation mode) and verify the execution logs
   

Connector configuration example

The following figure shows an example of a valid ETL configuration.

In this particular configuration, the following properties are set:

• Data about all metrics classes is imported
• Metrics are retrieved from all supported management packs: Windows, SQL Server and Hyper-V;
• Metrics are retrieved for all the available systems (no filters are set)
• Hour data is imported
• The series will be relocated to time zone "Europe/Rome"
• No time zone info will be assigned to each data series
• If lastcounter has not been already defined, data is retrieved from 2014-09-25 00:00:00
• In order to control data volume extraction, each run extract maximum 5 days of data

Historical data extraction

In order to perform an historical import, in tab "SCOM – Settings " is available the Recovery mode option. When activated, fields "Since date" and "To date" will be shown, and the task imports data for the defined time interval without taking into account the last counters and without modifying them, "max days to extract in a single run" property is ignored too.

Supported Platforms

The following table outlines the supported platform and the mapping to BMC Helix Continuous Optimizationentity types.

Hyper-V View

Hyper-V View leverages data imported by Moviri Integrator. For more information, see Hyper-V views.

Troubleshooting

Missing data can represent a common problem; as shown above, in this case a WARNING would be associated to the ETL task. This can be due to a number of scenarios, among which:

• The data is not available on SCOM. In this case, it is recommended to check the SCOM Console settings and status. May be that SCOM is not collecting data.
• The data is available on SCOM but the ETL is not able to extract it. For Raw data, this can be due to the lack of overlap of the "extraction window", as shown in Figure 19. In the Figure, the ETL is trying to extract data for a time period for which there are no data in the data warehouse DB. In any case, it is possible to manually run the actual queries used for data collection against the SCOM data warehouse, as they are listed in the ETL log (when the ETL Log Level is set to 5 – Medium), in order to ascertain the presence of the data in that desired time range.

• The parameter "Max days to extract" is set (greater then 0), if the lastcounter is too far in the past to find data: the oldest data available must be lower than the lastcounter+max days to extract (see Figure 20)

  

• In big SCOM environments the max days to extract property can be useful to avoid out of memory exception due to very large query result set; on the other side, if the datasource shows lacks of data (holes) more extended than the max days to extract, the lastcounter will not be updated and the ETL does not extract any data until the lastcounter is manually updated.
  In case of such a scenario, a better strategy can be to unset the max days to extract property and run the ETL more frequently (up to every hour, at least 3/4 times a day) so that each extraction collects less than one day of data without setting any limit.

Entity Naming Conventions

SCOM managed entities are mapped to BMC Helix Continuous Optimization ones according to their Display Name.

Exceptions to this rule are described in the next sections.

SCOM Lower-level entities

The BMC Helix Continuous Optimization entity name is extracted from the Path or Fullname SCOM managed entity attributes, in case SCOM metrics refers to lower-level entities.

This applies for example to Disks, which SCOM models as first-class managed entities (Display Name is for example “C:”) but BMC Helix Continuous Optimization treats them as metric subobjects assigned to the host system.

Hyper-V Root Partition

Every Hyper-V hypervisor hosts a particular virtual machine for management purposes, called root (or parent or host) partition. The root partition has the same name of the hypervisor system.

As traditional agents can be installed on this partition (such as SCOM Agent for Windows monitoring), metric clashing could possibly occur if the agent inside the root partition measures metrics that are also provided at the hypervisor level (i.e. CPU_UTIL).

In order to avoid such circumstances and clearly distinguish between the two concepts, BMC Helix Continuous Optimization represents root partitions and hypervisors as distinct entities.

To achieve this, the BMC Helix Continuous Optimization name of the root (parent) partition is prefixed with a specific label (“Root:”). A prefix is also used for naming lookup: the lookup name is prefixed with the “hvrootpart:” label for this purpose.

Lookup Fields

Moviri Integration for BMC ETL lookup mechanism. One single entry is used as entity identifier: Lookup field = “DEFAULT”. The following table shows, for each supported entity, what information is used as lookup value field. BMC Helix Continuous Optimization connector uses standard BMC Helix Continuous Optimization

Here below some lookup tables as example:

Object Relationships

The connector provides a set of object relationships that aims at organizing imported entities in BMC workspace according to the following rules. 

Assignment of imported entities to a specified domain

The connector arranges the collected systems under a domain of choice.

• Relationship Type: APP – SYS [identified by its data source name, DS_SYSNM], where APP is the domain of choice (as selected in the “Object Relationships” tab in the Run Configuration wizard)
• Meaning: To group systems collected from the same data source

A hierarchy rule associated to the ETL is automatically created by BMC to process the Object-Relationship data. 

Windows cluster, Windows computer and cluster disks

The Cluster nodes and cluster disks are shown in the BMC tree as children of the cluster they belong, 

Figure 25 – SQL Server tree example.

The cluster system which own the nodes relationships could be monitored from others management packs as well, and depending on cluster name configuration, the same cluster can be named with its name without domain or with its FQDN.

Using the default ETL loader settings the domain suffix is not removed from datasource name, and a duplicate cluster system can be created in Helix CO.

To prevent duplicate cluster system creation, it’s suggested to verify what system will be created running the ETL in simulation mode and looking at the log (if the ETL has already ran, check also the existing system in CO).

When a potential duplicate cluster has been detected, one of the following actions can be performed:

• Import the duplicate cluster and perform a manual reconciliation
  • Does not impact existing systems and other ETLs
  • Must be done manually (not suggested for a high number of clusters )
  • From the ETL loader configuration (advanced), change the “Remove domain suffix from datasource name” to true
    • If the lookup table is shared, can impacts other ETLs system naming
    • Suitable for a SCOM ETL first run, if the SCOM ETL is currently in place, it will duplicate all systems

SQL Server Instances and Databases

The SQL Server instances are shown in the BMC tree as children of the host on which they are running (Figure 25 – SQL Server tree exampleFigure 25). 

In the above figure each host has one or more SQL Server instances.

Each SQL Server instance could have one or more Databases. The connector represents databases as subobject of their SQL Server instance.

The connector also imports, for each database, data about filegroups. Filegroups are represented within BMC as subobject of the SQL Server instances.  

All the database space metrics are associated to the SQL Server instance. They have as subject the database name if they are related to the whole DB, while, if they are related to a specific filegroup of a database, they have a subobject composed by the database name and the filegroup name.

Hyper-V Clusters, Hosts, Datastore and Virtual Machines

Hyper-V entities are organized according to the following relationships:

• Hosting: represents the relation existing between an Hyper-V host and a Virtual Machine running on it
• Containment: represents the relation existing between an Hyper-V Cluster and the contained Hyper-V host / cluster shared volumes

An example of such relationships is presented in Figure 26.

In this particular example, the cluster system “Cluster04” contains a single Hyper-V system “hyperv-laptop” which is hosting three virtual machines (“Root:hyperv-laptop”, “WinSrv2008 – 3”, “WinSrv2008 – 4”). Furthermore, the standalone Hyper-V system “movsrv19” is hosting six virtual machines (“Mediaroom 2008”, “Media”, “RD”, “Root:movsrv19”, “WinSrv2008”, “WinSrv2008 – memdyn”).

A relation among the Hyper-V host and its root partition is also provided.

SQL queries to execute on SCOM DWH

(Q1) list all installed MPs

 (Q2) list all PERF objects and counters given a MP

(Q3) list all systems with max data timestamp given a combination of MP, object, counter

(Q4) list all CONF properties given a MP

(Q5) list all systems with value and validity period given a combination of MP, Propertyname