Moviri Integrator for BMC Helix Capacity Optimization - HP SiteScope

“Moviri Integrator for BMC Helix Continuous Optimization – HP SiteScope” allows extracting information from HP SiteScope monitoring tool, transferring monitored servers performance and configuration data relevant for capacity management purposes.

Requirements

Supported versions of data source software

HP SiteScope from version 10.12 (and following minor releases) to version 11.00 (and following minor releases)

Supported configurations of data source software

Moviri Integrator for HP SiteScope supports SiteScope instances configured to export data to an external database. See next section for information about the supported DBMS technologies.

HP SiteScope includes SOAP-based API to manage the configuration of the monitored environment. Access to this API is required too.

Supported DBMS technology for integration

Moviri Integrator for BMC Helix Continuous Optimization – HP SiteScope supports the following DBMS technologies as HP SiteScope export DB:

• MySQL DBMS. MySQL version 5 with the related JDBC connector available on MySQL official website is advised.
• Oracle version 9 or higher, with the related JDBC connector shipped together with BMC Helix Continuous Optimization.

Datasource Check and Configuration

Preparing to connect to datasource software

As a first step, please ensure that the correct version of SiteScope is installed. You can perform this check from the “Help” section of the SiteScope Console:      

Set up the DBMS for external logging

In order to enable the external database logging, you have to set up and configure a DBMS. Create a schema and a user with all privileges on it for reading and writing monitoring data.

Preliminary SiteScope checks

To configure your SiteScope instance for external database logging, please ensure the following requirements are respected:

1. Make sure you have a SiteScope login with necessary privileges to access Infrastructure Preferences inside your SiteScope instance.

   
2. By default, monitoring data are stored into SiteScope<YYYY-MM-DD>.log files under <SiteScope_root_path>\logs directory. Make sure that the date and number format of data is respectively yyyy-MM-dd HH24:mm:ss (e.g. 2012-01-01 00:00:00) and with ‘.’ as decimal separator (e.g. 344.78). If not so, configure appropriately the user locale settings under which SiteScope service runs.
3. Make sure to have a SiteScope login to use for web services access.
4. Ensure you have an account on the machine where SiteScope is installed with the necessary rights to start and stop SiteScope service.
5. Obtain the JDBC connector to use for the connection between your SiteScope instance and the DBMS and SiteScope ETL and the DBMS.

Configuring SiteScope for external database logging

To perform the configuration for external database logging, follow these steps:

1. If running, stop SiteScope service.
2. Open <SiteScope_root_path>\groups\master.config file (make a backup copy before editing) and search for the following parameters:
   1. _accessControlled: web services access type
   2. _logJdbcAutoCommitSiteScopeLog: number of monitors’ runs before collected data are committed to the database
   3. _logJdbcCreateSiteScopeLog: query for table creation
   4. _logJdbcInsertSiteScopeLog: insert query
   5. _logJdbcDriverSiteScopeLog: JDBC driver family
   6. _logJdbcURLSiteScopeLog: url for DBMS connection
   7. _logJdbcUserSiteScopeLog: DBMS user
   8. _logJdbcPasswordSiteScopeLog: encrypted password of DBMS user

Set them as follows:

• Enable the web services access with a SiteScope login authentication

_accessControlled=true

• Commit directly to the database the monitoring results so that SiteScope ETL can read fresh data everytime it runs

_logJdbcAutoCommitSiteScopeLog=1

• Create a table for logging data with an arbitrary number of columns depending on the configuration of the monitored environment. By default, all columns between 1 (datex) and 9 (monitorID) indicate the same properties for all SiteScope monitors. The next columns, from value1 to value10, have values which depend on each monitor and represent the measurements of metrics. Therefore, if necessary, you may need to increase the number of columns of the table to cover all measurements performed by a monitor. By default, the query is configured to store 10 measurements:

_logJdbcCreateSiteScopeLog=CREATE TABLE SiteScopeLog (datex VARCHAR(255), serverName VARCHAR(255), class VARCHAR(255), sample VARCHAR(255), category VARCHAR(255), groupName VARCHAR(255), monitorName VARCHAR(255), status VARCHAR(255), monitorID VARCHAR(255), value1 VARCHAR(255), value2 VARCHAR(255), value3 VARCHAR(255), value4 VARCHAR(255), value5 VARCHAR(255), value6 VARCHAR(255), value7 VARCHAR(255), value8 VARCHAR(255), value9 VARCHAR(255), value10 VARCHAR(255))

If you have a monitor which performs more than 10 measurements and you want to store them in your database, you will need to add the exceeding columns to the query. For example, if you have a monitor with the following measurements in your SiteScope console:

You can read the number of counters (therefore measurements in the database) active and add the exceeding number of columns to the query:

_logJdbcCreateSiteScopeLog=CREATE TABLE SiteScopeLog (datex VARCHAR(255), serverName VARCHAR(255), class VARCHAR(255), sample VARCHAR(255), category VARCHAR(255), groupName VARCHAR(255), monitorName VARCHAR(255), status VARCHAR(255), monitorID VARCHAR(255), value1 VARCHAR(255), value2 VARCHAR(255), value3 VARCHAR(255), value4 VARCHAR(255), value5 VARCHAR(255), value6 VARCHAR(255), value7 VARCHAR(255), value8 VARCHAR(255), value9 VARCHAR(255), value10 VARCHAR(255), value11 VARCHAR(255), value12 VARCHAR(255), value13 VARCHAR(255), value14 VARCHAR(255), value15  VARCHAR(255))

• After having configured the query to create the log table, you will need to make the changes (if any) also inside the insert query.  By default, the insert query is

_logJdbcInsertSiteScopeLog=INSERT INTO SiteScopeLog VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)

9 standard columns 10 counters measurements

with 19 parameters, the first 9 for standard columns from datex to monitorID and the other 10 for counters.

If you increased the number of columns to store data, you will have to add the correct number of “?”. In the previous example, 9 standard columns + 15 counters produces 24 columns to be filled by the insert query. Therefore other 5 “?” will be added:

  _logJdbcInsertSiteScopeLog=INSERT INTO SiteScopeLog VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)

9 standard columns 15 counters measurements

• Set the driver family to the MySQL server one

_logJdbcDriverSiteScopeLog=com.mysql.jdbc.Driver

• Configure the URL to connect to MySQL server

_logJdbcURLSiteScopeLog=jdbc:mysql://<server_address>:<server_port>/<database>

• Set the database user and password properties

_logJdbcUserSiteScopeLog=<mysql_user>

_logJdbcPasswordSiteScopeLog=<mysql_password>

• Move the DBMS driver (mysql-connector-java-<version>.jar) to the <SiteScope_root_path>\WEB-INF\lib directory
• Start SiteScope service
• In SiteScope Console, navigate to Preferences, Infrastructure Preferences, Monitor Settings and enable JDBC logging.
• Save the changes.

 

Instructions have been provided for the MySQL case. The Oracle case is omitted since it comprises the same steps with similar values. Please refer to HP SiteScope documentation for further reference.

1. Restart SiteScope service by clicking the corresponding button.
2. Verify the files error.log and RunMonitor.log in the <SiteScope_root_path>\logs\ directory for the correct activation of the external database for logging.

Some important considerations

1. Check your monitoring environment to correctly size the log table column number before activating the external database. You have to make sure that the maximum number of counters for every single monitor does not exceed the columns of the log table (as for the previous example).

2. Notice that the order of counters reported into the log table columns doesn’t correspond to the one showed in the “Dashboard” panel of SiteScope Console. The association order between counters and columns represents their selection order during the creation of the monitor and is reported in the “Properties” panel. For the previous example, after the default “counters in error” column, the other columns are associated with the counters as shown in the following figure:

    

   
   
   This leads to other 3 important points you should consider when activating database logging:
   
   • • If a counter of interest is in a position that exceeds the number of columns of the log table, you will not store it in your database unless you increase the number of columns.
     • If you change the counters selection order, SiteScope will save the last current configuration to associate columns to counters and will perform this change also in the database for the next measurements. The columns will store different counters than before and, as a consequence, the new configuration sent to SiteScope connector will produce a misinterpretation for data stored before the change.
     • The storage of new counters for a monitor doesn’t produce counters misinterpretation. Nevertheless you may need to increase the number of columns of your log table to store all data. To perform the change, you have to repeat the steps described above to modify the master.conf file. Before restarting SiteScope, you have to manually alter the table structure from inside your DBMS, adding new columns (the alter operation is not performed by SiteScope directly).
3. Database aging policies are not managed from SiteScope. Configure your policies and pay attention to them when using SiteScope connector for historical data recovery

Connector configuration

Connector configuration attributes

There are three sections (tabs) in the ETL run configuration page where the behaviour of “Moviri Integrator for BMC Helix Continuous Optimization - HP SiteScope” can be specified:

• Connection parameters (common BMC Helix Continuous Optimization setting)
• SiteScope - Settings
• SiteSCope - Data Selection

SiteScope - Settings

 

SiteScope - Data Selection

 

Guidelines to choose the most appropriate ETL configuration

In order to set the ETL configuration that match your needs, in this paragraph some common use cases are presented.

Scenario 1: “Typical” systems capacity analysis and planning

• Goal : ensure the right servers capacity (CPU, Memory, Logical/Physical disk and Network resource utilization)
• Environment size : medium/big
• Environment peculiarity : quite steady
• Planning time period :  long (6-12 months)
• Outputs: Analyses and forecast on resource utilization and violation threshold

Scenario 2: “in-depth” capacity analysis and planning for specific systems

• Goal : react on short-term growth (CPU, Memory, Logical/Physical disk and Network resource utilization)
• Environment size : small
• Environment peculiarity :   new service introduction / across a change of application or devices
• Planning time period :  short (weeks)
• Outputs: Growth trend, short-term forecast, correlation between resource utilization and service business drivers (users, transactions)

The choice of correct configuration brings you benefits also in terms of shorter execution time and saved disk space on database.

Every SiteScope ETL instance configuration applies to all systems selected, if there are different scenario needs for different systems, is recommended to create many ETL instances with different configuration which filters on particular systems (e.g. There are both a big and stable environment (scenario 1) and the introduction of a new service (Scenario2))

Data volume considerations

The following table is a reference to estimate the data extraction volume produced by the HP SiteScope connector as a function of the number of extracted days.

The results presented above assume the following connector configuration (as per metric filters specified in Scenario 1):

• 8 monitors for each system
• 10 counters per monitor
• Monitors’ runs frequency: 10 minutes

 

Under these assumptions, the estimated number of daily rows extracted by the connector, as a function of the number of servers, is presented below

.

Historical data extraction

In order to perform a historical recovery of data, please use “recovery mode properties” settings:
Recovery mode active=true and filling the two properties “since date” and “to date” in order to specify the time window. When recovery mode is active, the last counter will not be updated and max day to extract will be ignored.

If the ETL is newly created and has no last counter defined, the “default last counter” is used.

The pre-condition for an historical extraction is the presence of data inside the external database in the selected time period.

SSL Authentication

In order to communicate with Web Services with SSL protocol over HTTP (HTTPS) you must configure SiteScope to use SSL. Below an example of accessing SiteScope by HTTPS protocol.

 

To properly configure the ETL, you need to enable the SSL authentication property in SiteScope – Settings tab and specify correctly the IP of the host where SiteScope is installed and the port with SSL (8443 in the following example).

 

Supported Platforms

For all metrics imported by the ETL and described below in this section, the following apply:

• Supported OS families: Red Hat Enterprise Linux, Linux CentOS, Solaris, Windows NT
• Supported BMC Helix Continuous Optimization Entity Type: System (Generic)

Troubleshooting

For ETL troubleshooting, please refer to official BMC documentation available here.

Configuration and Performance Metrics Mapping

Performance Metrics

The SiteScope ETL connects to SiteScope and reads from external database monitors measurements performed on systems remotely. In the following table the supported metrics for OS families and the related SiteScope monitors and objects are reported.

Configuration metrics

This is a list of configuration metrics extracted by SiteScope ETL.

Derived metrics

In addition to the metrics available from the data source, the following metrics are computed by the connector:

The SiteScope ETL can compute a derived metric obtained by 2 source metrics only if they are included in the same monitor, i.e. they will be stored in the same raw inside the external database. For example if you want to include the SWAP_SPACE_UTIL metric for a Solaris system, you will need to instantiate a UNIX Resources Monitor which includes both the swap_resv and the swap_avail metrics related to the Memory object:

Metric Classes

All of the metrics managed by the connector are grouped in the following “Metric Classes”

• CPU and Memory Basic Metrics

• Memory Advanced Metrics

• Logical Disk Occupation Metrics

• Physical Disk Performance Metrics

• Network Interface Metrics

The following table shows for each metric class the contained metrics

The SiteScope ETL can be configured to collect only specific metric classes by configuring the related custom properties.

Overlapping Metrics

SiteScope can measure server performance metrics with two distinct categories of monitors:

• generic monitors, which apply to all OS families, and specific OS monitors, which apply to a specific OS type server. The first category is composed by CPU Monitor, Memory Monitor, Dynamic Disk Space Monitor and Disk Space Monitor (for SiteScope version 10).
• UNIX Resources Monitors and Microsoft Windows Resource Monitors.

This leads to some metrics measured by both generic and OS specific monitors with the consequence of repeating or overlapping those measurements. In order to avoid the overwrite of common metrics measurements, the connector implements a mechanism of precedence for which, if both OS specific and generic monitors are monitoring the same metric for the same server, the OS specific monitor will have precedence over the generic one and only its measurements will be imported. Be aware of what follows:

The ETL filters measurements only in case two monitors with overlapping metrics collect data for the same system

The ETL will not make distinction on the sub-resource monitored, only on the monitor metrics and the host monitored. For example, following the table 10, if both the Dynamic Disk Space Monitor and the Microsoft Windows Resources Monitor collect the metric BYLDISK_UTIL for the host 10.31.168.156 and the sub-resource “C:”, only the measurement performed by the Microsoft Windows Resource Monitor will be imported. Besides if the Dynamic Disk Space Monitor collects metrics on a “D:” sub-resource, also this will not be imported, even if the Microsoft Windows Resource Monitor is not making measurements on it (i.e. the mechanism doesn’t distinguish any sub resource, only checks for monitor, host and metric). In the example, to monitor the “C:” and “D:” sub resources you can use a Dynamic Disk Space Monitor alone, which collects data on both logical disks, or a Microsoft Windows Resource Monitor on both “C:” and “D:”.

Below the list of overlapped metrics:

Lookup Fields in SiteScope

Moviri Integrator for BMC Helix Continuous Optimization connector uses standard BMC Helix Continuous Optimization ETL lookup mechanism. One single entry is used as entity identifier : Lookup field = “DEFAULT”.

Object Relationships

Through Object-Relationship panel in the ETL edit run configuration,  the ETL arranges the collected systems under a domain of choice.

When the “Recreate SiteScope hierarchy” option is set to true (see Section 5.1.3), the connector builds the hierarchy of groups and systems of SiteScope inside CO. The groups of SiteScope are inserted under the root domain and mapped with corresponding BMC Helix Continuous Optimization domains; the monitors instantiated into SiteScope are associated to the related systems which will be put under the respective domains.

In the example below, each group under “Capacity” contains monitors associated to only one server, resulting in the following tree:

The correspondent structure in HP SiteScope is showed here:

Otherwise, if the “Recreate SiteScope hierarchy” option is set to false, all the systems related to SiteScope monitors will be inserted directly under the root domain (in the example below is called “Flat domain”).

BMC Helix Continuous Optimization automatically creates a hierarchy rule associated to the ETL to process the Object-Relationship data.