Page tree
Skip to end of metadata
Go to start of metadata

Updating the data warehouse (DWH) is one of the most important activities of BMC TrueSight Capacity Optimization. The Data Warehouse administration menu lets you inspect and tune the mechanisms that control it. To access the Data Warehouse menus, click Data Warehouse from the Administration tab.

Tip

Near-Real-Time Warehousing is a process that stores, organizes and calculates statistics for the collected data. This service is always running inside the Data Hub.

The data warehousing activity consists of:

  • Data aggregation
    • ETL tasks collect data into a stage table
    • The warehousing engine calculates aggregations, and splits data in summary tables at different time resolution levels (detail, hour, day, month)
    • Data is aggregated based on hierarchical rules (derived rows); this process is called hierarchical data aggregation
  • Data classification
    • The day and hour class definitions are used to classify data as specified by the Calendar
  • Data aging
    • Data classified as old by customizable aging parameters is deleted from the warehouse tables
  • Custom statistics
    • Additional statistics to perform calculations on data series can be created (For example, percentiles or baselines)

Additional information

Data collected by ETL tasks is not available to the analysis module before it is processed by the warehousing engine.

Data flow reports allow you to keep BMC TrueSight Capacity Optimization imports under control, as the number of rows processed per day is a good indicator of the system's health.

When performing historical imports (for example, if you are planning to bulk load more than two million rows), it is strongly recommended to split the data in smaller chunks, limiting the amount of information processed at one time. This prevents congestion in the Near-Real-Time Warehouse engine.

The DWH information model

The BMC TrueSight Capacity Optimization Data Warehouse can host different types of data:

  • Time series(TS): Performance or business driver data that represent a metric over time, with two subtypes:
    • Sampled, that is metric samples, at regular time intervals
    • Delta records
  • Custom structures(CS): Data that represent generic records with custom attributes, with two subtypes:
    • Buffer tables, containing data that is copied into BMC TrueSight Capacity Optimization for further processing, but is generally not important for direct analysis
    • Item-level detail tables, containing data that represent the details of an item that are important for analysis purposes. For example, errors for a specific page
  • Object relationships (OBJREL): Data that represent relationships between entities
  • Events (EVDAT): Data that represents events

Additional information

For more information about data structure, see Data format.

The primary purpose of the BMC TrueSight Capacity Optimization DWH is the collection of historical time series.

Tip

A time series is a sequence of samples or statistics for a certain measurement, each corresponding to a point in time. The BMC TrueSight Capacity Optimization DWH contains both time series samples and statistics, aggregated at different time resolution (hour, day, month).

All time series are associated with a measured object, described according to a reference model. In its most general form, the reference model for measured objects is displayed in the figure below.

The model comprises of the following components:

  • An entity represents a single system (for example, a database instance) or a business driver, that is the load a given application undergoes. For example, the load of an FTP server. Refer to Systems and business drivers for details.
  • An object is a metric for a system resource or a business driver for which data is collected. For example, the CPU utilization percentage of a server or the FTP transfer bit rate.
  • The location tracks the physical location from which a metric was observed. For example, the FTP transfer bit rate when a file is downloaded from New York or from Milan.
  • subobject represents finer details of a metric. For example, a metric measuring the free space of a disk could have details about the free space of each disk partition as its subobjects.

Each available object/metric has a standard name which adheres to the naming convention defined in the datasets. For a complete listing, refer to BMC TrueSight Capacity Optimization ETL Development Studio (also see Developing custom ETLs), or to the Datasets and metrics page in the Data Warehousing menu of the Administration section.

Each metric has a type which defines the measure unit and how statistics about that data should be collected. A complete listing is given below:

Name

Description

GENERIC

Generic counter, absolute value

PERCENTAGE

Percentage counter

COUNT

A count of events, absolute number

RATE

A frequency, in events/sec

POSACCUM

Positive accumulation counter

CONF

Configuration data (string)

NEGACCUM

Negative accumulation counter

ELAPSED

Elapsed time, in seconds

WEIGHTED_GENERIC

Percentage counter, weighted

PEAK_PERCENTAGE

Peak Percentage counter

PEAK_RATE

Peak frequency, in events/sec

DELTA

Difference between subsequent samples

WEIGHTED_PERCENTAGE

Generic counter, absolute value, weighted

Measurement units and formats also use common standards:

Type

Format

Timestamps

YYYY-MM-DD HH24:MI:SS

Elapsed times

seconds (duration)

Percentage metrics

from 0 to 1

Rate metrics

events/sec

In summary, the reference model specifies:

  • The standard for object structure
  • The standard for metric names
  • The standard for measurement units
  • The time granularity, which is automatically adjusted by the data warehouse

Non-standard data sources

BMC TrueSight Capacity Optimization uses ETL tasks to import data that is collected by third party applications or logged by the monitored entities themselves. This data, from BMC TrueSight Capacity Optimization perspective, is persistent because another application takes care of its collection and storage. In a second phase, BMC TrueSight Capacity Optimization accesses this data and imports it into its own data warehouse.

Standard data imported by BMC TrueSight Capacity Optimization is always in the form of a time series, which means that it is described by a value (numeric or textual, for configuration properties) which changes over time.

The Data Mover components are used by BMC TrueSight Capacity Optimization to deal with data which does not respect the two aforementioned properties. The Data Mover is able to access data which is not a time series and store it into BMC TrueSight Capacity Optimization.

The workflow of these components is displayed in the following image.

The available options of the Data Warehouse menu are described in the following topics: