BMC - TrueSight Operations Management 10.1/10.5/10.7/11.0/11.3 Generic extractor

TrueSight Capacity Management supports integration with TrueSight Operations Management versions 10.1, 10.5, 10.7, 11.0, 11.3 for any platform through the BMC - TrueSight Operations Management 10.1/10.5/10.7/11.0/11.3 Generic extractor. 

This extractor is a Java-based extractor that uses REST APIs to extract performance and configuration data from TrueSight Operations Management. The extractor communicates with the Data Integration APIs of TrueSight Operations Management. For more information, see  Retrieving Presentation Server data. .

Note

You can use the BMC - TrueSight Operations Management 10.1/10.5/10.7/11.0/11.3 extractor for the following platforms:

  • IBM AIX
  • Oracle Solaris
  • Stand-alone hosts supported for Windows, Linux, AIX, HP-UX, and Solaris
  • Hyper-V (Clusters, hosts, and virtual machines)
  • KVM

If you are using TrueSight Operations Management version 10.0 or earlier, use the BMC - TrueSight Operations Management 9.0/9.5/9.6/10.0 extractor or BMC - TrueSight Operations Management 9.0/9.5/9.6/10.0 Generic View Extractor, depending on the platforms for which you want to extract data. 

This topic contains the following sections:

Integrating with TrueSight Operations Management by using the generic extractor

Step #StepReference section
1

Create a JSON file.

The JSON file enables you to extract data from a set of Knowledge Modules (KM) for a new platform through TrueSight Operations Management.

Creating a platform definition JSON file
2Configure and run the extractor. Integration steps


To create a platform definition JSON file

The JSON file defines the mapping of the source (TrueSight Operations Management) devices, instance types, and parameters to the destination (TrueSight Capacity Optimization) entities and metrics. 

PATROLTrueSight Operations ManagementTrueSight Capacity Optimization
ParameterAttributeMetric
PATROL Agent/Remote Server/Application Instance with metadataDeviceEntity (system/business driver)
Application classMonitorDevicetype
InstanceInstanceRelationship/child Entity (system/business driver)

To create the JSON file:

  1. Analyze the Truesight Operations Management Knowledge Module (KM) to understand hierarchy of the devices and instances.

    1. Log on to the TrueSight console and access the Monitoring > Devices page.

    2. Click a device tile to access the monitors and monitor instances associated with that device.

      Alternatively, you can also use the PATROL console. To do so, access the PATROL console and verify the hierarchy tree to identify which node in the tree maps to a device, application, or monitoring instance.

  2. Analyze the KM to understand the configuration and performance parameters to be extracted.
    Check the KM PSL output to understand the mapping.

    KM propertyTrueSight Capacity Optimization mapping
    MetaFQDNsystem name/lookup field
    MetaEntityTypeentitytype
    MetaTokenIDlookup field
    Instancesrelationships/child devices
    Config variablesconfiguration metrics

    You can check the preceding properties by using PSL commands on the PATROL Console or Agent Query Actions on TrueSight Operations Management.

    The following table provides details of the properties, how to retrieve it via PSL commands, an example for the PATROL for VMware VSphere KM and how to map it into TrueSight Capacity Optimization.

    KM propertyDescriptionPSL command and example
    MetaFQDN

    DNS name/IP address

    DNS name and forward slash (/) are mandatory; IP address is not.

    %PSL print(get("/<APPLICATION_CLASS>/<INSTANCE_ID>/MetaFQDN"));

    Example:

    OS>%PSL print(get("/VSM_VM/vw-pun-bpm-qa54@vm-62_vw-vc-qa54-vm2/MetaFQDN"));

    Output: vw-vc-qa54-vm2/10.129.56.174


    Map DNS Name and IP Address to TrueSight Capacity Optimization system name and a possible lookup field.
    MetaEntityType

    String that identifies the type of devices.

    For example, Amazon.EC2 for AWS EC2, VmWare.Host for VMware Host, VmWare.Virtual_Machine for VMware VM, and so on.

    %PSL print(get("/<APPLICATION_CLASS>/<INSTANCE_ID>/MetaEntityType"));

    Example:

    OS>%PSL print(get("/VSM_VM/vw-pun-bpm-qa54@vm-62_vw-vc-qa54-vm2/MetaEntityType"));

    Output: VmWare.Virtual_MachineMap existing devicetypes and MetaEntityType with existing TrueSight Capacity Optimization entity types
    MetaTokenIDString that represents the token ID. Through token IDs, the same devices that are monitored via various KMs are reconciled and data is used for correlation. It uses the CMDB token ID format.
    For example, EC2-UUID:<InstanceID> for AWS EC2, hostname:domain (test-123:bmc.com), and so on.

    %PSL print(get("/<APPLICATION_CLASS>/<INSTANCE_ID>/MetaTokenID"));

    Example:

    OS>%PSL print(get("/VSM_VM/vw-pun-bpm-qa54@vm-62_vw-vc-qa54-vm2/MetaTokenID"));

    Output: VI-UUID:4237b33f-30a4-da23-afd4-9fa3163e7370Map this output to a TrueSight Capacity Optimization lookup field in the JSON file.
    InstancesEach running copy of an application discovered by PATROL is called an instance of the particular application class.

    %PSL print(get("/<APPLICATION_CLASS>/instances"));

    Example:

    OS>%PSL print(get("/VSM_VM/instances"));

    Output: vw-pun-bpm-qa54@vm-62_vw-vc-qa54-vm2

    Map instances in the KM to relationships or child devices in TrueSight Capacity Optimization. For more information, see Understanding entity identification and lookup.

    Configuration variablesThe configuration variables are found in the KM XML file. These are defined with the configVariables attribute.
    For example, configVariables="osName"
    Map configuration variables in KM to configuration metrics in TrueSight Capacity Optimization. Navigate to the Capacity Optimization web console Administration > DATA WAREHOUSE > Datasets & metrics and look for a suitable configuration metric (or create a custom one) to map the configuration variable.
  3. Verify the KM parameters to TrueSight Operations Management mapping and add it to the JSON file.
    1. Check the parameters in the KM XML file. 
      The default location is <TrueSight Presentation Server installedDirectory>\truesightpserver\modules\cma\pproxy\depot_directory\bmc_products\kmfiles\<KM name>\lib\knowledge.
      From the Applications tag in the XML, identify the application class that contains the list of metrics (parameters).
      An example for the VSM_CPU_VM application class in the VMware vSphere KM. The highlighted parameters are column names that contain the metric and its values.

       Click here to expand...

    2. After you identify the application class name, prefix _PATROL__ to the application class name to form the view name that you need to add in the custom JSON file. This is the source view name that contains the metric and its value.
      For example, "src_monitor_type": "_PATROL__VSM_CPU_VM"

      The entry _PATROL__VSM_CPU_VM corresponds to a table name in the TrueSight Operations Management SAP SQL Anywhere (Sybase) database, where the data is located. If the database is Oracle, the name is similar, but with an ID inside. This is due to an Oracle database table name limitation of 32 characters.

      The parameter name corresponds to a column in the same table and needs to be inserted in the JSON file as a value of src_metric.
      For example, "src_metric": "CPU_Guaranteed"

    3. To identify the TrueSight Capacity Optimization metric that is equivalent to the KM parameter, navigate to the TrueSight Capacity Optimization web console Administration > DATA WAREHOUSE > Datasets & metrics and look for a suitable metric (or create a custom one) to map the attribute.

      See the following example for filtering the metric name.


      The name of the TrueSight Capacity Optimization metric chosen or created needs to be entered in the JSON file as a value of dst_metric.
      For example: "dst_metric": "BYGMCPU_CPU_GUARANTEED"

       Click here to view a sample JSON snippet...

  4. Download the sample JSON file.
  5. Update the following information in the JSON file: 
    1. Define a platform “type” name for the JSON file; for example, "VMware KM”.
    2. Specify entity-level structure details:
      1. Source-to-Destination mapping
      2. Children for the entity
      3. Lookup entries for the Destination
    3. Specify metric-level mapping details for Basic metrics:
      1. Basic metrics are a 1:1 mapping between the Source and the Destination metric with a conversion formula, if required
      2. Specify how to map “BY” metrics and how to filter subobjects using criteria
    4. Define the Derived and Aggregated metrics, if required.

       About Derived and Aggregated metrics

      • “Derived” metrics are computed at an entity level by using more than one metric of the same entity, using formula.
      • “Aggregated” metrics are computed at an entity level by using metrics of different entities that can be children, parent or subject, using formula.

      For more information about the structure and content of the file, see Platform definition JSON file structure.


  6. Save the JSON file to your local computer. For example, <platformname>-platform-def.json.
  7. Upload the file to a folder on the ETL Engine server.

Integration steps

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL page is displayed.
  3. In the Run configuration tab, for the ETL module property, select BMC - TrueSight Operations Management 10.1/10.5/10.7/11.0/11.3 Generic extractor.
  4. Specify values for the following properties under each expandable tab:

    Note

    Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

    Basic properties

    Property Description
    Run configuration
    ETL task name Accept the default name based on the selected ETL module or specify a different name for the ETL task. Duplicate names are allowed.
    Run configuration name Review the default name. This field is used to differentiate configurations that you can specify for the ETL task. You can then run the ETL task based on the configuration name.
    Deploy status
    Select Production or Test to mark the ETL tasks. For example, you can start by marking the task as Test and change it to Production after you have seen that you are getting what you wanted.
    Description (Optional) Enter a brief description for this ETL.
    Log level Select how detailed you want the ETL log to be. The log includes Error, Warning and Info type of log information.
    • 1 - Light: Add bare minimum activity logs to the log file.
    • 5 - Medium: Add medium-detailed activity logs to the log file.
    • 10 - Verbose: Add detailed activity logs to the log file.

    Note: Log levels 5 and 10 are typically used for debugging or troubleshooting ETL issues. Using a log level of 5 is general practice; however, you may choose level 10 to get a high level of detail while troubleshooting.

    Execute in simulation mode Select Yes if you want to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues.
    When set to Yes, the ETL will not store actual data into the data warehouse. This option is useful while testing a new ETL task.
    Module selection

    Ensure that the Based on datasource option is selected.


    Note: If you select Based on Open ETL template, TrueSight Capacity Optimization is integrated with a Generic extractor based on the selected Open ETL template. For more information, see Generic ETL based on a template.
    ETL module

    Ensure that BMC - TrueSight Operations Management 10.1/10.5 /10.7/11.0/11.3 Generic extractor is selected.

    Note: You selected this property in the earlier step.

    Associate new entities to

    Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

    infoInfo: By default, a new domain is created for each ETL, with the same name as the extractor module. As the ETL is created, a new hierarchy rule with the same name as the ETL task is created automatically, with an active status. if you update the ETL specifying a different domain, the hierarchy rule will be updated automatically.

    Select any one of the following options:

    • New domain: Create a new domain. Specify the following properties under New domain:
      • Parent: Select a parent domain for your new domain from the domain selector control.
      • Name: Specify a name for your new domain.
    • Existing domain: Select an existing domain. Make a selection for the following property:
      • Domain: Select an existing domain from the domain selector control.
    TrueSight Operations Management Connection
    Data Integration API URL

    Specify the host name of the Presentation Server in the provided URL format: https://<host>/tsws. You must replace <host> with the host name.

    Username Type the user name to connect to the Presentation Server. You can use the default user, apiuser.
    Password Type the password for the specified user name to connect to the Presentation Server. The password for the default user, apiuser, is displayed in encrypted format.
    Tenant name Specify the name of the tenant for which the data needs to be extracted. The tenants are configured on the Presentation Server. The default tenant BmcRealm is displayed.
    Platform definition JSON file path Specify the path where you saved the JSON file that has the platform specific entity definition and their configuration and performance metrics mapping. For more information, see Creating a platform definition JSON file.
    ETL task properties
    Task group Select a task group into which you want to classify this ETL.
    Running on scheduler Select the scheduler on which you want to run the ETL.
    Maximum execution time before warning Select the number of hours, minutes, or days to to execute the ETL for before generating warnings, if any.
    Frequency Select the frequency of ETL execution. Available options are:
    • Predefined: Select a Predefined frequency from Each Day, Each Week or Each Month.
    • Custom: Enter a Custom frequency (time interval) as the number of minutes, hours, days or weeks to run the ETL in.
    Start timestamp: hour\minute (Applies to Predefined frequency) Select the HH:MM start timestamp to add to the ETL execution that is running on a Predefined frequency.
    Custom start timestamp Select a YYYY-MM-DD HH:MM timestamp to add to the ETL execution running on a Custom frequency.


    To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

    Property Description
    Run configuration
    Datasets

    Select or clear metric groups for which data will be populated in the Available datasets. The connector allows you to choose only from the given list of datasets, and you cannot include additional datasets to the run configuration of the ETL.

    • Click Edit.
    • Select one (click) or more (shift+click) datasets that you want to exclude from Available datasets and click >> to move them to Selected datasets.
    • Click Apply.
    Collection level
    Metric profile selection

    Select either of the following options:

    • Use Global metric profile: Select this option to use an out-of-the-box global profile that is available on the Adding and managing metric profiles page. By default, all ETL modules use this profile.
    • Select a custom metric profile: Select from any of the metric profiles that you add in the Add metric profile page (Administration > DATAWAREHOUSE > Metric profiles).

    For more information, see Adding and managing metric profiles.

    Levels up to

    Define the amount of metric imported into the data warehouse. Increase the level to add load to the data warehouse or decrease the metric level to reduce the number of imported metrics.

    Choose the metric level to apply on selected metrics:

    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended

    For more information, see Aging Class mapping.

    Additional properties
    List of properties

    Specify any additional properties for this ETL that act as user inputs during execution. You can specify values for these properties either at this time or from the "You can manually edit ETL properties from this page" link that is displayed for the ETL in view mode.

    • Click Add.
    • Add an additional property in the etl.additional.prop.n box.
    • Click Apply.
    • Repeat this task to add more properties.
    Loader configuration
    Empty dataset behavior Select one of the following actions to take if the loader encounters an empty dataset:
    • Warn: Abort the loader.
    • Ignore: Ignore the empty dataset and continue parsing.
    ETL log file name Type or verify the name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
    Maximum number of rows for CSV output Select a number which limits the size of the output files.
    CSV loader output file name Type or verify the name of the file generated by the CSV loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
    Capacity Optimization loader output file name Type or verify the name of the file generated by the TrueSight Capacity Optimization loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
    Detail mode Select the level of detail:
    • Standard: Data will be stored on the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw also: Data will be stored on the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw only: Data will be stored on the database in a table only at Raw granularity (as available from the original data source).

    For more information on granularities, see Accessing data using public views and Sizing and scalability considerations.

    Reduce priority

    Select one of the following priorities:

    • Normal
    • High
    Remove domain suffix from datasource name (Only for systems) Set to True to remove the domain name from the data source name. For example, server.domain.com will be saved as server.
    Leave domain suffix to system name (Only for systems) Set to True to retain the domain name in the system name. For example: server.domain.com will be saved as the system name.
    Update grouping object definition Set to True to allow the ETL to update the grouping object definition for a metric loaded by an ETL.
    Skip entity creation (Only for ETL tasks sharing lookup information with other tasks) Set to True to prevent this ETL from creating an entity. With this setting, the ETL discards data from its data source for entities not found in BMC TrueSight Capacity Optimization. It uses one of the other ETLs that share lookup to create the new entity.
    Scheduling options
    Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
    Day of week mask Select the days so that the task can be executed only during the selected days of the week. To omit setting this filter, do not select any option for this field.
    Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
    Apply mask validation Set to False to disable the preceding Scheduling options that you specified. This property is useful if you want to temporarily turn off the mask validation without removing any values. By default this property is set to True.
    Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to indicate the time to wait before the task must be executed. After the task is scheduled, the task execution starts only after the specified time passes.
    Enqueueable Select one of the following options:
    • False (Default): While a task is running, additional execution commands are ignored.
    • True: While a task is running, additional execution commands are placed in a queue and executed in order after the current task is finished.

  5. Click Save.
    You return to the Last run tab under the ETL tasks page.
  6. In simulation mode, validate the results: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click Run  to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    • OK: The ETL executed without any error in simulation mode.
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  7. After you verify that the ETL is running correctly, switch the ETL to production mode:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL name under the Name column.
    2. In the Run configurations table in the ETL details page, click Edit  to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  8. Locate the ETL in the ETL tasks table and either schedule an ETL run or click Run  to run it now.
    After you run or schedule the ETL, it will extract the data from the source and transfer it to the TrueSight Capacity Optimization database.

Was this page helpful? Yes No Submitting... Thank you

Comments