×
 
 
  •  
$helper.renderConfluenceMacro('{bmc-global-announcement:$space.key}')
BMC TrueSight Capacity Optimization 20.02 ... Collecting data Sentry ETL modules

Storage All-in-One ETL for TrueSight Capacity Optimization

Use the Storage All-in-One ETL for TrueSight Capacity Optimization to poll and collect configuration and performance data from several storage systems all at once. The ETL uses the Storage - All-in-One Extractor Service task to collect all the metrics required for performing critical capacity optimization tasks, such as: analysis, trending, simulation and analytic modeling.


To collect data by using the Storage All-in-One ETL 1.20.00 for TrueSight Capacity Optimization, perform the following tasks:

I. Complete the preconfiguration tasks.

II. Configure the ETL.

III. Run the ETL.

IV. Post-configuration steps.

Step I. Complete the preconfiguration tasks

The Storage All-in-One ETL supports the following platforms:

  • Ceph Storage
  • Dell EMC Isilon
  • Dell EMC PowerMax v4 (REST)
  • Dell EMC PowerStore (REST)
  • Dell EMC PowerVault (REST)
  • Dell EMC ScaleIO (REST)/Dell EMC PowerFlex (REST)
  • Dell EMC SC Series (REST)
  • Dell EMC Unity (REST)
  • Dell EMC VMAX (3rd Generation, All Flash or newer), and PowerMax (WBEM)
  • Dell EMC VMAX/PowerMax (REST)
  • Dell EMC XtremIO
  • Fujitsu Ethernus AF/DX (REST)
  • Hitachi G Series (REST)
  • Hitachi GUM (REST)
  • HPE Primera (WBEM)
  • Huawei OceanStor (REST)
  • IBM DS8000 (REST)
  • IBM DS888x (WBEM)
  • IBM Spectrum Scale
  • NetApp Active IQ Unified Manager (REST)
  • NetApp E-Series
  • NetApp ONTAP (REST)
  • NetApp SolidFire
  • Oracle ZFS (REST)
  • Pure Storage FlashArray (REST)
  • Pure Storage FlashBlade (REST)
  • Scality RING (Scality Supervisor API)

Important information

Some platforms can be monitored by both the Storage All-in-One ETL and a manufacturer specific ETL. Please read the following information:

EMC VMAX Systems

For recent systems (3rd Generation and AllFlash) the use of the All-in-One ETL is recommended as it is better optimized for these larger systems.  For older VMAX, VNX and Clariion systems, the EMC ETL should be preferred.

Pure and XtremIO Systems

Code Level 144 will be the last version released for these manufacturer specific ETLs.  It is recommended to migrate to the All-in-One ETL for monitoring Pure and XtremIO systems.

Ceph

The Storage All-in-One ETL relies on the Ceph REST API to collect capacity information about the Ceph cluster. The connection to the REST API is performed via the HTTP protocol. The ETL first identifies itself to the REST API with a user name and password to retrieve a bearer token from the API and then uses this token for further REST requests.

The default https port for the CEPH Rest API is 8003 (5000 for http).

Note: The ETL has been tested with the version 14.2.5 of the Ceph REST API.

Prerequisites

The Ceph dashboard must be activated with the following command: ceph mgr module enable dashboard.

Dell EMC Isilon

The Storage All-in-One ETL relies on the Dell EMC OneFS REST API to collect capacity information about the Dell EMC Isilon systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Isilon REST API is 8080.

The ETL requires at least version 8 of EMC OneFS.

Dell EMC PowerMax v4 (REST)

The Storage All-in-One ETL relies on the Dell EMC Unisphere REST API v10 to collect capacity information about the Dell EMC PowerMax v4 systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine
  • Requires Dell EMC Unisphere REST API v10 

Supported Platforms

  • PowerMax 2500
  • PowerMax 8500

Dell EMC PowerStore (REST)

The Storage All-in-One ETL relies on the Dell EMC PowerStore Management REST API to collect capacity information about the Dell EMC PowerStore systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine.

Limitations

  • The maximum number of supported volumes is limited to 2000.

Dell EMC PowerVault (REST)

The Storage All-in-One ETL relies on the Dell EMC PowerVault Management REST API to collect capacity information about the Dell EMC PowerVault systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Version 1.10.00 or higher of the engine.

Dell EMC ScaleIO (REST)/Dell EMC PowerFlex (REST)

The Storage All-in-One ETL leverages the REST API installed as part of the ScaleIO Gateway to collect capacity metrics about Dell EMC ScaleIO/Dell EMC PowerFlex systems.

The ScaleIO Gateway:

  • can be installed on the same node as other ScaleIO components if the node has adequate memory to run the Installation Manager (minimum of 2 GB) and any other applications.
  • must have connectivity to the data and management ScaleIO networks.

Java should also be installed on the server where you plan to install the ScaleIO Gateway.


To install the ScaleIO Gateway:

  1. Download from EMC Online Support the installation files needed for your operating system.
  2. Extract the installation files on the Installation Manager (IM) server
  3. For Linux:

    1. Run the following command (all in one line):

      GATEWAY_ADMIN_PASSWORD=<new_GW_admin_password> rpm -U /tmp/EMC-ScaleIO-gateway-1.32-XXX.X.noarch.rpm


      where <new_GW_admin_password> is the password to access the IM

  4. For Windows: 
    1. Run the .msi file, enter (and confirm) a new Gateway Admin password that will be used to access the IM
    2. Prepare disks for storage
  5. Install and configure ScaleIO components using the Installation Manager via the web client:
    1. In a Web browser, enter the following URL https:// <IM_server_IP>/ where <IM_Server_IP> is the IP address of the server where you installed the Installation Manager package
    2. Log in with the default user name (admin) and with the password defined when the Installation Manager was prepared, then click Login
    3. Click Get Started
    4. Browse to where the ScaleIO packages are located, select files, click Open, and click Upload. Click Proceed to Install.
    5. In the Provide Installation Topology screen, click Installation wizard. The Installation Configuration window appears:

    6.  In the Credentials Configuration section, enter and confirm a new MDM Password and the LIA password. In the Topology section, enter server information. Click Start Installation.
    7. Follow the post installation instructions displayed in the wizard.

For more information, refer to the EMC ScaleIO installation guide.

Dell EMC SC Series

The Storage All-in-One ETL relies on the Dell EMC SC Series REST API to collect capacity information about the Dell EMC SC Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Dell EMC SC Series REST API is 3033.

Migrating from Dell Compellent ETL to Storage All-In-- One ETL

When migrating from Dell Compellent Enterprise Manager ETL to Storage All-in-One ETL to monitor Dell Compellent systems, you may need to reconcile duplicate entities in order to guarantee the continuity of your historical data. Refer to the Lookup reconciliation and splitting on the BMC documentation website.

Dell EMC Unity (REST)

The Storage All-in-One ETL relies on the Dell EMC Unity REST API to collect capacity information about the Dell EMC Unity systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine.

Dell EMC VMAX (WBEM)

The Storage All-in-One ETL relies on the EMC SMI-S provider to collect capacity information about the Dell EMC VMAX systems. The Storage All-in-One ETL supports EMC VMAX3s models only.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

  • TrueSight Capacity Optimization version must be supported by the ETL
  • The Operating System of the ETL Engine running the ETL task must be a 64-bit version of Linux or Solaris
  • It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.


EMC SMI-S Provider

Before configuring the Storage All-in-One ETL to collect EMC VMAX metrics, you must ensure that the following requirements are met:

    •  (Recommended) Use the SMI-S provider pre-installed in the Management Console. No further action is required.
    •  (Alternate option) Install EMC SMI-S Provider on a different system. In this case, version 8.0.3 or higher is required.  If multiple VMAX systems are added to the same provider, the total number of volumes should not exceed 30,000 (hard limit of the provider is 50,000).

To manage EMC VMAX systems, the EMC SMI-S Provider requires at least one LUN to be mounted from each array to manage. These special LUNs are called "gatekeepers" and are used for the communication between the Symmetrix array and the SMI-S Provider (which does not have an IP address). EMC recommends having between 4 and 6 gatekeeper LUNs for each managed array. Increasing the number of gatekeepers dramatically improves the performance of the EMC SMI-S Provider.

Dell EMC VMAX/PowerMax (REST)

The Storage All-in-One ETL relies on the Unisphere REST API to collect capacity information about the Dell EMC VMAX/PowerMax systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Unisphere REST API is 8443.

Prerequisites

Dell EMC Unisphere REST API  v.9.1+

Dell EMC XtremIO

The Storage All-in-One ETL relies on the EMC XtremIO REST API to collect capacity information about the EMC XtremIO systems. The connection to the REST API is performed via the HTTPS protocol.

The ETL identifies itself to the EMC XtremIO REST API v1 with user name and password or with an API token for EMC XtremIO REST API v2.

Fujitsu Eternus AF/DX (REST)

The Storage All-in-One ETL relies on the ETERNUS AF/DX RESTful API to collect capacity information about Eternus AF/DX Systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the ETERNUS AF/DX RESTful API with user name and password.

Due to REST API limitations, I/O statistics for byte rates or operation rates are not available.

Prerequisites

  • ETERNUS AF/DX RESTful API

Supported Platforms

The ETL supports the following Fujitsu Eternus models:

  • AF S3 series All-Flash Arrays
  • DX S5 series
  • DX8100 S4 Hybrid
  • DX8900 S4 Hybrid

Hitachi E, G Series (REST)

The Storage All-in-One ETL relies on the Hitachi Configuration Manager REST API to collect capacity information about the Hitachi E and G Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Hitachi Configuration Manager REST API is 23451.

HPE Primera (WBEM)

The Storage All-in-One ETL relies HPE Primera web services API (WSAPI) to collect capacity information about the HPE Primera systems. The connection is performed via the HTTPS protocol through port 5989. The ETL identifies itself with a user name and password. The ETL supports HPE Primera and HP 3PAR StoreServ models.

This storage intelligence adapter is not compatible with HPE Primera software versions 4.5.3 and 4.5.4 due to the HPE Primera issue 370176, C28803.

Hitachi GUM (REST)

The Storage All-in-One ETL relies on the Hitachi GUM REST API to collect capacity information about the Hitachi GUM systems. The connection to the GUM REST API is performed via the HTTPS protocol. The ETL identifies itself to the GUM REST API with a username and password.

The default https port for the GUM REST API is 443.

Prerequisites

The Connection settings must provide the hostname or IP address of the controller as the Storage All-in-One ETL queries directly the controller(s).

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL  page displays the configuration properties.
  3. Expand the Run configuration section. In te Hostname field, enter the hostname or IP address of a controller.

Known Limitation

Due to known limitations, the host-related information (ST_HOSTCNT, ST_SYSTEM_SUBSCRIBED_CAPACITY, ST_POOL_SUBSCRIBED_CAPACITY) is not available via GUM if your Hitachi storage system is running with Service Processor (SVP), We recommend using the Hitachi E Series, G Series (REST) adapter to collect host-related data.

Huawei OceanStor

The Storage All-in-One ETL relies on the Huawei OceanStor API to collect capacity information about the Huawei OceanStor systems. The connection to the REST API is performed via the HTTPS protocol through port 8088 . The ETL identifies itself to the REST API with a user name and password.

IBM DS8000 (REST)

The Storage All-in-One ETL relies on the DS8000 RESTful API v1.0 and higher to discovers storage systems and collects performance/capacity metrics for IBM DS8000 series platforms. It collects metrics for the Storage System, Nodes, Ports, Pools and Volumes.

Prerequisites

  • DS8000 RESTful API v1.0 and higher

Supported Platforms

  • IBM DS8000 Series


Important information about entities reconciliation

Additional information for specifically Reconcile IBM DS888x (WBEM) and IBM DS8000 (REST) entities

By following these steps meticulously, you can seamlessly transition to utilizing IBM DS8000 (REST) while ensuring the reconciliation of Storage Pool and Storage Volume entities (Storage System entities are automatically reconciled), thereby preventing duplication and data collection discrepancies.

  1. Prepare Catalog:
    Begin by creating a catalog specifically for the IBM reconciliation process. For detailed instructions, refer to the documentation provided by BMC.

  2. Create WBEM Task:
    Generate a WBEM task and navigate to the Shared Entity Catalog section. Here, designate the catalog you previously created for IBM reconciliation.

  3. Create REST Task:
    Set up a REST task, ensuring to select the designated catalog in the Shared Entity Catalog section. Note: Use the same workspace as for the WBEM task.

  4. Stop WBEM Task:
    The WBEM task should be stopped to avoid conflicts during the reconciliation process.

  5. Review Catalog:
    Access the created catalog to proceed with reconciliation tasks. Important: In the Compare entities by option, select Entity name and type. Detailed guidelines can be found in the BMC documentation.

  6. Initiate Reconciliation:
    Execute the reconciliation process as outlined in the BMC documentation.


Known limitation

Due to a limitation in the REST API:

  1. disks are not discovered.
  2. some performance metrics are not available:

    • Storage Systems: no byte rate

    • Ports: bandwidth usage

    • Storage Pools: I/O stats

    • Volumes: I/O stats

IBM DS888x (WBEM)

The Storage All-in-One ETL relies on the IBM System Storage DS Open API, natively embedded within the Hardware Management Console (HMC) up to v8x, through the use of SMI-S provider to collect capacity information about the IBM DS888x systems.

The default port for HTTPS communication with the IBM DS888x SMI-S Provider is 6989.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

  • TrueSight Capacity Optimization version must be supported by the ETL
  • Valid credentials to the HMC must be provided
  • It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.

IBM Spectrum Scale

The Storage All-in-One ETL relies on the IBM Spectrum Scale Management API to collect capacity information about the IBM Spectrum Scale systems. The connection to the REST API is performed via the HTTPS protocol through port 443. The ETL identifies itself to the REST API with a user name and password.

NetApp Active IQ Unified Manager (AIQUM) REST API

To collect NetApp AIQUM metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the Active IQ Unified Manager (AIQUM) (REST API) endpoint. This storage intelligence adapter discovers all NetApp storage clusters configured on the Active IQ Unified Manager. It collects performance and capacity metrics for Clusters, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the AIQUM REST API is 443.

NetApp E-Series REST API

To collect NetApp E-Series metrics, the Storage All-in-One ETL relies on NetApp SANtricity REST API.

The default HTTP port for the NetApp SANtricity REST API is 8443.

Prerequisites

The NetApp SANtricity Web Services need to be installed and configured.

Accessing the Embedded NetApp SANtricity Web Services

The following information is required to access the NetApp SANtricity Web Services:

  • Credential Type (RESTful Web API with basic authentication)
  • SANtricity System Manager IP address.
  • Username and password that are used to access to the SANtricity System Manager


The configuration of the NetApp SANtricity Web Services Proxy differs according to the operating system running on the host:

Configuring the NetApp SANtricity Web Services Proxy on a Windows Host
    • Configure the Windows proxy to connect to the Windows host.
    • Configure credentials for the Windows Host.
    • Configure the RESTful Web API with basic authentication credentials:
Configuring the NetApp SANtricity Web Services Proxy on a Linux Host
    • Configure SSH Credentials for the Linux Host
    • Configure the RESTful Web API with basic authentication credentials:

NetApp ONTAP REST API

To collect NetApp ONTAP metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the ONTAP REST API endpoint. This storage intelligence adapter discovers the NetApp ONTAP storage system. It collects performance and capacity metrics for the Cluster, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the ONTAP REST API is 443.

NetApp SolidFire

To collect NetApp SolidFire metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the NetApp SolidFire Element OS API endpoint. HTTP basic authentication is sufficient to access the cluster and discover the system and all the physical nodes connected to the cluster. 

For more information, refer to NetApp SolidFire Element OS API Reference Guide.

Oracle ZFS REST

Introduction

The Storage All-in-One ETL relies on the Oracle ZFS Storage Appliance RESTful API v2.0 or higher to discover and collect capacity information about Oracle ZFS Storage systems. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Prerequisites

Oracle ZFS Storage Appliance RESTful API v2.0 or higher.

Supported Platforms

Oracle ZFS Storage Appliance ZS9-2 (ZFS Storage Appliance | Oracle).

Pure Storage FlashArray REST

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API to collect capacity information about the Pure Storage systems. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password or a token provided by the Pure Storage REST API.

Prerequisites

The Pure Storage flash array must support Pure Storage REST API v2.4 or higher released with Purity 6.1.0.

Pure Storage FlashBlade REST API

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API v2.0, or higher, to collect capacity information about the Pure Storage FlashBlade S and E series All-Flash Arrays. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password or a token provided by the Pure Storage REST API. The ETL collects performance and capacity metrics for the Storage Arrays, Fabric Modules, Connectors, Drives, Buckets, File Systems and Object Stores.

Prerequisites

The Pure Storage flash array must support Pure Storage REST API v2.0 or higher.

Supported Platforms

  • Pure Storage FlashBlade S
  • Pure Storage FlashBlade E

Scality RING (Scality Supervisor API)

Introduction

The Storage All-in-One ETL relies on the Scality Supervisor API (SupAPI) to collect capacity information about the Scality RING systems through the supervisor node. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Limitation

The Scality Supervisor API does not report I/O stats.

Deploying the ETL module

Complete the following steps to deploy the ETL Module:

  1. Navigate to Administration > SYSTEM > Maintenance.
  2. Verify the installation status of the already uploaded packages. If their status is "READY" or "RUNNING", wait for the installation to complete before proceeding.
  3. At the bottom of the Maintenance page, locate the Upload patch or additional package file section.
  4. Click Browse.
  5. Locate and select the ETL installation package.
  6. Click Upload.
  7. Read the Overview and click Next to continue.
  8. Read and accept the End User License Agreement.
  9. Select the server on which the operation must be executed; that is the server on which the package will be saved. Click Proceed to continue.

  10. A successful installation will result in a green line in the Additional Packages table.

    Note

     In case of unsuccessful outcome (e.g. a yellow WARNING line), click the  icon to inspect the deployment log.

Once the Storage All-in-One ETL is installed, the Storage Engine and Storage Intelligence rows appear in the Administration > System > Maintenance > Additional Packages tab:


Step II. Configure the ETL

You must configure the ETL to connect to the required storage system for data collection. ETL configuration includes specifying the basic and optional advanced properties. While configuring the basic properties is sufficient, you can optionally configure the advanced properties for additional customization.

A. Configuring the basic properties

Some of the basic properties display default values. You can modify these values if required.

To configure the basic properties:

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL  page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, and ETL additional settings.
  3. On the Run configuration tab, select Storage - All-in-One Extractor Service from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You can edit this field to customize the name.
  4. Click the Entity catalog tab, and select one of the following options:

    • Shared Entity Catalog:

      Select if ETLs access the same entities.
      • From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
    • Private Entity Catalog: Select if this is the only ETL that extracts data.

  5. Click the Connection Settings tab, and configure the following properties:

    PropertyDescription
    Storage Solution

    Storage solution to be configured. Select:

      • Ceph Storage REST to collect capacity information about your Ceph Storage systems
      • Dell EMC Isilon to collect capacity information about your Dell EMC Isilon systems
      • Dell EMC ScaleIO REST/Dell EMC PowerFlex REST to collect capacity information about your Dell EMC ScaleIO and Dell EMC PowerFlex systems
      • Dell EMC SC Series to collect capacity information about your Dell EMC SC Series systems
      • Dell EMC Unity REST to collect capacity information about your Dell EMC Unity systems
      • Dell EMC PowerMax v4 REST to collect capacity information about your Dell EMC PowerMax v4 systems
      • Dell EMC PowerStore REST to collect capacity information about your Dell EMC PowerStore systems
      • Dell EMC PowerVault REST to collect capacity information about your Dell EMC PowerVault systems
      • Dell EMC VMAX WBEM to collect capacity information about your Dell EMC VMAX systems
      • Dell EMC VMAX/PowerMax REST to collect capacity information about your Dell EMC VMAX/PowerMax systems
      • Dell EMC XtremIO REST to collect capacity information about your Dell EMC XtremIO systems

      • HPE Primera WBEM to collect capacity information about your HPE Primera systems

      • Huawei OceanStor REST to collect capacity information about your Huawei OceanStor systems

      • Hitachi G Series REST to collect capacity information about your Hitachi G Series systems

      • Hitachi GUM REST to collect capacity information about your Hitachi GUM systems

      • IBM DS8880 WBEM to collect capacity information about your IBM DS888x systems

      • IBM Spectrum Scale REST to collect capacity information about your IBM Spectrum Scale systems
      • NetApp Active IQ Unified Manager REST to collect capacity information about your NetApp Active IQ Unified Manager systems
      • NetApp E-Series REST to collect capacity information about your NetApp E-Series systems
      • NetApp ONTAP REST to collect capacity information about your NetApp ONTAP systems
      • NetApp SolidFire REST to collect capacity information about your NetApp SolidFire systems
      • Oracle ZFS REST to collect capacity information about your Oracle ZFS Storage systems
      • Pure Storage FlashBlade REST to collect capacity information about your Pure Storage FlashBlade systems
      • Pure Storage REST to collect capacity information about your Pure Storage systems
      • Scality RING (Scality Supervisor API) to collect capacity information about your Scality RING systems
      • Other Storage Solution to collect capacity information about any other storage solution supported by the Storage-All-in-One ETL. In this case, you will have to manually enter the name of the Si (Storage Intelligence) file to be used.
    Other Storage SolutionName of the Storage Intelligence file (without extension) to be used. This field should only be specified if the Storage Solution property is set to Other Storage Solution.
    Hostname

    Hostname or IP address of the storage solution.

    Port

    Port number used to connect to the storage solution.

    Connect As

    Password to connect to the storage solution.

    Transport Protocol

    Transport protocol used to connect to the storage solution (HTTP or HTTPS).

  6. Click Save. You are returned to the Last run tab under the ETL tasks page.

    1. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
      • OK: the ETL executed without any error in simulation mode.
      • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
      • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
    2. Switch the ETL to production mode: Perform the following steps:
      1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
      2. In the Run configuration table in the ETL details page, click  to edit the active Run configuration.
      3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
      4. Click Save. 

  7. Locate the ETL in the ETL tasks table and click  to run it, or schedule an ETL run. 
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to TrueSight Capacity Optimization database. You can see the entity hierarchy in the left Navigation pane, under Workspace.

Step III. Run the ETL

After you configure the ETL, you can run it to collect data. You can run the ETL in the following modes:

A. Simulation mode: Only validates connection to the data source, does not collect data. Use this mode when you want to run the ETL for the first time or after you make any changes to the ETL configuration.

B. Production mode: Collects data from the data source.

A. Running the ETL in the simulation mode

To run the ETL in the simulation mode:

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
  2. On the ETL tasks page, click the ETL. The ETL details are displayed.


  1. In the Run configurations table, click Edit to modify the ETL configuration settings.
  2. On the Run configuration tab, ensure that the Execute in simulation mode option is set to Yes, and click Save.
  3. Click Run active configuration. A confirmation message about the ETL run job submission is displayed.
  4. On the ETL tasks page, check the ETL run status in the Last exit column. 
    OK Indicates that the ETL ran without any error. You are ready to run the ETL in the production mode.
  5.  If the ETL run status is Warning, Error, or Failed:
    1. On the ETL tasks page, click  in the last column of the ETL name row.
    2. Check the log and reconfigure the ETL if required.
    3. Run the ETL again.
    4. Repeat these steps until the ETL run status changes to OK.

B. Running the ETL in the production mode

You can run the ETL manually when required or schedule it to run at a specified time.

a. Running the ETL manually

  1. On the ETL tasks page, click the ETL. The ETL details are displayed.
  2. In the Run configurations table, click Edit  to modify the ETL configuration settings. The Edit run configuration page is displayed.
  3. On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
  4. To run the ETL immediately, click Run active configuration. A confirmation message about the ETL run job submission is displayed.
    When the ETL is run, it collects data from the source and transfers it to the TrueSight Capacity Optimization database.

b. Scheduling the ETL run

By default, the ETL is scheduled to run daily. You can customize this schedule by changing the frequency and period of running the ETL.

To configure the ETL run schedule:

  1. On the ETL tasks page, click the ETL, and click Edit. The ETL details are displayed.

  2. On the Edit task page, do the following, and click Save:

    • Specify a unique name and description for the ETL task.
    • In the Maximum execution time before warning field, specify the duration for which the ETL must run before generating warnings or alerts, if any.
    • Select a predefined or custom frequency for starting the ETL run. The default selection is Predefined.
    • Select the task group and the scheduler to which you want to assign the ETL task.

  3. Click Schedule. A message confirming the scheduling job submission is displayed.
    When the ETL runs as scheduled, it collects data from the source and transfers it to the TrueSight Capacity Optimization database.

Step IV. Post-configuration steps

Reconciling data

If you used the same Shared Entity Catalog while creating your ETL tasks, data will be automatically reconciled since the ETL tasks are using the same lookup details. If you used a Private Entity Catalog or a different entity catalog, you will have to reconcile data. The procedure is as follows:

  1. Create the new entity catalog:
    1. In the Administration tab, click ETL & System Tasks > Entity catalogs
    2. Click Add entity catalog
    3. Name your entity catalog and eventually provide a description
    4. Click Save.
  2. Configure both ETL tasks to use this entity catalog:
    1. In the Administration tab, click ETL & System Tasks > ETL tasks.Click the link of the ETL task for which you wish to set the entity catalog
    2. Click the Stop button to stop the service execution; then click the  button available in the Run configuration section.
    3. In the Entity Catalog section:
      1. Select the Shared Entity Catalog radio button
      2. From the Sharing with Entity Catalog pull-down list, select the entity catalog previously created (in our example: EMC)
    4. Click Save.
    5. Start the ETL.
    6. Resume the procedure for the second ETL task. 
  3. Perform the reconcile operation:
    1. In the Administration tab, click ETL & System Tasks > Entity catalogs
    2. Click the entity catalog previously created (in our example: EMC)
    3. Click the Reconcile entities button
  4. Perform the entity reconciliation: 

    1. Select the Datasource for which you wish to compare the Lookup fields

    2. Compare entities by Lookup fields and select the look fields
    3. Click Search to determine the entities that can be reconciled
    4. Either select the entities to be reconciled and click Reconcile selected rows or click Reconcile all.

  5. Verify in your workspace that all the selected entities are properly reconciled. Only one instance should be created.

Running the Extractor Service task

 

An ETL task can be run manually to retrieve performance metrics. Please note that the steps listed in this section are not mandatory and are only required if the ETL is not already scheduled for execution.

To run the ETL task:

  1. Access the TrueSight Capacity Optimization Console.
  2. In the Administration tab, select ETL & System Tasks > ETL Tasks.
  3. Click  to execute the ETL task.

When the execution is complete, the value of the Status column in the ETL tasks table changes to SERVICE_ON.

Setting the collect frequency

Contrary to the other Storage ETLs for which the collect frequency is set to 15 minutes, the Storage All-in-One ETL follows the cycle below:

  1. The Discovery: The ETL discovers the different entities and metrics. A discovery is performed:
    • each time the ETL Task is started or restarted
    • when the scheduler is restarted
    • when the discovery failed
    • when the number of collects (23) between two discoveries is reached
  2. The Full Collect: A full collect is triggered right after a discovery to collect data for all the entities previously discovered.
  3. The Partial Collect: Data is partially collected. Resource consuming collects (notably for Storage Volumes) are skipped at this stage. By default, partial collects are performed every 15 minutes. If you do not want partial collects to be performed, you can set the sentry.collect.per.volume.collect property as follows:
    1. Edit the ETL Run Configuration:
      1. In the Administration tab, click ETL & System Tasks > ETL tasks.
      2. Click the link of the ETL task for which you wish to set the collect frequency.
      3. Click the Stop button to stop the service execution; then click the Edit button available in the  section.
    2. Click the link You can manually edit ETL properties from this page provided at the bottom of the page to display a list of editable options.
    3. Create the sentry.collect.per.volume.collect property and set its value to 0.
    4. Click Save.
    5. Start the ETL.
 


Was this page helpful? Yes No Submitting... Thank you
Last modified by Manisha Moon on Mar 31, 2024

Comments

  1. Lorenzo Lissoni

    I can't see this page: Unable to render {include} The included page could not be found.

    Jun 03, 2021 10:02
    1. Bharati Poddar

      Hello,

      Thanks for reporting this. Please check now.

      Thanks, Bharati

      Jun 03, 2021 10:30
  2. John Ziegler

    I have the same problem that Lorenzo had. Unable to render {include} The included page could not be found. Thanks, John

    Jul 22, 2022 12:22
    1. Bipin Inamdar

      Hi John,

      It's accessible now. Please check.

      Thanks, Bipin Inamdar

      Jul 25, 2022 12:03
  3. KYNDRYL OMEGA CAPACITY MANAGEMENT ES

    Hi, can't see the page, it says "unable to render {include} The included page could not be found"

    Jul 12, 2023 02:44
    1. Bipin Inamdar

      Please check now. It should be visible.

      Jul 12, 2023 02:55
      1. KYNDRYL OMEGA CAPACITY MANAGEMENT ES

        Hi Bipin, still cannot see.. Same as before

        Jul 17, 2023 05:14
        1. Bipin Inamdar

          The issue is fixed now. Please check and confirm.

          Jul 17, 2023 07:17

Log in or register to comment.

Entities and metrics of Pure Storage REST API ETL for TrueSight Capacity Optimization
Entities and Metrics of Storage All-in-One ETL for TrueSight Capacity Optimization
© Copyright 2014 - 2020 BMC Software, Inc. © Copyright 2014 - 2020 BladeLogic, Inc.
Legal notices

Powered by Atlassian Confluence and Scroll Viewport

© Copyright 2005-2024 BMC Software, Inc. Use of this site signifies your acceptance of BMC’s Terms of Use . BMC, the BMC logo, and other BMC marks are assets of BMC Software, Inc. These trademarks are registered and may be registered in the U.S. and in other countries.