Storage All-in-One ETL for TrueSight Capacity Optimization


 Use the Storage All-in-One ETL for TrueSight Capacity Optimization to poll and collect configuration and performance data from several storage systems all at once. The ETL uses the Storage - All-in-One Extractor Service task to collect all the metrics required for performing critical capacity optimization tasks, such as: analysis, trending, simulation and analytic modeling.

For a comprehensive list of platforms supported by Storage All-in-One ETL, visit:​​​​​ https:www.sentrysoftware.com/solutions/complete-coverage-tsco.html 

Important

We highly recommend verifying the roles or permissions needed to monitor your hardware and storage systems and ensuring that all necessary prerequisites, such as credentials and connectivity, are properly configured.

To collect data by using the Storage All-in-One ETL 1.30.00 for TrueSight Capacity Optimization, perform the following tasks:

I. Complete the preconfiguration tasks.

II. Configure the ETL.

III. Run the ETL.

IV. Post-configuration steps.

Step I. Complete the preconfiguration tasks

The Storage All-in-One ETL comes with the following si adapters:

  • Ceph Storage
  • Dell EMC Avamar (REST)
  • Dell EMC Isilon
  • Dell EMC PowerMax v4 (REST)
  • Dell EMC PowerMax (through Unisphere v10.1 or later via REST)
  • Dell EMC PowerProtect DD (REST)
  • Dell EMC PowerStore (REST)
  • Dell EMC PowerVault (REST)
  • Dell EMC ScaleIO/Dell EMC PowerFlex (REST)
  • Dell EMC SC Series (REST)
  • Dell EMC Unity (REST)
  • Dell EMC VMAX (3rd Generation, All Flash or newer), and PowerMax (WBEM)
  • Dell EMC VMAX/PowerMax (REST)
  • Dell EMC VPLEX (REST)
  • Dell EMC XtremIO
  • Fujitsu Ethernus AF/DX (REST)
  • Hitachi G Series (REST)
  • Hitachi GUM (REST)
  • HPE MSA 1000/2000 Series (REST)
  • HPE Primera (WBEM)
  • Huawei OceanStor (REST)
  • IBM DS8000 (REST)
  • IBM DS888x (WBEM)
  • IBM FlashSystem (REST)
  • IBM Spectrum Control (REST)
  • IBM Spectrum Protect (REST)
  • IBM Spectrum Scale (REST) 
  • NetApp Active IQ Unified Manager (REST)
  • NetApp E-Series
  • NetApp ONTAP (REST)
  • NetApp SolidFire
  • NetApp StorageGRID (REST)
  • Oracle ZFS (REST)
  • Pure Storage FlashArray (REST)
  • Pure Storage FlashBlade (REST)
  • Scality RING (Scality Supervisor API)
Important information

Some platforms can be monitored by both the Storage All-in-One ETL and a manufacturer specific ETL. Please read the following information:

EMC VMAX Systems

For recent systems (3rd Generation and AllFlash) the use of the All-in-One ETL is recommended as it is better optimized for these larger systems.  For older VMAX, VNX and Clariion systems, the EMC ETL should be preferred.

Pure and XtremIO Systems

Code Level 144 will be the last version released for these manufacturer specific ETLs.  It is recommended to migrate to the All-in-One ETL for monitoring Pure and XtremIO systems.

Ceph

Click here to expand...

The Storage All-in-One ETL relies on the Ceph REST API to collect capacity information about the Ceph cluster. The connection to the REST API is performed via the HTTP protocol. The ETL first identifies itself to the REST API with a user name and password to retrieve a bearer token from the API and then uses this token for further REST requests.

The default https port for the CEPH Rest API is 8003 (5000 for http).

Note: The ETL has been tested with the version 14.2.5 of the Ceph REST API.

Prerequisites

The Ceph dashboard must be activated with the following command: ceph mgr module enable dashboard.

Dell EMC Avamar (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC Avamar REST API v10 to collect capacity information about the Dell EMC Avamar systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with an access-token.

Prerequisites

  • Avamar REST API
  • A non-expiring access-token.

 

Important

Some entities are named differently in Dell EMC Avamar and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

Dell EMC Avamar Entity

​​​​​​

Storage All-in-One ETL Entity

Server

Storage System

Data Domain

Storage Filer

File System/MTree

Storage Share

Dell EMC Isilon

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC OneFS REST API to collect capacity information about the Dell EMC Isilon systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Isilon REST API is 8080.

The ETL requires at least version 8 of EMC OneFS.

Dell EMC PowerMax (through Unisphere v10.1 or later via REST)

Click here to expand...

The Storage All-in-One ETL relies on Unisphere v10.1 and higher to collect capacity metrics about Dell EMC PowerMax systems.

Dell EMC PowerMax v4 (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC Unisphere REST API v10 to collect capacity information about the Dell EMC PowerMax v4 systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine
  • Requires Dell EMC Unisphere REST API v10 

Supported Platforms

  • PowerMax 2500
  • PowerMax 8500

Dell EMC PowerProtect DD (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Data Domain RESTfull API to collect capacity information about the Dell EMC PowerProtect DD systems. The connection to the REST API is performed via the HTTPS protocol (Default port: 3009). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Dell EMC Data Domain RESTful API (included in DD OS v7.10+).

Known Limitations

Due to REST API limitations, Fiber Channels and Chassis are not discovered and some performance metrics are not available.

Important

Some entities are named differently in the Dell EMC PowerProtect DD user interface and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

Dell EMC PowerProtect DD Entity

​​​​​​

Storage All-in-One ETL Entity

File System

Storage System

Device (Disk)

Disk

Ethernet Interface

Port

Tier

Storage Pool

Mtree

Storage Share

Dell EMC PowerStore (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC PowerStore Management REST API to collect capacity information about the Dell EMC PowerStore systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine.

Limitations

  • The maximum number of supported volumes is limited to 2000.

Dell EMC PowerVault (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC PowerVault Management REST API to collect capacity information about the Dell EMC PowerVault systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Version 1.10.00 or higher of the engine.

Dell EMC ScaleIO (REST)/Dell EMC PowerFlex (REST)

Click here to expand...

The Storage All-in-One ETL leverages the REST API installed as part of the ScaleIO Gateway to collect capacity metrics about Dell EMC ScaleIO/Dell EMC PowerFlex systems.

The ScaleIO Gateway:

  • can be installed on the same node as other ScaleIO components if the node has adequate memory to run the Installation Manager (minimum of 2 GB) and any other applications.
  • must have connectivity to the data and management ScaleIO networks.

Java should also be installed on the server where you plan to install the ScaleIO Gateway.

To install the ScaleIO Gateway:

  1. Download from EMC Online Support the installation files needed for your operating system.
  2. Extract the installation files on the Installation Manager (IM) server
  3. For Linux:
    1. Run the following command (all in one line):

      GATEWAY_ADMIN_PASSWORD=<new_GW_admin_password> rpm -U /tmp/EMC-ScaleIO-gateway-1.32-XXX.X.noarch.rpm

      where <new_GW_admin_password> is the password to access the IM

  4. For Windows
    1. Run the .msi file, enter (and confirm) a new Gateway Admin password that will be used to access the IM
    2. Prepare disks for storage
  5. Install and configure ScaleIO components using the Installation Manager via the web client:
    1. In a Web browser, enter the following URL https:// <IM_server_IP>/ where <IM_Server_IP> is the IP address of the server where you installed the Installation Manager package
    2. Log in with the default user name (admin) and with the password defined when the Installation Manager was prepared, then click Login
    3. Click Get Started
    4. Browse to where the ScaleIO packages are located, select files, click Open, and click Upload. Click Proceed to Install.
  6. In the Provide Installation Topology screen, click Installation wizard. The Installation Configuration window appears:
    1.  In the Credentials Configuration section, enter and confirm a new MDM Password and the LIA password. In the Topology section, enter server information. Click Start Installation.
      ​​​​
    2. Follow the post installation instructions displayed in the wizard.

For more information, refer to the EMC ScaleIO installation guide.

Dell EMC SC Series

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC SC Series REST API to collect capacity information about the Dell EMC SC Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Dell EMC SC Series REST API is 3033.

Migrating from Dell Compellent ETL to Storage All-In-- One ETL

When migrating from Dell Compellent Enterprise Manager ETL to Storage All-in-One ETL to monitor Dell Compellent systems, you may need to reconcile duplicate entities in order to guarantee the continuity of your historical data. Refer to the Lookup reconciliation and splitting on the BMC documentation website.

Dell EMC VPLEX (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC VPLEX REST API to collect capacity information about all Dell EMC clusters configured on Dell EMC VPLEX systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Dell EMC VPLEX REST API
Important

Some entities are named differently in the Dell EMC VPLEX user interface and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

Dell EMC VPLEX REST Entity
Storage All-in-One ETL Entity

​​​​​

Cluster

Storage System

Director

Controller

Port

Port

Volume

Volume

Dell EMC Unity (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC Unity REST API to collect capacity information about the Dell EMC Unity systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine.

Dell EMC VMAX (WBEM)

Click here to expand...

The Storage All-in-One ETL relies on the EMC SMI-S provider to collect capacity information about the Dell EMC VMAX systems. The Storage All-in-One ETL supports EMC VMAX3s models only.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

  • TrueSight Capacity Optimization version must be supported by the ETL
  • The Operating System of the ETL Engine running the ETL task must be a 64-bit version of Linux or Solaris
  • It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.

EMC SMI-S Provider

Before configuring the Storage All-in-One ETL to collect EMC VMAX metrics, you must ensure that the following requirements are met:


    •  (Recommended) Use the SMI-S provider pre-installed in the Management Console. No further action is required.
    •  (Alternate option) Install EMC SMI-S Provider on a different system. In this case, version 8.0.3 or higher is required.  If multiple VMAX systems are added to the same provider, the total number of volumes should not exceed 30,000 (hard limit of the provider is 50,000).

To manage EMC VMAX systems, the EMC SMI-S Provider requires at least one LUN to be mounted from each array to manage. These special LUNs are called "gatekeepers" and are used for the communication between the Symmetrix array and the SMI-S Provider (which does not have an IP address). EMC recommends having between 4 and 6 gatekeeper LUNs for each managed array. Increasing the number of gatekeepers dramatically improves the performance of the EMC SMI-S Provider.

Dell EMC VMAX/PowerMax (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Unisphere REST API to collect capacity information about the Dell EMC VMAX/PowerMax systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Unisphere REST API is 8443.

Prerequisites

Dell EMC Unisphere REST API  v.9.1+

Dell EMC XtremIO

Click here to expand...

The Storage All-in-One ETL relies on the EMC XtremIO REST API to collect capacity information about the EMC XtremIO systems. The connection to the REST API is performed via the HTTPS protocol.

The ETL identifies itself to the EMC XtremIO REST API v1 with user name and password or with an API token for EMC XtremIO REST API v2.

Fujitsu Eternus AF/DX (REST)

Click here to expand...

The Storage All-in-One ETL relies on the ETERNUS AF/DX RESTful API to collect capacity information about Eternus AF/DX Systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the ETERNUS AF/DX RESTful API with user name and password.

Due to REST API limitations, I/O statistics for byte rates or operation rates are not available.

Prerequisites

  • ETERNUS AF/DX RESTful API

Supported Platforms

The ETL supports the following Fujitsu Eternus models:

  • AF S3 series All-Flash Arrays
  • DX S5 series
  • DX8100 S4 Hybrid
  • DX8900 S4 Hybrid

Hitachi E, G Series (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Hitachi Configuration Manager REST API to collect capacity information about the Hitachi E and G Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Hitachi Configuration Manager REST API is 23451.

HPE MSA 1000/2000 Series (REST)

Click here to expand...

The Storage All-in-One ETL relies on the HPE MSA REST API to collect capacity information about the HPE MSA 1000/2000 Series systems over port 443 and authenticates with a username and password (minimum required role: Monitor).

HPE Primera (WBEM)

Click here to expand...

The Storage All-in-One ETL relies HPE Primera web services API (WSAPI) to collect capacity information about the HPE Primera systems. The connection is performed via the HTTPS protocol through port 5989. The ETL identifies itself with a user name and password. The ETL supports HPE Primera and HP 3PAR StoreServ models.

This storage intelligence adapter is not compatible with HPE Primera software versions 4.5.3 and 4.5.4 due to the HPE Primera issue 370176, C28803.

Hitachi GUM (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Hitachi GUM REST API to collect capacity information about the Hitachi GUM systems. The connection to the GUM REST API is performed via the HTTPS protocol. The ETL identifies itself to the GUM REST API with a username and password.

The default https port for the GUM REST API is 443.

Prerequisites

The Connection settings must provide the hostname or IP address of the controller as the Storage All-in-One ETL queries directly the controller(s).

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL  page displays the configuration properties.
  3. Expand the Run configuration section. In te Hostname field, enter the hostname or IP address of a controller.

Known Limitation

Due to known limitations, the host-related information (ST_HOSTCNT, ST_SYSTEM_SUBSCRIBED_CAPACITY, ST_POOL_SUBSCRIBED_CAPACITY) is not available via GUM if your Hitachi storage system is running with Service Processor (SVP), We recommend using the Hitachi E Series, G Series (REST) adapter to collect host-related data.

Huawei OceanStor

Click here to expand...

The Storage All-in-One ETL relies on the Huawei OceanStor API to collect capacity information about the Huawei OceanStor systems. The connection to the REST API is performed via the HTTPS protocol through port 8088 . The ETL identifies itself to the REST API with a user name and password.

IBM DS8000 (REST)

Click here to expand...

The Storage All-in-One ETL relies on the DS8000 RESTful API v1.0 and higher to discover storage systems and collect performance/capacity metrics for IBM DS8000 series platforms. It collects metrics for the Storage System, Nodes, Ports, Pools and Volumes.

Prerequisites

  • DS8000 RESTful API v1.0 and higher

Supported Platforms

  • IBM DS8000 Series

Important information about entities reconciliation

Additional information for specifically Reconcile IBM DS888x (WBEM) and IBM DS8000 (REST) entities

By following these steps meticulously, you can seamlessly transition to utilizing IBM DS8000 (REST) while ensuring the reconciliation of Storage Pool and Storage Volume entities (Storage System entities are automatically reconciled), thereby preventing duplication and data collection discrepancies.

  1. Prepare Catalog:
    Begin by creating a catalog specifically for the IBM reconciliation process. For detailed instructions, refer to the documentation provided by BMC.

  2. Create WBEM Task:
    Generate a WBEM task and navigate to the Shared Entity Catalog section. Here, designate the catalog you previously created for IBM reconciliation.

  3. Create REST Task:
    Set up a REST task, ensuring to select the designated catalog in the Shared Entity Catalog section. Note: Use the same workspace as for the WBEM task.

  4. Stop WBEM Task:
    The WBEM task should be stopped to avoid conflicts during the reconciliation process.

  5. Review Catalog:
    Access the created catalog to proceed with reconciliation tasks. Important: In the Compare entities by option, select Entity name and type. Detailed guidelines can be found in the BMC documentation.

  6. Initiate Reconciliation:
    Execute the reconciliation process as outlined in the BMC documentation.


Known limitation

Due to a limitation in the REST API:

  1. disks are not discovered.
  2. some performance metrics are not available:
    • Storage Systems: no byte rate
    • Ports: bandwidth usage
    • Storage Pools: I/O stats
    • Volumes: I/O stats

IBM DS888x (WBEM)

Click here to expand...

The Storage All-in-One ETL relies on the IBM System Storage DS Open API, natively embedded within the Hardware Management Console (HMC) up to v8x, through the use of SMI-S provider to collect capacity information about the IBM DS888x systems.

The default port for HTTPS communication with the IBM DS888x SMI-S Provider is 6989.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

  • TrueSight Capacity Optimization version must be supported by the ETL
  • Valid credentials to the HMC must be provided
  • It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.

IBM Spectrum Control (REST)

Click here to expand...

The Storage All-in-One ETL relies on the IBM Spectrum Control API to collect capacity information about storage systems connected to IBM Spectrum Control. The connection to the REST API is performed via the HTTPS protocol through port 9569. The ETL identifies itself to the REST API with a user name and password.

IBM Spectrum Protect (REST)

Click here to expand...

The Storage All-in-One ETL relies on the IBM Spectrum Protect API to collect capacity information about storage systems connected to IBM Spectrum Protect Operations Center. The connection to the REST API is performed via the HTTPS protocol through port 11090. The ETL identifies itself to the REST API with a user name and password.

IBM Spectrum Scale (REST)

Click here to expand...

The Storage All-in-One ETL relies on the IBM Spectrum Scale Management API to collect capacity information about the IBM Spectrum Scale systems. The connection to the REST API is performed via the HTTPS protocol through port 443. The ETL identifies itself to the REST API with a user name and password.

NetApp Active IQ Unified Manager (AIQUM) REST API

Click here to expand...

To collect NetApp AIQUM metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the Active IQ Unified Manager (AIQUM) (REST API) endpoint. This storage intelligence adapter discovers all NetApp storage clusters configured on the Active IQ Unified Manager. It collects performance and capacity metrics for Clusters, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the AIQUM REST API is 443.

NetApp E-Series REST API

Click here to expand...

To collect NetApp E-Series metrics, the Storage All-in-One ETL relies on NetApp SANtricity REST API.

The default HTTP port for the NetApp SANtricity REST API is 8443.

Prerequisites

The NetApp SANtricity Web Services need to be installed and configured.

Accessing the Embedded NetApp SANtricity Web Services

The following information is required to access the NetApp SANtricity Web Services:

  • Credential Type (RESTful Web API with basic authentication)
  • SANtricity System Manager IP address.
  • Username and password that are used to access to the SANtricity System Manager

The configuration of the NetApp SANtricity Web Services Proxy differs according to the operating system running on the host:

Configuring the NetApp SANtricity Web Services Proxy on a Windows Host

    • Configure the Windows proxy to connect to the Windows host.
    • Configure credentials for the Windows Host.
    • Configure the RESTful Web API with basic authentication credentials:
Configuring the NetApp SANtricity Web Services Proxy on a Linux Host

    • Configure SSH Credentials for the Linux Host
    • Configure the RESTful Web API with basic authentication credentials:

NetApp ONTAP REST API

Click here to expand...

To collect NetApp ONTAP metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the ONTAP REST API endpoint. This storage intelligence adapter discovers the NetApp ONTAP storage system. It collects performance and capacity metrics for the Cluster, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the ONTAP REST API is 443.

NetApp SolidFire

Click here to expand...

To collect NetApp SolidFire metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the NetApp SolidFire Element OS API endpoint. HTTP basic authentication is sufficient to access the cluster and discover the system and all the physical nodes connected to the cluster. 

For more information, refer to NetApp SolidFire Element OS API Reference Guide.

NetApp StorageGRID REST API

Click here to expand...

To collect NetApp StorageGRID metrics, the Storage All-in-One ETL relies on the NetApp StorageGRID Management REST API.

The default HTTP port for the the NetApp StorageGRID Management REST API is 443.

Prerequisites

StorageGRID Management REST API v3.5 (including the Private API).

For detailed information about refer to the NetApp StorageGRID documentation.

Important

The Storage All-in-One ETL for TrueSight Capacity Optimization displays metric values in gibibyte (GiB) and megabyte per second (MB/s) while the NetApp StorageGRID user interface displays them in gigabyte (GB) and megabit per second (Mb/s).

Also, some entities are named differently in the NetApp StorageGRID user interface and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

NetApp StorageGRID UI Entity
Storage All-in-One ETL Entity

​​​​​​

Site
Storage System
Node
Controller
Port
Port
Object Store
Storage Pool
Disk Device
Storage Volume
Volume
Storage Share

Oracle ZFS REST

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Oracle ZFS Storage Appliance RESTful API v2.0 or higher to discover and collect capacity information about Oracle ZFS Storage systems. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Prerequisites

Oracle ZFS Storage Appliance RESTful API v2.0 or higher.

Supported Platforms

Oracle ZFS Storage Appliance ZS9-2 (ZFS Storage Appliance | Oracle).

Pure Storage FlashArray REST

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API to collect capacity information about the Pure Storage systems. Depending on the API version available, you will use either Pure Storage FlashArray (REST v1.x) or Pure Storage FlashArray (REST v2.4+). To determine the version supported by your system, refer to the article Storage All-in-One ETL: Verifying Prerequisites (Credentials and Connectivity).

Prerequisites

  • When using Pure Storage FlashArray (REST v1.x):
    • Pure Storage REST API v1.6 or higher
    • Username and password for authentication
  • When using Pure Storage FlashArray (REST v2.4+):
    • Pure Storage REST API v2.4 and higher
    • API token for authentication.

Migration from standalone ETL

If migrating from the standalone Pure Storage REST API ETL, note that the token-based authentication method is no longer supported by Pure or the Storage All-in-One ETL. You must now use a username and password for authentication. 

Pure Storage FlashBlade REST API

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API v2.0, or higher, to collect capacity information about the Pure Storage FlashBlade S and E series All-Flash Arrays. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies with an API token you would have previously generated (Refer to Generating an API Token for more information). The ETL collects performance and capacity metrics for the Storage Arrays, Fabric Modules, Connectors, Drives, Buckets, File Systems and Object Stores.

Prerequisites

The Pure Storage flash array must support Pure Storage REST API v2.0 or higher.

Supported Platforms

  • Pure Storage FlashBlade S
  • Pure Storage FlashBlade E

Scality RING (Scality Supervisor API)

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Scality Supervisor API (SupAPI) to collect capacity information about the Scality RING systems through the supervisor node. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Limitation

The Scality Supervisor API does not report I/O stats.

Deploying the ETL module

Complete the following steps to deploy the ETL Module:

  1. Navigate to Administration > SYSTEM > Maintenance.
  2. Verify the installation status of the already uploaded packages. If their status is "READY" or "RUNNING", wait for the installation to complete before proceeding.
  3. At the bottom of the Maintenance page, locate the Upload patch or additional package file section.
  4. Click Browse.
  5. Locate and select the ETL installation package.
  6. Click Upload.
  7. Read the Overview and click Next to continue.
  8. Read and accept the End User License Agreement.
  9. Select the server on which the operation must be executed; that is the server on which the package will be saved. Click Proceed to continue.
  10. A successful installation will result in a green line in the Additional Packages table.

    Note

     In case of unsuccessful outcome (e.g. a yellow WARNING line), click the View_Deployment_Log.png icon to inspect the deployment log.

Once the Storage All-in-One ETL is installed, the Storage Engine and Storage Intelligence rows appear in the Administration > System > Maintenance > Additional Packages tab.


Use the Storage All-in-One ETL for TrueSight Capacity Optimization to poll and collect configuration and performance data from several storage systems all at once. The ETL uses the Storage - All-in-One Extractor Service task to collect all the metrics required for performing critical capacity optimization tasks, such as: analysis, trending, simulation and analytic modeling.

Important

We highly recommend verifying the roles or permissions needed to monitor your hardware and storage systems and ensuring that all necessary prerequisites, such as credentials and connectivity, are properly configured.

Upgrading to Storage All-in-One ETL v1.28.00 requires data reconciliation based on name and parent. This process ensures data continuity for most entities; however, historical data for filesystems cannot be retained and will be lost.

To collect data by using the Storage All-in-One ETL 1.28.00 for TrueSight Capacity Optimization, perform the following tasks:

I. Complete the preconfiguration tasks.

II. Configure the ETL.

III. Run the ETL.

IV. Post-configuration steps.

Step I. Complete the preconfiguration tasks

The Storage All-in-One ETL supports the following platforms:

  • Ceph Storage
  • Dell EMC Avamar (REST)
  • Dell EMC Isilon
  • Dell EMC PowerMax v4 (REST)
  • Dell EMC PowerProtect DD (REST)
  • Dell EMC PowerStore (REST)
  • Dell EMC PowerVault (REST)
  • Dell EMC ScaleIO/Dell EMC PowerFlex (REST)
  • Dell EMC SC Series (REST)
  • Dell EMC Unity (REST)
  • Dell EMC VMAX (3rd Generation, All Flash or newer), and PowerMax (WBEM)
  • Dell EMC VMAX/PowerMax (REST)
  • Dell EMC VPLEX (REST)
  • Dell EMC XtremIO
  • Fujitsu Ethernus AF/DX (REST)
  • Hitachi G Series (REST)
  • Hitachi GUM (REST)
  • HPE Primera (WBEM)
  • Huawei OceanStor (REST)
  • IBM DS8000 (REST)
  • IBM DS888x (WBEM)
  • IBM FlashSystem (REST)
  • IBM Spectrum Control (REST)
  • IBM Spectrum Protect (REST)
  • IBM Spectrum Scale (REST) 
  • NetApp Active IQ Unified Manager (REST)
  • NetApp E-Series
  • NetApp ONTAP (REST)
  • NetApp SolidFire
  • NetApp StorageGRID (REST)
  • Oracle ZFS (REST)
  • Pure Storage FlashArray (REST)
  • Pure Storage FlashBlade (REST)
  • Scality RING (Scality Supervisor API)
Important information

Some platforms can be monitored by both the Storage All-in-One ETL and a manufacturer specific ETL. Please read the following information:

EMC VMAX Systems

For recent systems (3rd Generation and AllFlash) the use of the All-in-One ETL is recommended as it is better optimized for these larger systems.  For older VMAX, VNX and Clariion systems, the EMC ETL should be preferred.

Pure and XtremIO Systems

Code Level 144 will be the last version released for these manufacturer specific ETLs.  It is recommended to migrate to the All-in-One ETL for monitoring Pure and XtremIO systems.

Ceph

Click here to expand...

The Storage All-in-One ETL relies on the Ceph REST API to collect capacity information about the Ceph cluster. The connection to the REST API is performed via the HTTP protocol. The ETL first identifies itself to the REST API with a user name and password to retrieve a bearer token from the API and then uses this token for further REST requests.

The default https port for the CEPH Rest API is 8003 (5000 for http).

Note: The ETL has been tested with the version 14.2.5 of the Ceph REST API.

Prerequisites

The Ceph dashboard must be activated with the following command: ceph mgr module enable dashboard.

Dell EMC Avamar (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC Avamar REST API v10 to collect capacity information about the Dell EMC Avamar systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with an access-token.

Prerequisites

  • Avamar REST API
  • A non-expiring access-token.

 

Important

Some entities are named differently in Dell EMC Avamar and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

Dell EMC Avamar Entity
    • ======
Storage All-in-One ETL Entity
    • ======

Server

Storage System

Data Domain

Storage Filer

File System/MTree

Storage Share

Dell EMC Isilon

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC OneFS REST API to collect capacity information about the Dell EMC Isilon systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Isilon REST API is 8080.

The ETL requires at least version 8 of EMC OneFS.

Dell EMC PowerMax v4 (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC Unisphere REST API v10 to collect capacity information about the Dell EMC PowerMax v4 systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine
  • Requires Dell EMC Unisphere REST API v10 

Supported Platforms

  • PowerMax 2500
  • PowerMax 8500

Dell EMC PowerProtect DD (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Data Domain RESTfull API to collect capacity information about the Dell EMC PowerProtect DD systems. The connection to the REST API is performed via the HTTPS protocol (Default port: 3009). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Dell EMC Data Domain RESTful API (included in DD OS v7.10+).

Known Limitations

Due to REST API limitations, Fiber Channels and Chassis are not discovered and some performance metrics are not available.

Important

Some entities are named differently in the Dell EMC PowerProtect DD user interface and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

Dell EMC PowerProtect DD Entity
    • ======
Storage All-in-One ETL Entity
    • ======

File System

Storage System

Device (Disk)

Disk

Ethernet Interface

Port

Tier

Storage Pool

Mtree

Storage Share

Dell EMC PowerStore (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC PowerStore Management REST API to collect capacity information about the Dell EMC PowerStore systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine.

Limitations

  • The maximum number of supported volumes is limited to 2000.

Dell EMC PowerVault (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC PowerVault Management REST API to collect capacity information about the Dell EMC PowerVault systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Version 1.10.00 or higher of the engine.

Dell EMC ScaleIO (REST)/Dell EMC PowerFlex (REST)

Click here to expand...

The Storage All-in-One ETL leverages the REST API installed as part of the ScaleIO Gateway to collect capacity metrics about Dell EMC ScaleIO/Dell EMC PowerFlex systems.

The ScaleIO Gateway:

  • can be installed on the same node as other ScaleIO components if the node has adequate memory to run the Installation Manager (minimum of 2 GB) and any other applications.
  • must have connectivity to the data and management ScaleIO networks.

Java should also be installed on the server where you plan to install the ScaleIO Gateway.

To install the ScaleIO Gateway:

  1. Download from EMC Online Support the installation files needed for your operating system.
  2. Extract the installation files on the Installation Manager (IM) server
  3. For Linux:
    1. Run the following command (all in one line):

      GATEWAY_ADMIN_PASSWORD=<new_GW_admin_password> rpm -U /tmp/EMC-ScaleIO-gateway-1.32-XXX.X.noarch.rpm

      where <new_GW_admin_password> is the password to access the IM

  4. For Windows
    1. Run the .msi file, enter (and confirm) a new Gateway Admin password that will be used to access the IM
    2. Prepare disks for storage
  5. Install and configure ScaleIO components using the Installation Manager via the web client:
    1. In a Web browser, enter the following URL https:// <IM_server_IP>/ where <IM_Server_IP> is the IP address of the server where you installed the Installation Manager package
      ScaleIO_Installer_01.png
    2. Log in with the default user name (admin) and with the password defined when the Installation Manager was prepared, then click Login
      ScaleIO_Installer_02.png
    3. Click Get Started
      ScaleIO_Installer_03.png
    4. Browse to where the ScaleIO packages are located, select files, click Open, and click Upload. Click Proceed to Install.
      ScaleIO_Installer_04.png
    5. In the Provide Installation Topology screen, click Installation wizard. The Installation Configuration window appears:
      ScaleIO_Installer_05.png
    1.  In the Credentials Configuration section, enter and confirm a new MDM Password and the LIA password. In the Topology section, enter server information. Click Start Installation.
      ScaleIO_Installer_06.png
    2. Follow the post installation instructions displayed in the wizard.

For more information, refer to the EMC ScaleIO installation guide.

Dell EMC SC Series

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC SC Series REST API to collect capacity information about the Dell EMC SC Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Dell EMC SC Series REST API is 3033.

Migrating from Dell Compellent ETL to Storage All-In-- One ETL

When migrating from Dell Compellent Enterprise Manager ETL to Storage All-in-One ETL to monitor Dell Compellent systems, you may need to reconcile duplicate entities in order to guarantee the continuity of your historical data. Refer to the Lookup reconciliation and splitting on the BMC documentation website.

Dell EMC VPLEX (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC VPLEX REST API to collect capacity information about all Dell EMC clusters configured on Dell EMC VPLEX systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Dell EMC VPLEX REST API
Important

Some entities are named differently in the Dell EMC VPLEX user interface and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

Dell EMC VPLEX REST Entity
    • ======
Storage All-in-One ETL Entity
    • ======

Cluster

Storage System

Director

Controller

Port

Port

Volume

Volume

Dell EMC Unity (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Dell EMC Unity REST API to collect capacity information about the Dell EMC Unity systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

Prerequisites

  • Requires version 1.10.00 or higher or the engine.

Dell EMC VMAX (WBEM)

Click here to expand...

The Storage All-in-One ETL relies on the EMC SMI-S provider to collect capacity information about the Dell EMC VMAX systems. The Storage All-in-One ETL supports EMC VMAX3s models only.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

  • TrueSight Capacity Optimization version must be supported by the ETL
  • The Operating System of the ETL Engine running the ETL task must be a 64-bit version of Linux or Solaris
  • It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.

EMC SMI-S Provider

Before configuring the Storage All-in-One ETL to collect EMC VMAX metrics, you must ensure that the following requirements are met:


    •  (Recommended) Use the SMI-S provider pre-installed in the Management Console. No further action is required.
    •  (Alternate option) Install EMC SMI-S Provider on a different system. In this case, version 8.0.3 or higher is required.  If multiple VMAX systems are added to the same provider, the total number of volumes should not exceed 30,000 (hard limit of the provider is 50,000).

To manage EMC VMAX systems, the EMC SMI-S Provider requires at least one LUN to be mounted from each array to manage. These special LUNs are called "gatekeepers" and are used for the communication between the Symmetrix array and the SMI-S Provider (which does not have an IP address). EMC recommends having between 4 and 6 gatekeeper LUNs for each managed array. Increasing the number of gatekeepers dramatically improves the performance of the EMC SMI-S Provider.

Dell EMC VMAX/PowerMax (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Unisphere REST API to collect capacity information about the Dell EMC VMAX/PowerMax systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Unisphere REST API is 8443.

Prerequisites

Dell EMC Unisphere REST API  v.9.1+

Dell EMC XtremIO

Click here to expand...

The Storage All-in-One ETL relies on the EMC XtremIO REST API to collect capacity information about the EMC XtremIO systems. The connection to the REST API is performed via the HTTPS protocol.

The ETL identifies itself to the EMC XtremIO REST API v1 with user name and password or with an API token for EMC XtremIO REST API v2.

Fujitsu Eternus AF/DX (REST)

Click here to expand...

The Storage All-in-One ETL relies on the ETERNUS AF/DX RESTful API to collect capacity information about Eternus AF/DX Systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the ETERNUS AF/DX RESTful API with user name and password.

Due to REST API limitations, I/O statistics for byte rates or operation rates are not available.

Prerequisites

  • ETERNUS AF/DX RESTful API

Supported Platforms

The ETL supports the following Fujitsu Eternus models:

  • AF S3 series All-Flash Arrays
  • DX S5 series
  • DX8100 S4 Hybrid
  • DX8900 S4 Hybrid

Hitachi E, G Series (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Hitachi Configuration Manager REST API to collect capacity information about the Hitachi E and G Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Hitachi Configuration Manager REST API is 23451.

HPE Primera (WBEM)

Click here to expand...

The Storage All-in-One ETL relies HPE Primera web services API (WSAPI) to collect capacity information about the HPE Primera systems. The connection is performed via the HTTPS protocol through port 5989. The ETL identifies itself with a user name and password. The ETL supports HPE Primera and HP 3PAR StoreServ models.

This storage intelligence adapter is not compatible with HPE Primera software versions 4.5.3 and 4.5.4 due to the HPE Primera issue 370176, C28803.

Hitachi GUM (REST)

Click here to expand...

The Storage All-in-One ETL relies on the Hitachi GUM REST API to collect capacity information about the Hitachi GUM systems. The connection to the GUM REST API is performed via the HTTPS protocol. The ETL identifies itself to the GUM REST API with a username and password.

The default https port for the GUM REST API is 443.

Prerequisites

The Connection settings must provide the hostname or IP address of the controller as the Storage All-in-One ETL queries directly the controller(s).

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL  page displays the configuration properties.
  3. Expand the Run configuration section. In te Hostname field, enter the hostname or IP address of a controller.

Prerequisites.png

Known Limitation

Due to known limitations, the host-related information (ST_HOSTCNT, ST_SYSTEM_SUBSCRIBED_CAPACITY, ST_POOL_SUBSCRIBED_CAPACITY) is not available via GUM if your Hitachi storage system is running with Service Processor (SVP), We recommend using the Hitachi E Series, G Series (REST) adapter to collect host-related data.

Huawei OceanStor

Click here to expand...

The Storage All-in-One ETL relies on the Huawei OceanStor API to collect capacity information about the Huawei OceanStor systems. The connection to the REST API is performed via the HTTPS protocol through port 8088 . The ETL identifies itself to the REST API with a user name and password.

IBM DS8000 (REST)

Click here to expand...

The Storage All-in-One ETL relies on the DS8000 RESTful API v1.0 and higher to discover storage systems and collect performance/capacity metrics for IBM DS8000 series platforms. It collects metrics for the Storage System, Nodes, Ports, Pools and Volumes.

Prerequisites

  • DS8000 RESTful API v1.0 and higher

Supported Platforms

  • IBM DS8000 Series

Important information about entities reconciliation

Additional information for specifically Reconcile IBM DS888x (WBEM) and IBM DS8000 (REST) entities

By following these steps meticulously, you can seamlessly transition to utilizing IBM DS8000 (REST) while ensuring the reconciliation of Storage Pool and Storage Volume entities (Storage System entities are automatically reconciled), thereby preventing duplication and data collection discrepancies.

  1. Prepare Catalog:
    Begin by creating a catalog specifically for the IBM reconciliation process. For detailed instructions, refer to the documentation provided by BMC.

  2. Create WBEM Task:
    Generate a WBEM task and navigate to the Shared Entity Catalog section. Here, designate the catalog you previously created for IBM reconciliation.

  3. Create REST Task:
    Set up a REST task, ensuring to select the designated catalog in the Shared Entity Catalog section. Note: Use the same workspace as for the WBEM task.

  4. Stop WBEM Task:
    The WBEM task should be stopped to avoid conflicts during the reconciliation process.

  5. Review Catalog:
    Access the created catalog to proceed with reconciliation tasks. Important: In the Compare entities by option, select Entity name and type. Detailed guidelines can be found in the BMC documentation.

  6. Initiate Reconciliation:
    Execute the reconciliation process as outlined in the BMC documentation.


Known limitation

Due to a limitation in the REST API:

  1. disks are not discovered.
  2. some performance metrics are not available:
    • Storage Systems: no byte rate
    • Ports: bandwidth usage
    • Storage Pools: I/O stats
    • Volumes: I/O stats

IBM DS888x (WBEM)

Click here to expand...

The Storage All-in-One ETL relies on the IBM System Storage DS Open API, natively embedded within the Hardware Management Console (HMC) up to v8x, through the use of SMI-S provider to collect capacity information about the IBM DS888x systems.

The default port for HTTPS communication with the IBM DS888x SMI-S Provider is 6989.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

  • TrueSight Capacity Optimization version must be supported by the ETL
  • Valid credentials to the HMC must be provided
  • It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.

IBM Spectrum Control (REST)

Click here to expand...

The Storage All-in-One ETL relies on the IBM Spectrum Control API to collect capacity information about storage systems connected to IBM Spectrum Control. The connection to the REST API is performed via the HTTPS protocol through port 9569. The ETL identifies itself to the REST API with a user name and password.

IBM Spectrum Protect (REST)

Click here to expand...

The Storage All-in-One ETL relies on the IBM Spectrum Protect API to collect capacity information about storage systems connected to IBM Spectrum Protect Operations Center. The connection to the REST API is performed via the HTTPS protocol through port 11090. The ETL identifies itself to the REST API with a user name and password.

IBM Spectrum Scale (REST)

Click here to expand...

The Storage All-in-One ETL relies on the IBM Spectrum Scale Management API to collect capacity information about the IBM Spectrum Scale systems. The connection to the REST API is performed via the HTTPS protocol through port 443. The ETL identifies itself to the REST API with a user name and password.

NetApp Active IQ Unified Manager (AIQUM) REST API

Click here to expand...

To collect NetApp AIQUM metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the Active IQ Unified Manager (AIQUM) (REST API) endpoint. This storage intelligence adapter discovers all NetApp storage clusters configured on the Active IQ Unified Manager. It collects performance and capacity metrics for Clusters, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the AIQUM REST API is 443.

NetApp E-Series REST API

Click here to expand...

To collect NetApp E-Series metrics, the Storage All-in-One ETL relies on NetApp SANtricity REST API.

The default HTTP port for the NetApp SANtricity REST API is 8443.

Prerequisites

The NetApp SANtricity Web Services need to be installed and configured.

Accessing the Embedded NetApp SANtricity Web Services

The following information is required to access the NetApp SANtricity Web Services:

  • Credential Type (RESTful Web API with basic authentication)
  • SANtricity System Manager IP address.
  • Username and password that are used to access to the SANtricity System Manager

The configuration of the NetApp SANtricity Web Services Proxy differs according to the operating system running on the host:

Configuring the NetApp SANtricity Web Services Proxy on a Windows Host

    • Configure the Windows proxy to connect to the Windows host.
    • Configure credentials for the Windows Host.
    • Configure the RESTful Web API with basic authentication credentials:
Configuring the NetApp SANtricity Web Services Proxy on a Linux Host

    • Configure SSH Credentials for the Linux Host
    • Configure the RESTful Web API with basic authentication credentials:

NetApp ONTAP REST API

Click here to expand...

To collect NetApp ONTAP metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the ONTAP REST API endpoint. This storage intelligence adapter discovers the NetApp ONTAP storage system. It collects performance and capacity metrics for the Cluster, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the ONTAP REST API is 443.

NetApp SolidFire

Click here to expand...

To collect NetApp SolidFire metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the NetApp SolidFire Element OS API endpoint. HTTP basic authentication is sufficient to access the cluster and discover the system and all the physical nodes connected to the cluster. 

For more information, refer to NetApp SolidFire Element OS API Reference Guide.

NetApp StorageGRID REST API

Click here to expand...

To collect NetApp StorageGRID metrics, the Storage All-in-One ETL relies on the NetApp StorageGRID Management REST API.

The default HTTP port for the the NetApp StorageGRID Management REST API is 443.

Prerequisites

StorageGRID Management REST API v3.5 (including the Private API).

For detailed information about refer to the NetApp StorageGRID documentation.

Important

The Storage All-in-One ETL for TrueSight Capacity Optimization displays metric values in gibibyte (GiB) and megabyte per second (MB/s) while the NetApp StorageGRID user interface displays them in gigabyte (GB) and megabit per second (Mb/s).

Also, some entities are named differently in the NetApp StorageGRID user interface and the Storage All-in-One ETL. Refer to the table below to map entities from both environments:

NetApp StorageGRID UI Entity
    • ======
Storage All-in-One ETL Entity
    • ======
Site
Storage System
Node
Controller
Port
Port
Object Store
Storage Pool
Disk Device
Storage Volume
Volume
Storage Share

Oracle ZFS REST

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Oracle ZFS Storage Appliance RESTful API v2.0 or higher to discover and collect capacity information about Oracle ZFS Storage systems. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Prerequisites

Oracle ZFS Storage Appliance RESTful API v2.0 or higher.

Supported Platforms

Oracle ZFS Storage Appliance ZS9-2 (ZFS Storage Appliance | Oracle).

Pure Storage FlashArray REST

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API to collect capacity information about the Pure Storage systems. Depending on the API version available, you will use either Pure Storage FlashArray (REST v1.x) or Pure Storage FlashArray (REST v2.4+). To determine the version supported by your system, refer to the article Storage All-in-One ETL: Verifying Prerequisites (Credentials and Connectivity).

Prerequisites

  • When using Pure Storage FlashArray (REST v1.x):
    • Pure Storage REST API v1.6 or higher
    • Username and password for authentication
  • When using Pure Storage FlashArray (REST v2.4+):
    • Pure Storage REST API v2.4 and higher
    • API token for authentication.

Migration from standalone ETL

If migrating from the standalone Pure Storage REST API ETL, note that the token-based authentication method is no longer supported by Pure or the Storage All-in-One ETL. You must now use a username and password for authentication. 

Pure Storage FlashBlade REST API

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API v2.0, or higher, to collect capacity information about the Pure Storage FlashBlade S and E series All-Flash Arrays. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies with an API token you would have previously generated (Refer to Generating an API Token for more information). The ETL collects performance and capacity metrics for the Storage Arrays, Fabric Modules, Connectors, Drives, Buckets, File Systems and Object Stores.

Prerequisites

The Pure Storage flash array must support Pure Storage REST API v2.0 or higher.

Supported Platforms

  • Pure Storage FlashBlade S
  • Pure Storage FlashBlade E

Scality RING (Scality Supervisor API)

Click here to expand...

Introduction

The Storage All-in-One ETL relies on the Scality Supervisor API (SupAPI) to collect capacity information about the Scality RING systems through the supervisor node. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Limitation

The Scality Supervisor API does not report I/O stats.

Deploying the ETL module

Complete the following steps to deploy the ETL Module:

  1. Navigate to Administration > SYSTEM > Maintenance.
  2. Verify the installation status of the already uploaded packages. If their status is "READY" or "RUNNING", wait for the installation to complete before proceeding.
  3. At the bottom of the Maintenance page, locate the Upload patch or additional package file section.
  4. Click Browse.
  5. Locate and select the ETL installation package.
  6. Click Upload.
  7. Read the Overview and click Next to continue.
  8. Read and accept the End User License Agreement.
  9. Select the server on which the operation must be executed; that is the server on which the package will be saved. Click Proceed to continue.
  10. A successful installation will result in a green line in the Additional Packages table.

    Note

     In case of unsuccessful outcome (e.g. a yellow WARNING line), click the View_Deployment_Log.png icon to inspect the deployment log.

Once the Storage All-in-One ETL is installed, the Storage Engine and Storage Intelligence rows appear in the Administration > System > Maintenance > Additional Packages tab:

2102_Storage-All-in-One_DeployResult.png


Step II. Configure the ETL

You must configure the ETL to connect to the required storage system for data collection. ETL configuration includes specifying the basic and optional advanced properties. While configuring the basic properties is sufficient, you can optionally configure the advanced properties for additional customization.

A. Configuring the basic properties

Some of the basic properties display default values. You can modify these values if required.

To configure the basic properties:

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL  page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, and ETL additional settings.
  3. On the Run configuration tab, select Storage - All-in-One Extractor Service from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You can edit this field to customize the name.
    115Storage-All-in-One_RunConfiguration_Tab.png
  4. Click the Entity catalog tab, and select one of the following options:
    • Shared Entity Catalog:

      Select if ETLs access the same entities.

      • From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
    • Private Entity Catalog: Select if this is the only ETL that extracts data.
  5. Click the Connection Settings tab, and configure the following properties:

    Property

    Description

    Storage Solution

    Storage solution to be configured. Select:

      • Ceph Storage REST to collect capacity information about your Ceph Storage systems
      • Dell EMC Isilon to collect capacity information about your Dell EMC Isilon systems
      • Dell EMC ScaleIO REST/Dell EMC PowerFlex REST to collect capacity information about your Dell EMC ScaleIO and Dell EMC PowerFlex systems
      • Dell EMC SC Series to collect capacity information about your Dell EMC SC Series systems
      • Dell EMC Unity REST to collect capacity information about your Dell EMC Unity systems
      • Dell EMC PowerMax v4 REST to collect capacity information about your Dell EMC PowerMax v4 systems
      • Dell EMC PowerStore REST to collect capacity information about your Dell EMC PowerStore systems
      • Dell EMC PowerVault REST to collect capacity information about your Dell EMC PowerVault systems
      • Dell EMC VMAX WBEM to collect capacity information about your Dell EMC VMAX systems
      • Dell EMC VMAX/PowerMax REST to collect capacity information about your Dell EMC VMAX/PowerMax systems
      • Dell EMC XtremIO REST to collect capacity information about your Dell EMC XtremIO systems
      • HPE Primera WBEM to collect capacity information about your HPE Primera systems
      • Huawei OceanStor REST to collect capacity information about your Huawei OceanStor systems
      • Hitachi G Series REST to collect capacity information about your Hitachi G Series systems
      • Hitachi GUM REST to collect capacity information about your Hitachi GUM systems
      • IBM DS8880 WBEM to collect capacity information about your IBM DS888x systems
      • IBM Spectrum Scale REST to collect capacity information about your IBM Spectrum Scale systems
      • NetApp Active IQ Unified Manager REST to collect capacity information about your NetApp Active IQ Unified Manager systems
      • NetApp E-Series REST to collect capacity information about your NetApp E-Series systems
      • NetApp ONTAP REST to collect capacity information about your NetApp ONTAP systems
      • NetApp SolidFire REST to collect capacity information about your NetApp SolidFire systems
      • Oracle ZFS REST to collect capacity information about your Oracle ZFS Storage systems
      • Pure Storage FlashBlade REST to collect capacity information about your Pure Storage FlashBlade systems
      • Pure Storage REST to collect capacity information about your Pure Storage systems
      • Scality RING (Scality Supervisor API) to collect capacity information about your Scality RING systems
      • Other Storage Solution to collect capacity information about any other storage solution supported by the Storage-All-in-One ETL. In this case, you will have to manually enter the name of the Si (Storage Intelligence) file to be used.

    Other Storage Solution

    Name of the Storage Intelligence file (without extension) to be used. This field should only be specified if the Storage Solution property is set to Other Storage Solution.

    Hostname

    Hostname or IP address of the storage solution.

    Port

    Port number used to connect to the storage solution.

    Connect As

    Password to connect to the storage solution.

    Transport Protocol

    Transport protocol used to connect to the storage solution (HTTP or HTTPS).

    Property

    Description

    Storage Solution

    Storage solution to be configured. Select:

      • Ceph Storage REST to collect capacity information about your Ceph Storage systems
      • Dell EMC Isilon to collect capacity information about your Dell EMC Isilon systems
      • Dell EMC ScaleIO REST/Dell EMC PowerFlex REST to collect capacity information about your Dell EMC ScaleIO and Dell EMC PowerFlex systems
      • Dell EMC SC Series to collect capacity information about your Dell EMC SC Series systems
      • Dell EMC Unity REST to collect capacity information about your Dell EMC Unity systems
      • Dell EMC PowerMax v4 REST to collect capacity information about your Dell EMC PowerMax v4 systems
      • Dell EMC PowerStore REST to collect capacity information about your Dell EMC PowerStore systems
      • Dell EMC PowerVault REST to collect capacity information about your Dell EMC PowerVault systems
      • Dell EMC VMAX WBEM to collect capacity information about your Dell EMC VMAX systems
      • Dell EMC VMAX/PowerMax REST to collect capacity information about your Dell EMC VMAX/PowerMax systems
      • Dell EMC XtremIO REST to collect capacity information about your Dell EMC XtremIO systems
      • HPE Primera WBEM to collect capacity information about your HPE Primera systems
      • Huawei OceanStor REST to collect capacity information about your Huawei OceanStor systems
      • Hitachi G Series REST to collect capacity information about your Hitachi G Series systems
      • Hitachi GUM REST to collect capacity information about your Hitachi GUM systems
      • IBM DS8880 WBEM to collect capacity information about your IBM DS888x systems
      • IBM Spectrum Scale REST to collect capacity information about your IBM Spectrum Scale systems
      • NetApp Active IQ Unified Manager REST to collect capacity information about your NetApp Active IQ Unified Manager systems
      • NetApp E-Series REST to collect capacity information about your NetApp E-Series systems
      • NetApp ONTAP REST to collect capacity information about your NetApp ONTAP systems
      • NetApp SolidFire REST to collect capacity information about your NetApp SolidFire systems
      • Oracle ZFS REST to collect capacity information about your Oracle ZFS Storage systems
      • Pure Storage FlashBlade REST to collect capacity information about your Pure Storage FlashBlade systems
      • Pure Storage REST to collect capacity information about your Pure Storage systems
      • Scality RING (Scality Supervisor API) to collect capacity information about your Scality RING systems
      • Other Storage Solution to collect capacity information about any other storage solution supported by the Storage-All-in-One ETL. In this case, you will have to manually enter the name of the Si (Storage Intelligence) file to be used.

    Other Storage Solution

    Name of the Storage Intelligence file (without extension) to be used. This field should only be specified if the Storage Solution property is set to Other Storage Solution.

    Hostname

    Hostname or IP address of the storage solution.

    Port

    Port number used to connect to the storage solution.

    Connect As

    Password to connect to the storage solution.

    Transport Protocol

    Transport protocol used to connect to the storage solution (HTTP or HTTPS).

     

  6. Click Save. You are returned to the Last run tab under the ETL tasks page.
    1. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click Button_Run.png to run the ETL.
      • OK: the ETL executed without any error in simulation mode.
      • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
      • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
    2. Switch the ETL to production mode: Perform the following steps:
      1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
      2. In the Run configuration table in the ETL details page, click Edit_Icon.png to edit the active Run configuration.
      3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
      4. Click Save. 
  1. Locate the ETL in the ETL tasks table and click Button_Run.png to run it, or schedule an ETL run. 
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to TrueSight Capacity Optimization database. You can see the entity hierarchy in the left Navigation pane, under Workspace.