Storage - All-in-One Extractor Service

The Storage All-in-One ETL relies on the Ceph REST API to collect capacity information about the Ceph cluster. The connection to the REST API is performed via the HTTP protocol. The ETL first identifies itself to the REST API with a user name and password to retrieve a bearer token from the API and then uses this token for further REST requests.

The default https port for the CEPH Rest API is 8003 (5000 for http).

Note: The ETL has been tested with the version 14.2.5 of the Ceph REST API.

Prerequisites

The Ceph dashboard must be activated with the following command: ceph mgr module enable dashboard.

The Storage All-in-One ETL relies on the Dell EMC Avamar REST API v10 to collect capacity information about the Dell EMC Avamar systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with an access-token.

Prerequisites

• Avamar REST API
• A non-expiring access-token.

The Storage All-in-One ETL relies on the Dell EMC OneFS REST API to collect capacity information about the Dell EMC Isilon systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Isilon REST API is 8080.

The ETL requires at least version 8 of EMC OneFS.

The Storage All-in-One ETL relies on Unisphere v10.1 and higher to collect capacity metrics about Dell EMC PowerMax systems.

The Storage All-in-One ETL relies on the Dell EMC Unisphere REST API v10 to collect capacity information about the Dell EMC PowerMax v4 systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

• Requires version 1.10.00 or higher or the engine
• Requires Dell EMC Unisphere REST API v10 

Supported Platforms

• PowerMax 2500
• PowerMax 8500

The Storage All-in-One ETL relies on the Data Domain RESTfull API to collect capacity information about the Dell EMC PowerProtect DD systems. The connection to the REST API is performed via the HTTPS protocol (Default port: 3009). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

• Dell EMC Data Domain RESTful API (included in DD OS v7.10+).

Known Limitations

Due to REST API limitations, Fiber Channels and Chassis are not discovered and some performance metrics are not available.

The Storage All-in-One ETL relies on the Dell EMC PowerStore Management REST API to collect capacity information about the Dell EMC PowerStore systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

• Requires version 1.10.00 or higher or the engine.

Limitations

• The maximum number of supported volumes is limited to 2000.

The Storage All-in-One ETL relies on the Dell EMC PowerVault Management REST API to collect capacity information about the Dell EMC PowerVault systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

• Version 1.10.00 or higher of the engine.

The Storage All-in-One ETL leverages the REST API installed as part of the ScaleIO Gateway to collect capacity metrics about Dell EMC ScaleIO/Dell EMC PowerFlex systems.

The ScaleIO Gateway:

• can be installed on the same node as other ScaleIO components if the node has adequate memory to run the Installation Manager (minimum of 2 GB) and any other applications.
• must have connectivity to the data and management ScaleIO networks.

Java should also be installed on the server where you plan to install the ScaleIO Gateway.

To install the ScaleIO Gateway:

1. Download from EMC Online Support the installation files needed for your operating system.
2. Extract the installation files on the Installation Manager (IM) server
3. For Linux:

   1. Run the following command (all in one line):

      GATEWAY_ADMIN_PASSWORD=<new_GW_admin_password> rpm -U /tmp/EMC-ScaleIO-gateway-1.32-XXX.X.noarch.rpm

      where <new_GW_admin_password> is the password to access the IM

4. For Windows: 
   1. Run the .msi file, enter (and confirm) a new Gateway Admin password that will be used to access the IM
   2. Prepare disks for storage
5. Install and configure ScaleIO components using the Installation Manager via the web client:
   1. In a Web browser, enter the following URL https:// <IM_server_IP>/ where <IM_Server_IP> is the IP address of the server where you installed the Installation Manager package
   2. Log in with the default user name (admin) and with the password defined when the Installation Manager was prepared, then click Login
   3. Click Get Started
   4. Browse to where the ScaleIO packages are located, select files, click Open, and click Upload. Click Proceed to Install.
6. In the Provide Installation Topology screen, click Installation wizard. The Installation Configuration window appears:
   1.  In the Credentials Configuration section, enter and confirm a new MDM Password and the LIA password. In the Topology section, enter server information. Click Start Installation.
      ​​​​
   2. Follow the post installation instructions displayed in the wizard.

For more information, refer to the EMC ScaleIO installation guide.

The Storage All-in-One ETL relies on the Dell EMC SC Series REST API to collect capacity information about the Dell EMC SC Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Dell EMC SC Series REST API is 3033.

Migrating from Dell Compellent ETL to Storage All-In-- One ETL

When migrating from Dell Compellent Enterprise Manager ETL to Storage All-in-One ETL to monitor Dell Compellent systems, you may need to reconcile duplicate entities in order to guarantee the continuity of your historical data. Refer to the Lookup reconciliation and splitting on the BMC documentation website.

The Storage All-in-One ETL relies on the Dell EMC VPLEX REST API to collect capacity information about all Dell EMC clusters configured on Dell EMC VPLEX systems. The connection to the REST API is performed via the HTTPS protocol (default port: 443). The ETL identifies itself to the REST API with a user name and password.

Prerequisites

• Dell EMC VPLEX REST API

The Storage All-in-One ETL relies on the Dell EMC Unity REST API to collect capacity information about the Dell EMC Unity systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

Prerequisites

• Requires version 1.10.00 or higher or the engine.

The Storage All-in-One ETL relies on the EMC SMI-S provider to collect capacity information about the Dell EMC VMAX systems. The Storage All-in-One ETL supports EMC VMAX3s models only.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

• TrueSight Capacity Optimization version must be supported by the ETL
• The Operating System of the ETL Engine running the ETL task must be a 64-bit version of Linux or Solaris
• It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.

EMC SMI-S Provider

Before configuring the Storage All-in-One ETL to collect EMC VMAX metrics, you must ensure that the following requirements are met:

• 
  
  •  (Recommended) Use the SMI-S provider pre-installed in the Management Console. No further action is required.
  •  (Alternate option) Install EMC SMI-S Provider on a different system. In this case, version 8.0.3 or higher is required.  If multiple VMAX systems are added to the same provider, the total number of volumes should not exceed 30,000 (hard limit of the provider is 50,000).

To manage EMC VMAX systems, the EMC SMI-S Provider requires at least one LUN to be mounted from each array to manage. These special LUNs are called "gatekeepers" and are used for the communication between the Symmetrix array and the SMI-S Provider (which does not have an IP address). EMC recommends having between 4 and 6 gatekeeper LUNs for each managed array. Increasing the number of gatekeepers dramatically improves the performance of the EMC SMI-S Provider.

The Storage All-in-One ETL relies on the Unisphere REST API to collect capacity information about the Dell EMC VMAX/PowerMax systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Unisphere REST API is 8443.

Prerequisites

Dell EMC Unisphere REST API  v.9.1+

The Storage All-in-One ETL relies on the EMC XtremIO REST API to collect capacity information about the EMC XtremIO systems. The connection to the REST API is performed via the HTTPS protocol.

The ETL identifies itself to the EMC XtremIO REST API v1 with user name and password or with an API token for EMC XtremIO REST API v2.

The Storage All-in-One ETL relies on the ETERNUS AF/DX RESTful API to collect capacity information about Eternus AF/DX Systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the ETERNUS AF/DX RESTful API with user name and password.

Prerequisites

• ETERNUS AF/DX RESTful API

Supported Platforms

The ETL supports the following Fujitsu Eternus models:

• AF S3 series All-Flash Arrays
• DX S5 series
• DX8100 S4 Hybrid
• DX8900 S4 Hybrid

The Storage All-in-One ETL relies on the Hitachi Configuration Manager REST API to collect capacity information about the Hitachi E and G Series systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The default https port for the Hitachi Configuration Manager REST API is 23451.

The Storage All-in-One ETL relies on the HPE MSA REST API to collect capacity information about the HPE MSA 1000/2000 Series systems over port 443 and authenticates with a username and password (minimum required role: Monitor).

The Storage All-in-One ETL relies HPE Primera web services API (WSAPI) to collect capacity information about the HPE Primera systems. The connection is performed via the HTTPS protocol through port 5989. The ETL identifies itself with a user name and password. The ETL supports HPE Primera and HP 3PAR StoreServ models.

The Storage All-in-One ETL relies on the Hitachi GUM REST API to collect capacity information about the Hitachi GUM systems. The connection to the GUM REST API is performed via the HTTPS protocol. The ETL identifies itself to the GUM REST API with a username and password.

The default https port for the GUM REST API is 443.

Prerequisites

The Connection settings must provide the hostname or IP address of the controller as the Storage All-in-One ETL queries directly the controller(s).

1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
2. On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL  page displays the configuration properties.
3. Expand the Run configuration section. In te Hostname field, enter the hostname or IP address of a controller.

Known Limitation

Due to known limitations, the host-related information (ST_HOSTCNT, ST_SYSTEM_SUBSCRIBED_CAPACITY, ST_POOL_SUBSCRIBED_CAPACITY) is not available via GUM if your Hitachi storage system is running with Service Processor (SVP), We recommend using the Hitachi E Series, G Series (REST) adapter to collect host-related data.

The Storage All-in-One ETL relies on the Huawei OceanStor API to collect capacity information about the Huawei OceanStor systems. The connection to the REST API is performed via the HTTPS protocol through port 8088 . The ETL identifies itself to the REST API with a user name and password.

The Storage All-in-One ETL relies on the OceanStor Pacific RESTful API to collect capacity information about the Huawei OceanStor Pacific systems. The connection to the REST API is performed via the HTTPS protocol through port 8088 . The ETL identifies itself to the REST API with a user name and password.

The Storage All-in-One ETL relies on the DS8000 RESTful API v1.0 and higher to discover storage systems and collect performance/capacity metrics for IBM DS8000 series platforms. It collects metrics for the Storage System, Nodes, Ports, Pools and Volumes.

Prerequisites

• DS8000 RESTful API v1.0 and higher

Supported Platforms

• IBM DS8000 Series

The Storage All-in-One ETL relies on the IBM System Storage DS Open API, natively embedded within the Hardware Management Console (HMC) up to v8x, through the use of SMI-S provider to collect capacity information about the IBM DS888x systems.

The default port for HTTPS communication with the IBM DS888x SMI-S Provider is 6989.

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

• TrueSight Capacity Optimization version must be supported by the ETL
• Valid credentials to the HMC must be provided
• It is recommended that the SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task be set to at least 2048m to avoid out-of-memory issues. This variable can be customized according to the amount of managed volumes and only if data collection problems occur.

The Storage All-in-One ETL relies on the IBM Spectrum Control API to collect capacity information about storage systems connected to IBM Spectrum Control. The connection to the REST API is performed via the HTTPS protocol through port 9569. The ETL identifies itself to the REST API with a user name and password.

The Storage All-in-One ETL relies on the IBM Spectrum Protect API to collect capacity information about storage systems connected to IBM Spectrum Protect Operations Center. The connection to the REST API is performed via the HTTPS protocol through port 11090. The ETL identifies itself to the REST API with a user name and password.

The Storage All-in-One ETL relies on the IBM Spectrum Scale Management API to collect capacity information about the IBM Spectrum Scale systems. The connection to the REST API is performed via the HTTPS protocol through port 443. The ETL identifies itself to the REST API with a user name and password.

To collect NetApp AIQUM metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the Active IQ Unified Manager (AIQUM) (REST API) endpoint. This storage intelligence adapter discovers all NetApp storage clusters configured on the Active IQ Unified Manager. It collects performance and capacity metrics for Clusters, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the AIQUM REST API is 443.

To collect NetApp E-Series metrics, the Storage All-in-One ETL relies on NetApp SANtricity REST API.

The default HTTP port for the NetApp SANtricity REST API is 8443.

Prerequisites

The NetApp SANtricity Web Services need to be installed and configured.

Accessing the Embedded NetApp SANtricity Web Services

The following information is required to access the NetApp SANtricity Web Services:

• Credential Type (RESTful Web API with basic authentication)
• SANtricity System Manager IP address.
• Username and password that are used to access to the SANtricity System Manager

The configuration of the NetApp SANtricity Web Services Proxy differs according to the operating system running on the host:

Configuring the NetApp SANtricity Web Services Proxy on a Windows Host

• 
  
  • Configure the Windows proxy to connect to the Windows host.
  • Configure credentials for the Windows Host.
  • Configure the RESTful Web API with basic authentication credentials:
    • Windows Host IP address.
    • Login and Password that will be used for the connection to the proxy. Refer to the NetApp SANtricity Web Services Proxy User Guide (page 5) for detailed information.
    • Protocol HTTP.

Configuring the NetApp SANtricity Web Services Proxy on a Linux Host

• 
  
  • Configure SSH Credentials for the Linux Host
  • Configure the RESTful Web API with basic authentication credentials:
    • Linux Host IP address.
    • Login and Password that will be used for the connection to the proxy. Refer to the NetApp SANtricity Web Services Proxy User Guide (page 5) for detailed information.
    • Protocol HTTP.

To collect NetApp ONTAP metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the ONTAP REST API endpoint. This storage intelligence adapter discovers the NetApp ONTAP storage system. It collects performance and capacity metrics for the Cluster, Storage VMs, Nodes, Fiber and Ethernet Ports, Disks, Aggregates, Volumes and LUNs.

The default HTTP port for the ONTAP REST API is 443.

To collect NetApp SolidFire metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the NetApp SolidFire Element OS API endpoint. HTTP basic authentication is sufficient to access the cluster and discover the system and all the physical nodes connected to the cluster. 

For more information, refer to NetApp SolidFire Element OS API Reference Guide.

To collect NetApp StorageGRID metrics, the Storage All-in-One ETL relies on the NetApp StorageGRID Management REST API.

The default HTTP port for the the NetApp StorageGRID Management REST API is 443.

Prerequisites

StorageGRID Management REST API v3.5 (including the Private API).

For detailed information about refer to the NetApp StorageGRID documentation.

Introduction

The Storage All-in-One ETL relies on the Oracle ZFS Storage Appliance RESTful API v2.0 or higher to discover and collect capacity information about Oracle ZFS Storage systems. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Prerequisites

Oracle ZFS Storage Appliance RESTful API v2.0 or higher.

Supported Platforms

Oracle ZFS Storage Appliance ZS9-2 (ZFS Storage Appliance | Oracle).

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API to collect capacity information about the Pure Storage systems. Depending on the API version available, you will use either Pure Storage FlashArray (REST v1.x) or Pure Storage FlashArray (REST v2.4+). To determine the version supported by your system, refer to the article Storage All-in-One ETL: Verifying Prerequisites (Credentials and Connectivity).

Prerequisites

• When using Pure Storage FlashArray (REST v1.x):
  • Pure Storage REST API v1.6 or higher
  • Username and password for authentication
• When using Pure Storage FlashArray (REST v2.4+):
  • Pure Storage REST API v2.4 and higher
  • API token for authentication.

Migration from standalone ETL

If migrating from the standalone Pure Storage REST API ETL, note that the token-based authentication method is no longer supported by Pure or the Storage All-in-One ETL. You must now use a username and password for authentication. 

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API v2.0, or higher, to collect capacity information about the Pure Storage FlashBlade S and E series All-Flash Arrays. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies with an API token you would have previously generated (Refer to Generating an API Token for more information). The ETL collects performance and capacity metrics for the Storage Arrays, Fabric Modules, Connectors, Drives, Buckets, File Systems and Object Stores.

Prerequisites

The Pure Storage flash array must support Pure Storage REST API v2.0 or higher.

Supported Platforms

• Pure Storage FlashBlade S
• Pure Storage FlashBlade E

Introduction

The Storage All-in-One ETL relies on the Scality Supervisor API (SupAPI) to collect capacity information about the Scality RING systems through the supervisor node. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Limitation

The Scality Supervisor API does not report I/O stats.

The Storage All-in-One ETL relies on the DSM Web APIs to collect capacity information about the Synology NAS systems. The connection to the REST API is performed via the HTTPS protocol (default port: 5000) and the ETL identifies itself to the REST API with a user name and password (minimum required role: User).