Deploying and configuring Storage All-in-One ETL for TrueSight Capacity Optimization

This topic contains the following sections:

Compatibility details

TrueSight Capacity Optimization

TrueSight Capacity Optimization v11.5.

Supported platforms

  • Dell EMC ScaleIO
  • Dell EMC VMAX
  • NetApp SolidFire

The Storage All-in-One ETL and the EMC SMI-S Agent ETL can operate simultaneously.

Prerequisites

Dell EMC ScaleIO

Click here to expand...

The Storage All-in-One ETL leverages the REST API installed as part of the ScaleIO Gateway to collect capacity metrics about Dell EMC ScaleIO systems.

The ScaleIO Gateway:

  • can be installed on the same node as other ScaleIO components if the node has adequate memory to run the Installation Manager (minimum of 2 GB) and any other applications.
  • must have connectivity to the data and management ScaleIO networks.

Java should also be installed on the server where you plan to install the ScaleIO Gateway.


To install the ScaleIO Gateway:

  1. Download from EMC Online Support the installation files needed for your operating system.
  2. Extract the installation files on the Installation Manager (IM) server
  3. For Linux:

    1. Run the following command (all in one line):

      GATEWAY_ADMIN_PASSWORD=<new_GW_admin_password> rpm -U /tmp/EMC-ScaleIO-gateway-1.32-XXX.X.noarch.rpm

      where <new_GW_admin_password> is the password to access the IM

  4. For Windows:
    1. Run the .msi file, enter (and confirm) a new Gateway Admin password that will be used to access the IM
    2. Prepare disks for storage
  5. Install and configure ScaleIO components using the Installation Manager via the web client:
    1. In a Web browser, enter the following URL https:// <IM_server_IP>/ where <IM_Server_IP> is the IP address of the server where you installed the Installation Manager package
      ScaleIO_Installer_01.png
    2. Log in with the default user name (admin) and with the password defined when the Installation Manager was prepared, then click Login
      ScaleIO_Installer_02.png
    3. Click Get Started
      ScaleIO_Installer_03.png
    4. Browse to where the ScaleIO packages are located, select files, click Open, and click Upload. Click Proceed to Install.
      ScaleIO_Installer_04.png
    5. In the Provide Installation Topology screen, click Installation wizard. The Installation Configuration window appears:
      ScaleIO_Installer_05.png
    1.  In the Credentials Configuration section, enter and confirm a new MDM Password and the LIA password. In the Topology section, enter server information. Click Start Installation.
      ScaleIO_Installer_06.png
    2. Follow the post installation instructions displayed in the wizard.

For more information, refer to the EMC ScaleIO installation guide.

Dell EMC VMAX

Click here to expand...

The Storage All-in-One ETL relies on the EMC SMI-S provider to collect capacity information about the Dell EMC VMAX systems. Refer to the table below to know which version of the EMC SMI-S provider should be used.

EMC Disk Arrays

EMC Hardware

Supported versions of the EMC SMI-S Provider

v8.3.0.3

v9.0.00

VMAX (10K, 20K, 40K series)

supported.png

supported.png

VMAX3 (100K, 200K, 400K series)

supported.png

supported.png

Prerequisites

Before configuring the Storage All-in-One ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.

Hardware prerequisites

CPU

2 x 64 CPUs (or 2 virtual CPUs)

Memory

8GB

Disk space

50 GB, 10K rpm class

Network

1 Gb/s network adapter

Software prerequisites

TrueSight Capacity Optimization

A version supported by the ETL.

Operating System

A supported 64-bit version of Linux or Solaris.

Scheduler prerequisites

The SCHEDULER_HEAP_SIZE environment variable must be set to the following heap size: 2048m

EMC SMI-S Provider

Before configuring the Storage All-in-One ETL to collect EMC VMAX metrics, you must ensure that the following requirements are met:

  •  (Recommended) Use the SMI-S provider pre-installed in the Management Console. No further action is required.
  •  (Alternate option) Install EMC SMI-S Provider on a different system. In this case, version 8.0.3 or higher is required.

To manage EMC VMAX systems, the EMC SMI-S Provider requires at least one LUN to be mounted from each array to manage. These special LUNs are called "gatekeepers" and are used for the communication between the Symmetrix array and the SMI-S Provider (which does not have an IP address). EMC recommends having between 4 and 6 gatekeeper LUNs for each managed array. Increasing the number of gatekeepers dramatically improves the performance of the EMC SMI-S Provider.


NetApp SolidFire

Click here to expand...

To collect NetApp SolidFire metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the NetApp SolidFire Element OS API endpoint. HTTP basic authentication is sufficient to access the cluster and discover the system and all the physical nodes connected to the cluster. 

For more information, refer to NetApp SolidFire Element OS API Reference Guide.

Deploying the ETL module

Complete the following steps to deploy the ETL Module:

  1. Navigate to Administration > SYSTEM > Maintenance.
  2. Verify the installation status of the already uploaded packages. If their status is "READY" or "RUNNING", wait for the installation to complete before proceeding.
  3. At the bottom of the Maintenance page, locate the Upload patch or additional package file section.
  4. Click Browse.
  5. Locate and select the ETL installation package.
  6. Click Upload.
  7. Read the Overview and click Next to continue.
  8. Read and accept the End User License Agreement.
  9. Select the server on which the operation must be executed; that is the server on which the package will be saved. Click Proceed to continue.

  10. A successful installation will result in a green line in the Additional Packages table.

    Note

     In case of unsuccessful outcome (e.g. a yellow WARNING line), click the View_Deployment_Log.png icon to inspect the deployment log.

Once the Storage All-in-One ETL is installed, the Storage Engine and Storage Intelligence rows appear in the Additional Packages tab:

115Storage-All-in-One_Install.png

Configuring the ETL module

Complete the following steps to configure an ETL service and add it as an ETL task to extract capacity and performance metrics:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL page is displayed.
  3. In the Run configuration tab, in the ETL module property, select Storage - All-in-One Extractor Service 
    115Storage-All-in-One_RunConfiguration_Tab.png

Specify values for the common and advanced properties that are listed under the expandable tabs. To view or configure advanced properties, click Advanced.
For details about the common properties, see ETL common configuration properties. The following table lists the properties that are specific to the Storage All in One ETL module: 

Property

Description

Connection settings

Storage Solution

Storage solution to be configured. Select:

  • Dell EMC ScaleIO REST to collect capacity information about your Dell EMC ScaleIO systems
  • Dell EMC VMAX WBEM to collect capacity information about your Dell EMC VMAX systems
  • NetApp SolidFire REST to collect capacity information about your NetApp SolidFire systems
  • Other Storage Solution to collect capacity information about any other storage solution supported by the Storage-All-in-One ETL. In this case, you will have to manually enter the name of the Si (Storage Intelligence) file to be used.

Other Storage Solution

Name of the Storage Intelligence file (without extension) to be used. This field should only be specified if the Storage Solution property is set to Other Storage Solution.

Hostname

Hostname or IP address of the storage solution.

Port

Port number used to connect to the storage solution.

Connect As

Username to connect to the storage solution.

Password

Password to connect to the storage solution.

Transport Protocol

Transport protocol used to connect to the storage solution (HTTP or HTTPS).

  1. Click Save. You are returned to the Last run tab under the ETL tasks page. 
  1. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click Button_Run.png to run the ETL.
    • OK: the ETL executed without any error in simulation mode. 
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again. 
  1. Switch the ETL to production mode: Perform the following steps:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
    2. In the Run configuration table in the ETL details page, click edit_button.gif to edit the active Run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save. 
  1. Locate the ETL in the ETL tasks table and click Button_Run.png to run it, or schedule an ETL run. 
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to TrueSight Capacity Optimization database. You can see the entity hierarchy in the left Navigation pane, under Workspace.

Post-configuration steps

Reconciling data

If you used the same Shared Entity Catalog while creating your ETL tasks, data will be automatically reconciled since the ETL tasks are using the same lookup details. If you used a Private Entity Catalog or a different entity catalog, you will have to reconcile data. The procedure is as follows:

  1. Create the new entity catalog:
    1. In the Administration tab, click ETL & System Tasks > Entity catalogs
    2. Click Add entity catalog
    3. Name your entity catalog and eventually provide a description
      115_AddEntityCatalog-EMC.png
    4. Click Save.
  2. Configure both ETL tasks to use this entity catalog:
    1. In the Administration tab, click ETL & System Tasks > ETL tasks.Click the link of the ETL task for which you wish to set the entity catalog
    2. Click the Stop button to stop the service execution; then click the Edit_Icon.png button available in the Run configuration section.
    3. In the Entity Catalog section:
      1. Select the Shared Entity Catalog radio button
      2. From the Sharing with Entity Catalog pull-down list, select the entity catalog previously created (in our example: Storage)
        115_SelectEntityCatalog-EMC.png
    4. Click Save.
    5. Start the ETL.
    6. Resume the procedure for the second ETL task. 
  3. Perform the reconcile operation:
    1. In the Administration tab, click ETL & System Tasks > Entity catalogs
    2. Click the entity catalog previously created (in our example: EMC)
      115_ReconcileEntities-EMC.png
    3. Click the Reconcile entities button
  4. Perform the entity reconciliation:
    115_EntityReconciliation-EMC.png
    1. Select the Datasource for which you wish to compare the Lookup fields
    2. Compare entities by Lookup fields and select the look fields
    3. Click Search to determine the entities that can be reconciled
    4. Either select the entities to be reconciled and click Reconcile selected rows or click Reconcile all.
  5. Verify in your workspace that all the selected entities are properly reconciled. Only one instance should be created.

Configuring the discovery of VMAX remote arrays

VMAX remote arrays are not discovered by default. To enable the discovery, you will have to add the property discoverRemoteArrays and set it to true. The procedure is as follows:

  1. Edit the ETL Run Configuration:
    1. In the Administration tab, click ETL & System Tasks > ETL tasks.
    2. Click the link of the ETL task for which you wish to set the collect frequency.
    3. Click the Stop button to stop the service execution; then click the Edit_Icon.png button available in the Run configuration section.
  2. Click the link You can manually edit ETL properties from this page provided at the bottom of the page to display a list of editable options.
  3. In the Add new property field, enter discoverRemoteArrays and click Add.
    discoverRemoteArrays_01.png
  4. In the list of properties, locate discoverRemoteArrays and enter true.
    discoverRemoteArrays_02.png
  5. Click Save.
  6. Start the ETL.

Because these remote arrays will be discovered through their IP address and not through gatekeepers, statistics will not be available.

Running the Extractor Service task

Click here to expand...

 An ETL task can be run manually to retrieve performance metrics. Please note that the steps listed in this section are not mandatory and are only required if the ETL is not already scheduled for execution.To run the ETL task:

  1. Access the TrueSight Capacity Optimization Console.
  2. In the Administration tab, select ETL & System Tasks > ETL Tasks.
  3. Click Button_Run.png to execute the ETL task.
When the execution is complete, the value of the Status column in the ETL tasks table changes to SERVICE_ON.

Setting the collect frequency

Click here to expand...

Contrary to the other Storage ETLs for which the collect frequency is set to 15 minutes, the Storage All-in-One ETL follows the cycle below:

  1. The Discovery: The ETL discovers the different entities and metrics. A discovery is performed:
    • each time the ETL Task is started or restarted
    • when the scheduler is restarted
    • when the discovery failed
    • when the number of collects (23) between two discoveries is reached
  2. The Full Collect: A full collect is triggered right after a discovery to collect data for all the entities previously discovered.
  3. The Partial Collect: Data is partially collected. Resource consuming collects (notably for Storage Volumes) are skipped at this stage. By default, partial collects are performed every 15 minutes. If you do not want partial collects to be performed, you can set the sentry.collect.per.volume.collect property as follows:

    1. Edit the ETL Run Configuration:
      1. In the Administration tab, click ETL & System Tasks > ETL tasks.
      2. Click the link of the ETL task for which you wish to set the collect frequency.
      3. Click the Stop button to stop the service execution; then click the Edit_Icon.pngEdit button available in the  section.
    2. Click the link You can manually edit ETL properties from this page provided at the bottom of the page to display a list of editable options.
    3. Create the sentry.collect.per.volume.collect property and set its value to 0.
    4. Click Save.
    5. Start the ETL.


Lookup details

Storage All-in-One ETL for TrueSight Capacity Optimization defines the following lookup fields:

Storage System

DEFAULT

Storage Pool

DEFAULT

Storage Volume

DEFAULT

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*