×
 
 
  •  
$helper.renderConfluenceMacro('{bmc-global-announcement:$space.key}')
BMC Helix Capacity Optimization 20.02 ... Collecting data Sentry ETLs

Storage All-in-One ETL for BMC Helix Capacity Optimization

Use the Storage All-in-One ETL for BMC Helix Continuous Optimization to poll and collect configuration and performance data from several storage systems all at once. The ETL uses the Storage - All-in-One Extractor Service task to collect all the metrics required for performing critical capacity optimization tasks, such as: analysis, trending, simulation and analytic modeling.

Collecting data by using Storage All-in-One ETL forBMC Helix Continuous Optimization

To collect data by using the Storage All-in-One ETL 1.3.00 for BMC Helix Continuous Optimization, perform the following tasks:


I. Complete the preconfiguration tasks.

II. Configure the ETL.

III. Run the ETL.

IV. Post-configuration steps.

Step I. Complete the preconfiguration tasks

The Storage All-in-One ETL supports the following platforms:

  • Ceph Storage
  • Dell EMC Isilon
  • Dell EMC ScaleIO
  • Dell EMC VMAX  (3rd Generation, All Flash or newer) 
  • Dell EMC XtreamIO
  • IBM DS888x
  • NetApp SolidFire
  • Pure Storage REST API

Some platforms can be monitored by both the Storage All-in-One ETL and a manufacturer specific ETL. Please read the following information:

EMC VMAX Systems

For recent systems (3rd Generation and All Flash) the use of the All-in-One ETL is recommended as it is better optimized for these larger systems.  For older VMAX, VNX and Clariion systems, the EMC ETL should be preferred.

Pure and XtremIO Systems

Version 19.11 will be the last version released for these manufacturer specific ETLs.  It is recommended to migrate to the All-in-One ETL for monitoring Pure and XtremIO systems.

Ceph Storage

The Storage All-in-One ETL relies on the Ceph REST API to collect capacity information about the Ceph systems. The connection to the REST API is performed via the HTTP protocol. The ETL first identifies itself to the REST API with a user name and password to retrieve a bearer token from the API and then uses this token for further REST requests.

The default https port for the CEPH Rest API is 8003 (5000 for http).

Note: The ETL has been tested with the version 14.2.5 of the Ceph REST API.

Prerequisites

The Ceph dashboard must be activated with the following command: ceph mgr module enable dashboard.

Dell EMC Isilon

The Storage All-in-One ETL relies on the Dell EMC OneFS REST API to collect capacity information about the Dell EMC Isilon systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

The ETL has been tested on the 8.1 and 8.2 versions of the Dell EMC OneFS REST API.

Dell EMC ScaleIO

The Storage All-in-One ETL leverages the REST API installed as part of the ScaleIO Gateway to collect capacity metrics about Dell EMC ScaleIO systems.

The ScaleIO Gateway:

  • can be installed on the same node as other ScaleIO components if the node has adequate memory to run the Installation Manager (minimum of 2 GB) and any other applications.
  • must have connectivity to the data and management ScaleIO networks.

Java should also be installed on the server where you plan to install the ScaleIO Gateway.


To install the ScaleIO Gateway:

  1. Download from EMC Online Support the installation files needed for your operating system.
  2. Extract the installation files on the Installation Manager (IM) server
  3. For Linux:

    1. Run the following command (all in one line):

      GATEWAY_ADMIN_PASSWORD=<new_GW_admin_password> rpm -U /tmp/EMC-ScaleIO-gateway-1.32-XXX.X.noarch.rpm


      where <new_GW_admin_password> is the password to access the IM

  4. For Windows: 
    1. Run the .msi file, enter (and confirm) a new Gateway Admin password that will be used to access the IM
    2. Prepare disks for storage
  5. Install and configure ScaleIO components using the Installation Manager via the web client:
    1. In a Web browser, enter the following URL https:// <IM_server_IP>/ where <IM_Server_IP> is the IP address of the server where you installed the Installation Manager package
    2. Log in with the default user name (admin) and with the password defined when the Installation Manager was prepared, then click Login
    3. Click Get Started
    4. Browse to where the ScaleIO packages are located, select files, click Open, and click Upload. Click Proceed to Install.
    5. In the Provide Installation Topology screen, click Installation wizard. The Installation Configuration window appears:

    6.  In the Credentials Configuration section, enter and confirm a new MDM Password and the LIA password. In the Topology section, enter server information. Click Start Installation.
    7. Follow the post installation instructions displayed in the wizard.

For more information, refer to the EMC ScaleIO installation guide.

Dell EMC VMAX

The Storage All-in-One ETL relies on the EMC SMI-S provider to collect capacity information about the Dell EMC VMAX systems. The Storage All-in-One ETL supports EMC VMAX3s models only.

Prerequisites

Before configuring the Storage All-in-One ETL for BMC Helix Continuous Optimization, you must ensure that the following requirements are met.

  • BMC Helix Continuous Optimization version must be supported by the ETL
  • The Operating System of the ETL Engine running the ETL task must be a 64-bit version of Linux or Solaris
  • The SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task must be set to at least the following heap size: 2048m.


EMC SMI-S Provider

Before configuring the Storage All-in-One ETL to collect EMC VMAX metrics, you must ensure that the following requirements are met:

  •  (Recommended) Use the SMI-S provider pre-installed in the Management Console. No further action is required.
  •  (Alternate option) Install EMC SMI-S Provider on a different system. In this case, version 8.0.3 or higher is required.  If multiple VMAX systems are added to the same provider, the total number of volumes should not exceed 30,000 (hard limit of the provider is 50,000).

To manage EMC VMAX systems, the EMC SMI-S Provider requires at least one LUN to be mounted from each array to manage. These special LUNs are called "gatekeepers" and are used for the communication between the Symmetrix array and the SMI-S Provider (which does not have an IP address). EMC recommends having between 4 and 6 gatekeeper LUNs for each managed array. Increasing the number of gatekeepers dramatically improves the performance of the EMC SMI-S Provider.

Dell EMC XtreamIO

The Storage All-in-One ETL relies on the EMC XtremIO REST API to collect capacity information about the EMC XtremIO systems. The connection to the REST API is performed via the HTTPS protocol. The ETL identifies itself to the REST API with a user name and password.

IBM DS888x

The Storage All-in-One ETL relies on the IBM System Storage DS Open API, natively embedded within the Hardware Management Console (HMC), through the use of SMI-S provider to collect capacity information about the IBM DS888x systems.

Prerequisites

Before configuring the Storage All-in-One ETL for BMC Helix Continuous Optimization, you must ensure that the following requirements are met.

  • BMC Helix Continuous Optimization version must be supported by the ETL
  • Valid credentials to the HMC must be provided
  • Connection to the HMC must be secured over the 6989 port number
  • The SCHEDULER_HEAP_SIZE environment on the ETL Engine running the ETL task must be set to at least the following heap size: 2048m.

NetApp SolidFire

To collect NetApp SolidFire metrics, the Storage All-in-One ETL makes API requests via HTTPS POST requests to the NetApp SolidFire Element OS API endpoint. HTTP basic authentication is sufficient to access the cluster and discover the system and all the physical nodes connected to the cluster. 

For more information, refer to NetApp SolidFire Element OS API Reference Guide.

Pure Storage REST API

Introduction

The Storage All-in-One ETL relies on the Pure Storage REST API to collect capacity information about the Pure Storage systems. The connection to the REST API is performed via the HTTPS protocol and the ETL identifies itself to the REST API with a user name and password.

Important

If migrating from the standalone Pure Storage REST API ETL, you need to use a username and password for authentication  The token method previously used is no longer supported by Pure or the All-in-One ETL.

Prerequisites

The Pure Storage flash array must supports Pure Storage REST API v1.6 or higher.

Deploying the ETL module

Complete the following steps to deploy the ETL Module:

  1. Navigate to Administration > SYSTEM > Maintenance.
  2. Verify the installation status of the already uploaded packages. If their status is "READY" or "RUNNING", wait for the installation to complete before proceeding.
  3. At the bottom of the Maintenance page, locate the Upload patch or additional package file section.
  4. Click Browse.
  5. Locate and select the ETL installation package.
  6. Click Upload.
  7. Read the Overview and click Next to continue.
  8. Read and accept the End User License Agreement.
  9. Select the server on which the operation must be executed; that is the server on which the package will be saved. Click Proceed to continue.

  10. A successful installation will result in a green line in the Additional Packages table.


In case of unsuccessful outcome (e.g. a yellow WARNING line), click the  icon to inspect the deployment log.


Once the Storage All-in-One ETL is installed, the Storage Engine and Storage Intelligence rows appear in the Additional Packages tab:


Step II. Configure the ETL

You must configure the ETL to connect to Dell Compellent for data collection. ETL configuration includes specifying the basic and optional advancedproperties. While configuring the basic properties is sufficient, you can optionally configure the advanced properties for additional customization.

A. Configuring the basic properties

Some of the basic properties display default values. You can modify these values if required.

To configure the basic properties:

  1. In the Helix Continuous Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL.The Add ETL page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, and ETL additional settings.
  3. On the Run configuration tab, select Storage - All-in-One Extractor Service from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You can edit this field to customize the name.
  4. Click the Entity catalog tab, and select one of the following options:

    • Shared Entity Catalog:

      Select if other ETLs access the same entities that are used by the Dell Compellent ETL.
      • From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
    • Private Entity Catalog: Select if this is the only ETL that extracts data from the Dell Compellent resources.

  5. Click the Connection Settings tab, and configure the following properties:

    PropertyDescription
    Storage Solution

    Storage solution to be configured. Select:

    • Dell EMC ScaleIO REST to collect capacity information about your Dell EMC ScaleIO systems
    • Dell EMC VMAX WBEM to collect capacity information about your Dell EMC VMAX systems
    • IBM DS8880 WBEM to collect capacity information about your IBM DS888x systems
    • NetApp SolidFire REST to collect capacity information about your NetApp SolidFire systems
    • Other Storage Solution to collect capacity information about any other storage solution supported by the Storage-All-in-One ETL. In this case, you will have to manually enter the name of the Si (Storage Intelligence) file to be used.
    Other Storage SolutionName of the Storage Intelligence file (without extension) to be used. This field should only be specified if the Storage Solution property is set to Other Storage Solution.
    Hostname

    Hostname or IP address of the storage solution.

    Port

    Port number used to connect to the storage solution.

    Connect As

    Password to connect to the storage solution.

    Transport Protocol

    Transport protocol used to connect to the storage solution (HTTP or HTTPS).

  6. Click Save. You are returned to the Last run tab under the ETL tasks page.

    1. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
      • OK: the ETL executed without any error in simulation mode.
      • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
      • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
    2. Switch the ETL to production mode: Perform the following steps:
      1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
      2. In the Run configuration table in the ETL details page, click  to edit the active Run configuration.
      3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
      4. Click Save. 

  7. Locate the ETL in the ETL tasks table and click  to run it, or schedule an ETL run. 
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to BMC Helix Continuous Optimizationdatabase. You can see the entity hierarchy in the left Navigation pane, under Workspace.

Step III. Run the ETL

After you configure the ETL, you can run it to collect data. You can run the ETL in the following modes:

A. Simulation mode: Only validates connection to the data source, does not collect data. Use this mode when you want to run the ETL for the first time or after you make any changes to the ETL configuration.

B. Production mode: Collects data from the data source.

A. Running the ETL in the simulation mode

To run the ETL in the simulation mode:

  1. In the Helix Continuous Optimization console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
  2. On the ETL tasks page, click the ETL. The ETL details are displayed.


  1. In the Run configurations table, click Edit to modify the ETL configuration settings.
  2. On the Run configuration tab, ensure that the Execute in simulation mode option is set to Yes, and click Save.
  3. Click Run active configuration. A confirmation message about the ETL run job submission is displayed.
  4. On the ETL tasks page, check the ETL run status in the Last exit column. 
    OK Indicates that the ETL ran without any error. You are ready to run the ETL in the production mode.
  5.  If the ETL run status is Warning, Error, or Failed:
    1. On the ETL tasks page, click  in the last column of the ETL name row.
    2. Check the log and reconfigure the ETL if required.
    3. Run the ETL again.
    4. Repeat these steps until the ETL run status changes to OK.

B. Running the ETL in the production mode

You can run the ETL manually when required or schedule it to run at a specified time.

a. Running the ETL manually

  1. On the ETL tasks page, click the ETL. The ETL details are displayed.
  2. In the Run configurations table, click Edit  to modify the ETL configuration settings. The Edit run configuration page is displayed.
  3. On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
  4. To run the ETL immediately, click Run active configuration. A confirmation message about the ETL run job submission is displayed.
    When the ETL is run, it collects data from the source and transfers it to the BMC Helix Continuous Optimization database.

b. Scheduling the ETL run

By default, the ETL is scheduled to run daily. You can customize this schedule by changing the frequency and period of running the ETL.

To configure the ETL run schedule:

  1. On the ETL tasks page, click the ETL, and click Edit. The ETL details are displayed.

  2. On the Edit task page, do the following, and click Save:

    • Specify a unique name and description for the ETL task.
    • In the Maximum execution time before warning field, specify the duration for which the ETL must run before generating warnings or alerts, if any.
    • Select a predefined or custom frequency for starting the ETL run. The default selection is Predefined.
    • Select the task group and the scheduler to which you want to assign the ETL task.

  3. Click Schedule. A message confirming the scheduling job submission is displayed.
    When the ETL runs as scheduled, it collects data from the source and transfers it to the BMC Helix Continuous Optimization database.

Step IV. Post-configuration steps

Reconciling data

If you used the same Shared Entity Catalog while creating your ETL tasks, data will be automatically reconciled since the ETL tasks are using the same lookup details. If you used a Private Entity Catalog or a different entity catalog, you will have to reconcile data. The procedure is as follows:

  1. Create the new entity catalog:
    1. In the Administration tab, click ETL & System Tasks > Entity catalogs
    2. Click Add entity catalog
    3. Name your entity catalog and eventually provide a description
    4. Click Save.
  2. Configure both ETL tasks to use this entity catalog:
    1. In the Administration tab, click ETL & System Tasks > ETL tasks.Click the link of the ETL task for which you wish to set the entity catalog
    2. Click the Stop button to stop the service execution; then click the  button available in the Run configuration section.
    3. In the Entity Catalog section:
      1. Select the Shared Entity Catalog radio button
      2. From the Sharing with Entity Catalog pull-down list, select the entity catalog previously created (in our example: Storage)
    4. Click Save.
    5. Start the ETL.
    6. Resume the procedure for the second ETL task. 
  3. Perform the reconcile operation:
    1. In the Administration tab, click ETL & System Tasks > Entity catalogs
    2. Click the entity catalog previously created (in our example: EMC)
    3. Click the Reconcile entities button
  4. Perform the entity reconciliation: 

    1. Select the Datasource for which you wish to compare the Lookup fields

    2. Compare entities by Lookup fields and select the look fields
    3. Click Search to determine the entities that can be reconciled
    4. Either select the entities to be reconciled and click Reconcile selected rows or click Reconcile all.

  5. Verify in your workspace that all the selected entities are properly reconciled. Only one instance should be created.

Running the Extractor Service task

 

An ETL task can be run manually to retrieve performance metrics. Please note that the steps listed in this section are not mandatory and are only required if the ETL is not already scheduled for execution.

To run the ETL task:

  1. Access the Helix Continuous Optimization console.
  2. In the Administration tab, select ETL & System Tasks > ETL Tasks.
  3. Click  to execute the ETL task.

When the execution is complete, the value of the Status column in the ETL tasks table changes to SERVICE_ON.

Setting the collect frequency

Contrary to the other Storage ETLs for which the collect frequency is set to 15 minutes, the Storage All-in-One ETL follows the cycle below:

  1. The Discovery: The ETL discovers the different entities and metrics. A discovery is performed:
    • each time the ETL Task is started or restarted
    • when the scheduler is restarted
    • when the discovery failed
    • when the number of collects (23) between two discoveries is reached
  2. The Full Collect: A full collect is triggered right after a discovery to collect data for all the entities previously discovered.
  3. The Partial Collect: Data is partially collected. Resource consuming collects (notably for Storage Volumes) are skipped at this stage. By default, partial collects are performed every 15 minutes. If you do not want partial collects to be performed, you can set the sentry.collect.per.volume.collect property as follows:
    1. Edit the ETL Run Configuration:
      1. In the Administration tab, click ETL & System Tasks > ETL tasks.
      2. Click the link of the ETL task for which you wish to set the collect frequency.
      3. Click the Stop button to stop the service execution; then click the Edit button available in the  section.
    2. Click the link You can manually edit ETL properties from this page provided at the bottom of the page to display a list of editable options.
    3. Create the sentry.collect.per.volume.collect property and set its value to 0.
    4. Click Save.
    5. Start the ETL.


Was this page helpful? Yes No Submitting... Thank you
Last modified by Bharati Poddar on Feb 17, 2020
storage

Comments

Log in or register to comment.

Entities and metrics of Pure Storage REST API ETL for BMC Helix Capacity Optimization
Entities and Metrics of Storage All-in-One ETL for Helix Optimize Capacity Optimization
© Copyright 2019 - 2020 BMC Software, Inc. © Copyright 2019 - 2020 BladeLogic, Inc.
Legal notices

Powered by Atlassian Confluence and Scroll Viewport

© Copyright 2005-2024 BMC Software, Inc. Use of this site signifies your acceptance of BMC’s Terms of Use . BMC, the BMC logo, and other BMC marks are assets of BMC Software, Inc. These trademarks are registered and may be registered in the U.S. and in other countries.