EMC SMI-S Agent ETL for TrueSight Capacity Optimization
Collecting data by using EMC SMI-S Agent ETL for TrueSight Capacity Optimization
To collect data by using the EMC SMI-S Agent ETL 11.5.01 for TrueSight Capacity Optimization, perform the following tasks:
I. Complete the preconfiguration tasks.
II. Configure the ETL.
III. Run the ETL.
Refer to the table below to know which version of the EMC SMI-S provider should be used to collect information about your EMC Disk Arrays and EMC NAS.
A. Selecting the EMC SMI-S Provider
EMC Disk Arrays
EMC Hardware | Supported versions of the EMC SMI-S Provider | ||
---|---|---|---|
v4.6.2.30 | v8.3.0.3 | v9.0.00 | |
CLARiiON (CX, CX3, AX series) | |||
CLARiiON (CX4, AX4 series) | |||
VNX series | |||
Symmetrix DMX (DMX-2, DMX-3, DMX-4 series) | |||
VMAX (10K, 20K, 40K series) | |||
VMAX 3 (100K, 200K, 400K series) |
1 Older versions of EMC CLARiiON have been validated with EMC SMI-S Provider v4.4
EMC NAS
- EMC Celerra supports the embedded SMI-S provider v8.1.0.
- EMC VNX supports the embedded SMI-S provider v7.1.76-4.
Before installing the EMC SMI-S Agent ETL for TrueSight Capacity Optimization, you must ensure that the following requirements are met.
B. Verifying the hardware requirements
CPU | 2 x 64 CPUs (or 2 virtual CPUs) |
Memory | 8GB |
Disk space | 50 GB, 10K rpm class |
Network | 1 Gb/s network adapter |
C. Verifying the software requirements
TrueSight Capacity Optimization | A version supported by the ETL. |
Operating System | A supported 64-bit version of Linux or Solaris. |
D. Verifying the scheduler requirements
The SCHEDULER_HEAP_SIZE environment variable must be set to the following heap size: 2048m.
E. Configuring the EMC SMI-S Provider
To collect information about EMC Celerra storage systems, EMC SMI-S Agent ETL for TrueSight Capacity Optimization leverages the SMI-S Provider embedded within the Control Station. This SMI-S provider is installed by default when the base NAS software is installed in the directory /celerra/wbem on the Control Station.
To enable the embedded SMI-S provider:
Uncomment the following lines in the nas_mcd.cfg file:
# daemon "cim server" # executable "/celerra/wbem/bin/start_cim_server" # optional no # autorestart yes # cmdline "/celerra/wbem" # daemon "cim conf" # executable "/nas/sbin/dirsync" # optional no # autorestart yes # cmdline "-all /celerra/wbem/conf/ /nas/site/ecom_config 360" # daemon "SMISPlugin Log Trimmer" # executable "/nas/sbin/log_trimmer" # optional no # autorestart yes # ioaccess no # cmdline "-n /nas/log/smis/SMISPlugin.log 1000 l 2 h t 4 y " # daemon "SMIS securitylog.txt Log Trimmer" # executable "/nas/sbin/log_trimmer" # optional no # autorestart yes # ioaccess no # cmdline "-n /nas/log/smis/securitylog.txt 1000 l 2 h t 4 y " # daemon "SMIS HTTP_trace.log Log Trimmer" # executable "/nas/sbin/log_trimmer" # optional no # autorestart yes # ioaccess no # cmdline "-n /nas/log/smis/HTTP_trace.log 1000 l 2 h t 4 y " # daemon "SMIS cimomlog.txt Log Trimmer" # executable "/nas/sbin/log_trimmer" # optional no # autorestart yes # ioaccess no # cmdline "-n /nas/log/smis/cimomlog.txt 1000 l 2 h t 4 y "
Restart the NAS services:
/etc/rc.d/init.d/nas stop /etc/rc.d/init.d/nas start
Start the CIM Server:
Usage: start_cim_server <ecom-install-dir> <ecom-install-dir> : /celerra/wbem [root@emc-vnx ~]# /celerra/wbem/bin/start_cim_server /celerra/wbem
To collect capacity information about EMC CLARiiON and VNX systems, you must ensure that the following prerequisites are met:
Installing and configuring the EMC SMI-S Provider
EMC SMI-S Agent ETL for TrueSight Capacity Optimization leverages the EMC SMI-S Provider for CLARiiON and VNX systems that needs:
- to be installed on a Windows or Linux system.
- to have network access to the Storage processors. Monitor credentials are required to connect to the arrays.
The following procedure explains in details how to get, install and configure EMC SMI-S Provider to enable the ETL to efficiently interact with it.
Installing the EMC SMI-S Provider
Before installing the EMC SMI-S Provider, please note that the system hosting the EMC SMI-S Provider requires:
- 2 x 64 CPUs (or 2 virtual CPUs)
- 8 GB of memory
- 50 GB of disk, 10K rpm class
- A 1Gb/s network adapter
- A 4Gb/s dual port HBA
- A supported 64-bit version of Windows Server or Linux
The SMI-S Provider also needs to be configured with a "Heap Size" of 4 GB.
To install the SMI-S Provider:
- Access the EMC’s Online Support website: http://support.emc.com.
- Log on with the User ID and Password provided by EMC’s Online Support. If you do not have an account, you can register directly on EMC’s Online Support website.
- Click Downloads.
- Type SMI-S Provider and click Enter.
- Download the SMI-S Provider executable file. It is recommended to download the x64 package of the version 4.6 available for your operating system.
- Launch the executable file.
- Select the items required by EMC Solutions Enabler with SMI and click Install.
- The EMC Solutions Enabler with SMI X64 welcome page appears. Click Next to begin the installation.
- Select an install directory for EMC Solutions Enabler and click Next to continue. Please note that it is recommended using the default directory.
- Select both Array Provider and Host Provider and click Next to continue.
- In the Service List dialog box, click Next.
- Click Install to begin installing files to your selected folder.
- When the Installation Wizard Complete dialog box opens, click Finish to complete the setup.
- When the installation procedure is complete, update your environment variable for path to include the EMC Solutions Enabler installation directory, which by default is C:\Program Files\EMC\SYMCLI\bin, in order to run the command line utilities from any directory.
Adding CLARiiON/VNX disk arrays to the installed SMI-S provider
Prerequisites
Before adding CLARiiON/VNX disk arrays, please note that EMC states that the EMC SMI-S Provider can manage up to 5 arrays, with up to 10,000 volumes each. Real-life experience shows that the scalability of the EMC SMI-S Provider depends on:
- The number of arrays
- The number of ports in each array
- The number of disks in each array
- The number of volumes
- The number of hosts these volumes are mapped to.
The performance of the EMC SMI-S Provider is affected by the number of client applications performing concurrent data requests. For example, if both TrueSight Operations Management and TrueSight Capacity Optimization are to extract metrics for EMC, the SMI-S Provider has to be sized accordingly, i.e. to handle twice the workload described in this page.
The most important factor is the number of volumes. As a result, an EMC SMI-S Provider can handle up to 50,000 volumes. The maximum number of volumes that can be handled by EMC SMI-S Agent ETL for TrueSight Capacity Optimization are listed below:
Discovery cycle | System collect cycle | Storage collect cycle | Volume collect cycle | Maximum number of volumes |
---|---|---|---|---|
24h (default) | 15 minutes (default) | 1h (default) | 24h (default) | 30,000 |
24h | 1h | 6h | 24h | 80,000 |
24h | 2h | 6h | 24h | 120,000 |
EMC does not recommend managing more than 50,000 volumes per SMI-S Provider.
Adding CLARiiON/VNX disk arrays to the installed SMI-S provider
- Navigate to C:\Program Files\EMC\SYMCLI\storbin and run TestSmiProvider.exe.
- For connection type, type ssl
- For Host, type enter
- Repeat the operation for Port, Username, Password, Log output to console, Log output to file, and Logfile path:
- Type addsys
- Type y (yes)
- Type 1 (Default for CLARiiON/VNX)
- Type xxx.xxx.xxx.xxx (SPA IP Address of the CLARiiON/VNX array)
- Type xxx.xxx.xxx.xxx (SPB IP Address of the CLARiiON/VNX Array)
- Type enter
- Type 2
- Type 2
- Type the username (type a user that has at least a read access on the CLARiiON/VNX Array)
- Type the password (password of the above user)
- Then type enter
The CLARiiON/VNX Array is now added to the installed SMI-S provider.
Adding a user to SMI-S provider
- Access https://<ip-host>:5989/ecomconfig, where <ip-host> is either the IP address of the name of the SMI-S Provider host
- Log on using the username admin and #1Password as password.
- Click Add User and create a user with the role of Monitor. This newly created username can now be used to obtain access to the Array Provider.
Adding authorization information
You must list the information for storage processor A first, then storage processor B. Any time a locally-attached CLARiiON is added to the EMC SMI-S Provider, the EMC CIM Server must first be shut down. Once shut down, the authorization information can be added as described below and the EMC CIM Server must be restarted.
- Shut down the EMC CIM Server (The service name is ECOM.exe and it displays as ECOM in the Services control panel.)
Use the SYMCLI symcfg command, as shown in the following syntax example, to add the username and password for each CLARiiON storage processor:
symcfg authorization add -host HostName -username UserName -password PassWord
Add authorization for CLARiiON storage processor A with IP address <ip1>, username clariion and password clarpass, using the following command:
symcfg auth add -host Hostname -username clariion -password clarpass
- Then, repeat the operation with CLARiiON storage processor B
- Start EMC CIM Server.
Enabling Statistics
To enable statistics on EMC CLARiiON and VNX storage systems, you can either use Navisphere or Unisphere.
Using Navisphere
- In Navisphere, enable Statistics logging:
- Open Navisphere
- Right-click the icon for the storage system
- Click Properties
- Be sure SP A and SP B are in the Storage Processors section
- Click the box for Statistics Logging under the Configuration section
- For locally attached CLARiiONs, check if authorization info is present
- Run the dv command from EMC TestSMIProvider. Make sure all arrays are discovered without any error
- Make sure performance is turned on in the Profile for storage arrays
- Make sure there are instances for the CIM_BlockStorageStatisticalData class
- Run Tek-Tools testProvider or EMC’s testSMIProvider
- Type ein for command
- Type CIM_BlockStorageStatisticalData for Class Name
- If no instances are returned, reset EMC Provider
- Once complete, resume steps 1-3.
If you are using a version older than SMIKit 6.4.2, the EMC SMI-S Provider does not recognize changes in the array when a LUN or RAID Group is created or deleted from Navisphere. It is thus recommended to upgrade your EMC SMI-S Provider to version 6.4.2 or higher to benefit from the auto-synchronization mechanism set by default to one hour after the last update. If you do not upgrade your EMC SMI-S Provider, you will have to run the refsys command from testsmiprovider each time you create or delete LUN or RAID groups from Navisphere.
Using Unisphere
- Log on to the Unisphere console with the administrator’s account
- Select the array from the System List
- Under System Management, click Properties
- In the General tab:
- Check the Enable Statistics Logging box
- Click Apply
- Click OK to finish.
- (Recommended) Use the SMI-S provider pre-installed in the Management Console. No further action is required.
- (Alternate option) Install EMC SMI-S Provider on a different system. In this case, version 8.0.3 or higher is required. See docu59411_Solutions-Enabler-8.0.3-Installation-Guide.pdf for more details.
To manage EMC VMAX and Symmetrix DMX systems, the EMC SMI-S Provider requires at least one LUN to be mounted from each array to manage. These special LUNs are called "gatekeepers" and are used for the communication between the Symmetrix array and the SMI-S Provider (which does not have an IP address). EMC recommends having between 4 and 6 gatekeeper LUNs for each managed array. Increasing the number of gatekeepers dramatically improves the performance of the EMC SMI-S Provider.
F. Deploying the ETL module
Complete the following steps to deploy the ETL Module:
- Navigate to Administration > SYSTEM > Maintenance.
- Verify the installation status of the already uploaded packages. If their status is "READY" or "RUNNING", wait for the installation to complete before proceeding.
- At the bottom of the Maintenance page, locate the Upload patch or additional package file section.
- Click Browse.
- Locate and select the ETL installation package.
- Click Upload.
- Read the Overview and click Next to continue.
- Read and accept the End User License Agreement.
- Select the server on which the operation must be executed; that is the server on which the package will be saved. Click Proceed to continue.
A successful installation will result in a green line in the Additional Packages table.
Note
In case of unsuccessful outcome (e.g. a yellow WARNING line), click theicon to inspect the deployment log.
A. Configuring the basic properties
Some of the basic properties display default values. You can modify these values if required.
To configure the basic properties:
- In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
- On the ETL tasks page, under the Last run tab, click Add > Add ETL. The Add ETL.The Add ETL page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, and ETL additional settings.
On the Run configuration tab, select Storage - EMC SMI-S Agent Extractor Service from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You can edit this field to customize the name.
- Click the Entity catalog tab, and select one of the following options:
Shared Entity Catalog:
Select if other ETLs access the same entities that are used by the EMC SMI-S Agent ETL.- From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
- Private Entity Catalog: Select if this is the only ETL that extracts data from the EMC resources.
Click the Connection Settings tab, and configure the following properties:
Property
Description
Hostname Hostname or IP address of the server where the EMC SMI-S Agent is running.
Port Port number used by the EMC SMI-S Agent (Default: 5989).
Connect As Username to connect to the Unity SMI-S Agent (default: admin).
Password Password to connect to the Unity SMI-S Agent.
Transport Protocol Transport protocol used by the Unity SMI-S Agent (for HTTP, the default port is 80; for HTTPS, the default port is 443). Click Save. You are returned to the Last run tab under the ETL tasks page.
- Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click to run the ETL.
- OK: the ETL executed without any error in simulation mode.
- WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
- ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
- Switch the ETL to production mode: Perform the following steps:
- In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
- In the Run configuration table in the ETL details page, click to edit the active Run configuration.
- In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
- Click Save.
- Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click to run the ETL.
- Locate the ETL in the ETL tasks table and click
After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to TrueSight Capacity Optimization database. You can see the entity hierarchy in the left Navigation pane, under Workspace. to run it, or schedule an ETL run. Click the ETL additional settings tab, and configure the following properties:
Property Description Shares This property allows you to fine-tune the creation of shares entities in the TrueSight Capacity Optimization environment. For more information, refer to the section "Customizing Volumes and Shares Monitoring" below. Volumes This property allows you to disable or enable volumes monitoring and fine-tune the creation of volume entities in the TrueSight Capacity Optimization environment. For more information, refer to the section "Customizing Volumes and Shares Monitoring" below.
(Optional) B. Configuring the advanced properties
You can configure the advanced properties to change the way the ETL works or to collect additional metrics.
Note
Property | Description |
---|---|
Levels up to | By default, extended level metrics are not imported by the ETL. To import them, you will have to customize the collection level and set it to [4] Extended. |
Click Save. The ETL tasks page shows the details of the newly configured EMC SMI-S Agent ETL.
Complete the procedure by performing steps 6 to 8 from the Configuring the basic properties procedure.
After you configure the ETL, you can run it to collect data. You can run the ETL in the following modes:
A. Simulation mode: Only validates connection to the data source, does not collect data. Use this mode when you want to run the ETL for the first time or after you make any changes to the ETL configuration.
B. Production mode: Collects data from the data source.
A. Running the ETL in the simulation mode
To run the ETL in the simulation mode:
- In the TrueSight Capacity Optimization console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
- On the ETL tasks page, click the ETL. The ETL details are displayed.
- In the Run configurations table, click Edit to modify the ETL configuration settings.
- On the Run configuration tab, ensure that the Execute in simulation mode option is set to Yes, and click Save.
- Click Run active configuration. A confirmation message about the ETL run job submission is displayed.
- On the ETL tasks page, check the ETL run status in the Last exit column.
OK Indicates that the ETL ran without any error. You are ready to run the ETL in the production mode. - If the ETL run status is Warning, Error, or Failed:
- On the ETL tasks page, click in the last column of the ETL name row.
- Check the log and reconfigure the ETL if required.
- Run the ETL again.
- Repeat these steps until the ETL run status changes to OK.
B. Running the ETL in the production mode
You can run the ETL manually when required or schedule it to run at a specified time.
a. Running the ETL manually
- On the ETL tasks page, click the ETL. The ETL details are displayed.
- In the Run configurations table, click Edit to modify the ETL configuration settings. The Edit run configuration page is displayed.
- On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
- To run the ETL immediately, click Run active configuration. A confirmation message about the ETL run job submission is displayed.
When the ETL is run, it collects data from the source and transfers it to the TrueSight Capacity Optimization database.
b. Scheduling the ETL run
By default, the ETL is scheduled to run daily. You can customize this schedule by changing the frequency and period of running the ETL.
To configure the ETL run schedule:
- On the ETL tasks page, click the ETL, and click Edit. The ETL details are displayed.
On the Edit task page, do the following, and click Save:
- Specify a unique name and description for the ETL task.
- In the Maximum execution time before warning field, specify the duration for which the ETL must run before generating warnings or alerts, if any.
- Select a predefined or custom frequency for starting the ETL run. The default selection is Predefined.
- Select the task group and the scheduler to which you want to assign the ETL task.
Click Schedule. A message confirming the scheduling job submission is displayed.
When the ETL runs as scheduled, it collects data from the source and transfers it to the TrueSight Capacity Optimization database.
A. Testing the connection to the EMC SMI-S Provider
The ETL relies on the EMC SMI-S Provider to collect information about EMC storage systems. Because some connectivity issues may exist and cause the ETL to fail, it is highly recommended to test the connection to this specific component. The procedure is as follows:
- Download the troubleshooting tool from EPD.
- Run the command java –jar <Troubleshooting_Tool_Name.jar> to launch the tool.
- Enter the information required to connect to the system and click Test Connection.
- Wait for the test to complete.
- If a connectivity issue is detected, an error message will appear. In this case, the issue encountered does not concern the ETL. Check your configuration to diagnose the source of the issue.
- Click Save As to export the connection test results into a txt file.
B. Configuring the discovery of VMAX and Symmetrix remote arrays
VMAX and Symmetrix remote arrays are not discovered by default. To enable the discovery, you will have to add the property discoverRemoteArrays and set it to true. The procedure is as follows:
- Edit the ETL Run Configuration:
- In the Administration tab, click ETL & System Tasks > ETL tasks.
- Click the link of the ETL task for which you wish to set the collect frequency.
- Click the Stop button to stop the service execution; then click the button available in the Run configuration section.
- Click the link You can manually edit ETL properties from this page provided at the bottom of the page to display a list of editable options.
- In the Add new property field, enter discoverRemoteArrays and click Add.
- In the list of properties, locate discoverRemoteArrays and enter true.
- Click Save.
- Start the ETL.
Because these remote arrays will be discovered through their IP address and not through gatekeepers, statistics will not be available.
C. Customizing volumes and shares monitoring
Discovering volumes and/or shares and collecting their performance metrics are resource-intensive actions that can create extra workload on the system. To reduce the system resource consumption, administrators can disable or enable the volumes and/or shares monitoring at will and fine-tune the creation of volume and/or share entities into the TrueSight Capacity Optimization environment.
The following metrics are no longer collected and displayed in the console when volumes monitoring is disabled:
Dataset | Metrics |
---|---|
STOGLB | ST_AVAILABLE_FOR_STORAGE_POOLS (Available Capacity for Storage Pools) ST_AVAILABLE_IN_STORAGE_POOLS (Available Capacity in Storage Pools) ST_CONFIGURED_VOLUME_CAPACITY_PCT (Storage System Configured Capacity Percentage) ST_CONSUMED_CAPACITY (Storage Consumed Capacity) ST_CONSUMED_CAPACITY_PCT (Storage Consumed Capacity Percentage) ST_DEVICECOUNT (Number of Volumes) ST_DEVICE_TOTAL (Device Space) ST_HOSTCNT (Number of Hosts) ST_OVERSUBSCRIBED_CAPACITY (Storage Oversubscribed Capacity) ST_POOL_DEV_COUNT (Number of Storage Pools) ST_POOL_OVERSUBCRIPTION_SITUATION (Pool Oversubcription Situation) ST_POOL_SUBSCRIBED_CAPACITY (Pool Subscribed Capacity) ST_POOL_SUBSCRIBED_CAPACITY_PCT (Pool Subscribed Capacity Percentage) ST_POOL_TYPE (Array Pool Type) ST_PRIMARY_ALLOCATED (Primary Allocated Usable Space) ST_SYSTEM_SUBSCRIBED_CAPACITY (Storage Subscribed Capacity) ST_SYSTEM_VENDOR (Vendor) ST_THINDEV_COUNT (Thin devices count) ST_THINPOOL_CONFIG_TOTAL (Thin Pool Total capacity) ST_THINPOOL_COUNT (Thin pools count) ST_THINPOOL_FREE (Thin Pool Free Space) ST_THINPOOL_USED (Thin Pool Used Space) |
STOHOST | BYHOST_ALLOCATED (Allocated Space by Host) BYHOST_SIZE (by Host Physical Size) |
To customize volumes and shares monitoring
- Edit the EMC SMI-S Agent ETL Run Configuration.
- Expand the ETL additional settings menu.
From the Shares/Volumes pull-down lists, select one of the following options:
Option Result Disabled Volumes/Shares are not discovered.
Volumes/shares-related metrics are not loaded in the console.
Note
Use this configuration if the number of volumes/shares to manage significantly slows down the system performance. Disabling the volumes/shares monitoring prevents the discovery of volumes and consequently does not allow the ETL to collect, save or process their data. Note that the calculation of metrics for other sources may be impacted.Collected for metric computation (Default) All volumes/shares are discovered and their metrics are collected and processed.
Volumes/shares-related metrics are not loaded in the console.
Note
Use this configuration if you wish to collect and process volume/shares-related metrics without loading this information in the console. The collected data will however be used to calculate metrics for other sources, when required. This will reduce the resource consumption of your system.
Collected and Imported All volumes/shares are discovered and their metrics are collected and processed.
Volumes/shares-related metrics are saved in the database, processed and loaded in the console.
Note
Use this configuration if your system is powerful enough to manage all the volumes to monitor.
Host-related metrics are not available when the option Disabled and Collected for metric computation (Default) are selected.
Click Save.
Comments
Log in or register to comment.