Collecting data for additional performance and instance type configuration metrics

Collecting additional CloudWatch metrics

You can collect additional performance metrics from CloudWatch. These metrics are not available for collection by default. To collect these metrics, you must add them in a JSON file. When you configure the ETL to use this JSON file and run the ETL, the database is populated with these additional metrics.

To collect additional performance metrics:

1. Download the attached JSON file.
2. Open the JSON file, and add the details of additional performance metrics that you want to collect.
   1. (Optional) To add custom metrics in BMC Helix Continuous Optimization, define the metric in the Data Warehouse > Datasets & metrics section. For information about how to add a new metric, see Viewing-datasets-and-metrics-by-dataset-and-ETL-module.
   2. After you create the custom metric, add the details of the new metrics in the JSON file along with it's mapping with the AWS metrics. 

3. You can use the following JSON code as an example:

   Sample code for JSON file

   Collecting performance information

   {
      "systemTypeMap" :
   //Amazon Web Services
   // entity type mapping
   {
   "gm:aws": "EC2 Instance",
          "asg:aws": "Auto Scaling group",
          "volume:aws": "EBS Volume"
      "rds:aws": "RDS DB instance"
   },
   "typeConfiguration": [
   {
   "entypenm" : "gm:aws|asg:aws",
   //Metrics for "Instance" or "Autoscale group"
   "mapping"  : [
   //Mapping definition
   {
   "targetMetric": "NET_IN_BYTE_RATE",
   //  metric
   "formula"    : "(NetworkIn)/period",
   //Formula to compute the above metric using the CloudWatch metric
   "sourceMeters":[{
   //Method used for collecting the value from CloudWatch
   "awsMetric"    : "NetworkIn",
   //Amazon Web Services CloudWatch metric
   "statistic"    : "Sum"
   //This can be Average, Sum, SampleCount, Minimum, or Maximum
   }]
   },
   
   {
   //Additional metric mappings
   }
   }
   {
   "entypenm" : "volume:aws",
   //Metrics for "EBS Volume"
   "mapping"  : [
   {
   "targetMetric": "ST_VOLUME_TRANSFER_READ_BYTE_RATE",
   "formula"    :"(VolumeReadBytes)/duration",
   "sourceMeters":[{
   "awsMetric"    : "VolumeReadBytes",
   "statistic"    : "Sum",
   }]
   },
   {
                  "targetMetric":"BYFS_USED_SPACE_PCT",
                  "formula":"(DiskSpaceUtilization)/100",
                  "sourceMeters":[{
                  "awsMetric":"DiskSpaceUtilization",
   "statistic":"Average",
   "dimesionForSubObject":"Filesystem",
   "namespace": "System/Linux"
                   }]
   },
   {
   //Additional metric mappings
   }
   }
   
   {
   "entypenm" : "rds:aws",
   //Metrics for "RDS DB instance"
   "mapping"  : [
   {
   "targetMetric": "CPU_UTIL",
   "formula"    :"(CPUUtilization)/100",
   "sourceMeters":[{
   "awsMetric"    : "CPUUtilization",
   "statistic"    : "Average",
   }]
   },
   {
                  "targetMetric":"DB_TOTAL_FREE_SIZE",
                  "sourceMeters":[{
                  "awsMetric":"FreeStorageSpace",
   "statistic":"Average",
                   }]
   },
   {
   //Additional metric mappings
   }
   }
   
   ]
   }
   
4. Save the file. For example, aws-metric-conf.json.
5. Browse to the file location where you have saved the JSON file that contains the additional metrics and upload the file in BMC Helix Continuous Optimization:
   1. Click Administration > ETL and System Tasks > ETL Tasks.
   2. On the ETL tasks page, click the AWS API ETL that you want to configure. The ETL details are displayed.
   3. Click Edit . The ETL configuration settings are displayed.
   4. Click Advanced > Amazon Web Services Connection.
   5. In the Additional CloudWatch metrics JSON file path field, browse to the location where you have saved the JSON file and upload the file.
   6. Click Save.
6. Run the ETL or after the next scheduled ETL run, verify that the additional metrics are collected. For information about running the ETL and verifying data collection, see Amazon-Web-Services-AWS-API-Extractor.

Collecting additional instance type configuration metrics

The public cloud price catalogs are automatically updated in the product with each release or patch. The Migration Simulation Settings page displays the date when these customized price catalogs for all cloud providers were last updated in the product. For details, see Configuring-the-Migration-Simulation-settings. 

If you want to collect data for any additional instances that are not part of the catalog, you can define such instances along with configuration metrics in a JSON file. When you configure the ETL to use this JSON file and run it, the database is populated with these additional configuration metrics. If the same instance exists in the JSON file and catalog, the instance defined in the JSON file overrides the instance in the catalog. 

To collect additional instance types:

1. Download the sample attached file.
2. Open the file, and complete these steps:
   

   1. Add the details of additional configuration metrics.
      The following JSON file code shows some sample configuration metrics.

      Sample JSON format

      Collecting configuration information

      {
      "version": "20180130",
      "instanceTypeConfiguration" :{
      "m3.medium" :{
      "CPU_NUM": "1",
      "TOTAL_REAL_MEM": "4026531840",
      "DISK_SIZE" : "4000000000"
      },   
      "AWS_INSTANCE_TYPE_NAME" :{                  
      "CO_METRIC_NAME_1" : "VALUE",
      "CO_METRIC_NAME_2" :{
      "CO_METRIC_SUBRESOURCE_2" : "VALUE"
      }
      }
      }
      }

   2. Update the version parameter to add the current date in the yyyymmdd format.

      Information

      Info

      The ETL uses this parameter to verify that the file is the most recent.

   3. Save the file on your computer. For example, aws-instance-type.json.
3. Browse to the file location where you have saved the JSON file that contains the additional instance type metrics and upload the file in BMC Helix Continuous Optimization:
   1. Click Administration > ETL and System Tasks > ETL Tasks.
   2. On the ETL tasks page, click the AWS API ETL that you want to configure. The ETL details are displayed.
   3. Click Edit . The ETL configuration settings are displayed.
   4. Click Advanced > Amazon Web Services Connection.
   5. In the Instance type definition JSON file path field, browse to the location where you have saved the JSON file and upload the file.
   6. Click Save.
4. Run the ETL or after the next scheduled ETL run, verify that the additional metrics are collected. For information about running the ETL and verifying data collection, see Amazon-Web-Services-AWS-API-Extractor.