Collecting data for additional AWS metrics

You can use the AWS API ETL to collect additional metrics for more in-depth analysis of your AWS infrastructure. To collect these metrics, you must perform some additional steps before configuring and running the AWS API ETL.

Collecting CloudWatch custom metrics 

You can configure your Amazon EC2 instances to export information related to logs, events, and performance to Amazon CloudWatch as custom metrics. The AWS API ETL can collect these CloudWatch custom metrics. You must add these custom metrics to be collected to the monitoring scripts that Amazon EC2 provides. When you run the scripts, these custom metrics become available for collection by the AWS API ETL.

The AWS API ETL does not support collection of the custom metrics that are provided by the CloudWatch agent.

To collect the custom metrics:

1. Log in to your virtual machine on the AWS cloud.
2. Run the monitoring scripts.
   
   • (Linux) Download, install, and configure the monitoring script to add the metrics. For more information, see Enabling metrics on a Linux instances .
     

   • (Windows) Add the custom metrics to a JSON file. For more information, see the following resources:
     

     • To configure the JSON file, see  Configure the JSON file for CloudWatch .
     • To configure CloudWatch, see the Use EC2Config 4.x to Configure CloudWatch section in the  Enabling metrics on Windows instances .

     Ensure that the unit of measure for the metric values is not modified. Otherwise, the custom metrics cannot be mapped to the TrueSight Capacity Optimization metrics.

3. Configure the ETL, run it, and then verify the collection of custom metrics in the Workspace. For more information, see Amazon Web Services - AWS API Extractor.

The following table lists the custom metrics:

Collecting additional CloudWatch performance metrics

You can also collect additional CloudWatch performance metrics. To collect these metrics, you must add them in a JSON file. When you configure the ETL to use this JSON file and run the ETL, the database is populated with these additional CloudWatch performance metrics.

To collect additional CloudWatch metrics:

1. Download the attached JSON file.

2. Open the JSON file, and add the details of additional CloudWatch metrics. You can use the following JSON code as an example for adding metrics:

    Sample code for JSON file

   Collecting performance information

   {
       "systemTypeMap" :
   	//Amazon Web Services to TrueSight Capacity Optimization
   	// entity type mapping
   	{ 
   		"gm:aws"		: "EC2 Instance",
           "asg:aws"		: "Auto Scaling group",
            "volume:aws"	: "EBS Volume"
   	},
   	"typeConfiguration": [
   	{
   		"entypenm" : "gm:aws|asg:aws",
   		//Metrics for "Instance" or "Autoscale group"
   		"mapping"  : [
   		//Mapping definition
   			{
   				"targetMetric"	: "NET_IN_BYTE_RATE",
   				// TrueSight Capacity Optimization metric
   				"formula"    	: "(NetworkIn)/period",
   				//Formula to compute the above metric using the CloudWatch metric
   				"sourceMeters"	:	[{
   				//Method used for collecting the value from CloudWatch
   					"awsMetric"    : "NetworkIn",
   					//Amazon Web Services CloudWatch metric
   					"statistic"    : "Sum"
   					//This can be Average, Sum, SampleCount, Minimum, or Maximum
   									}]
   			},
   			
   			{
   			//Additional metric mappings
   			}
   	}
   	{
   		"entypenm" : "volume:aws",
   		//Metrics for "EBS Volume"
   		"mapping"  : [
   			{
   				"targetMetric"	: "ST_VOLUME_TRANSFER_READ_BYTE_RATE",
   				"formula"    	:"(VolumeReadBytes)/duration",
   				"sourceMeters"	:	[{
   					"awsMetric"    : "VolumeReadBytes",
   					"statistic"    : "Sum",
   									}]
   			},
   			{
                   "targetMetric"	:"BYFS_USED_SPACE_PCT",
                   "formula"		:"(DiskSpaceUtilization)/100",
                   "sourceMeters"	:[{
                   "awsMetric"		:"DiskSpaceUtilization",
   				"statistic"		:"Average",
   				"dimesionForSubObject"	:"Filesystem",
   				"namespace"		: "System/Linux"
                   }]
   			},
   			{
   			//Additional metric mappings
   			}
   	}
   
   						]
   }
   

3. Save the file. For example, aws-metric-perf.json.

4. Upload the file to a directory on the ETL engine server.
5. In the TrueSight Capacity Optimization console, complete these steps:
   
   1. Click Administration > ETL and System Tasks > ETL Tasks.
   2. On the ETL tasks page, click the AWS API ETL that you want to configure. The ETL details are displayed.
   3. Click Edit . The ETL configuration settings are displayed.
   4. Click Advanced > Amazon Web Services Connection.
   5. In the Additional CloudWatch metrics JSON file path field, specify the path where the JSON file is saved.
      The path must include the file name also. For example, /opt/bmc/bco/aws-metric-perf.json
   6. Click Save.
6. Run the ETL or after the next scheduled ETL run, verify that the additional metrics are collected. For information about running the ETL and verifying data collection, see Amazon Web Services - AWS API Extractor.

Collecting additional instance type configuration metrics

You can define additional configuration metrics of an instance that you want to collect in a JSON file. When you configure the ETL to use this JSON file and run it, the database is populated with these additional configuration metrics.

To collect additional configuration metrics:

1. Download the attached file.
2. Open the file, and complete these steps:
   

   1. Add the details of additional configuration metrics.
      The following JSON file code shows some sample configuration metrics.

       Sample JSON format

      Collecting configuration information

      {
      	"version": "20180130",
      	"instanceTypeConfiguration" :{
      		"m3.medium" :{
      			"CPU_NUM"		: "1",
      			"TOTAL_REAL_MEM": "4026531840",
      			"CPU_MHZ"		: "2500",
      			"CPU_MODEL"		: "Intel Xeon E5-2670 v2",
      			"DISK_SIZE" 	: "4000000000"
      		},   
      		"AWS_INSTANCE_TYPE_NAME" :{                  
      			"CO_METRIC_NAME_1" : "VALUE",
      			"CO_METRIC_NAME_2" :{
      				"CO_METRIC_SUBRESOURCE_2" : "VALUE"
      			}
      		}
      	}
      }

   2. Update the version parameter to add the current date in the yyyymmdd format.

      Info

      The ETL uses this parameter to verify that the file is the most recent.

   3. Save the file on your computer. For example, aws-metric-conf.json.

3. Upload the file to a directory on the ETL Engine Server.
4. In the TrueSight Capacity Optimization console, complete these steps:
   
   1. Click Administration > ETL and System Tasks > ETL Tasks.
   2. On the ETL tasks page, click the AWS API ETL that you want to configure. The ETL details are displayed.
   3. Click Edit . The ETL configuration settings are displayed.
   4. Click Advanced > Amazon Web Services Connection.
   5. In the Instance type definition JSON file path field, type the path where the JSON file is saved.
   6. Click Save.
5. Run the ETL or after the next scheduled ETL run, verify that the additional metrics are collected. For information about running the ETL and verifying data collection, see Amazon Web Services - AWS API Extractor.