Moviri Integrator for TrueSight Capacity Optimization - Pivotal Cloud Foundry

"Moviri Integrator for BMC TrueSight Capacity Optimization – PivotalCF" is an additional component of BMC TrueSight Capacity Optimization product. It allows extracting data from the Pivotal Cloud Foundry a unified, multi-cloud platform to manage enterprise applications at scale. Relevant capacity metrics imported into BMC TrueSight Capacity Optimization, which provides advanced analytics over the extracted data in the form of an interactive dashboard, the Pivotal Cloud Foundry View.

The integration supports the extraction of both performance and configuration data across the different components of the Pivotal Cloud Foundry Application Service (PAS) and can configurable via parameters that allow entity filtering and many other settings. The connector can also replicate relationships and logical dependencies among entities such as applications, organizations, and virtual machines.

For the virtual machines under the Pivotal Cloud Foundry Application Service and Isolation Segment, Pivotal CF connector extracts component virtual machines. In the Pivotal CF extractor, we categorized them into two parts, the Diego Node, and PCF Services. Diego Nodes, categorizes the component virtual machines contributed to Pivotal Cloud Foundry Diego component, and other component VMs are counted as PCF Services.

For the applications under the Pivotal Cloud Foundry domain, Pivotal CF connector extracts both system default applications, as long as the customized applications, and arrange them based on the hierarchy of organization, space and applications, in aligned with the Pivotal Cloud Foundry application manager.

This documentation is targeted at BMC TrueSight Capacity Optimization administrators, in charge of configuring and monitoring the integration between BMC TrueSight Capacity Optimization and Pivotal Cloud Foundry.

The latest version of the integrator Pivotal Cloud Foundry is available on EPD. Click the Moviri Integrator for TrueSight Capacity Optimization link. In the Patches tab, select the latest version of TrueSight Capacity Optimization. This ETL is compatible with TrueSight Capacity Optimization 20.02 and onward.

Step I. Complete the preconfiguration tasks

Step II. Configure the ETL

Step III. Run the ETL

Step IV. Verify the data collection

Step I. Complete the Pre-Configuration Tasks

Steps	Details
Check if the required API version is supported	Versions Pivotal CloudFoundry is tested using 2.7
Generate access to Pivotal Cloudfoundry	Pivotal Cloud Foundry Pivotal CloudFoundry uses username/password to authentication and authorization. Upon creation, each user will generate a read-only privileges for querying a set of specific API endpoints. Please use the following example to create the user from Pivotal Cloudfoundry UAA using UAA CLI - uaac. Create a user with log cache read right and cloud foundry API read rights: logs.admin and doppler.firehose for log cache api (click here for more information) and cloud_controller.global_auditor for cloud foundry api (click here for more information). Create the user: First, log in to uaac use admin client and its credentials, note that the admin client needs to have the right to create users: `> uaac target uaa.<your_domian> > uaac token client get admin -s <admin_client_token>` Then, create a user tsco with password `> uaac user add tsco -p <password> --emails tsco@example.com` Then, add scopes "doppler.firehose", "logs.admin" and "cloud_controller.global_auditor", if scopes not exist: `> uaac group add cloud_controller.global_auditor > uaac group add logs.admin > uaac group add doppler.firehose` Finally, add the user "tsco" into the scopes as member: `> uaac member add cloud_controller.global_auditor tsco > uaac member add doppler.firehose tsco > Uaac member add logs.admin tsco` check Pivotal cloudfoundry's documentations for more information about UAA and how to create user: https://docs.pivotal.io/platform/application-service/2-7/uaa/uaa-user-management.html Test the user's access right: In order to get access to Log cache API and cloud foundry API, the uaa API will generate a authorization token for the user "tsco" for accessing. usually, the token will expire very soon. Therefore, test the user's access right should be executed immediately after call the UAA API to generate token: The following curl command shows how to generate a token based on the user's id and password: `> curl -v -k -XPOST -H"Application/json" -u "cf:" --data "username=tsco&password=<password>&grant_type=password" https://uaa.<your-domain>/oauth/token {"access_token":"<token>", "token_type":"bearer", ....}` Now, Use the token to test the access for Log cache api: `> curl "https://log-cache.<your-domain>/api/v1/read/rep" -k -H "Authorization: Bearer <token>" {"envelopes":{"batch":[{"timestamp":"1578235197085691442","source_id"...}` Now, Use the same token to test the access for cloud foundry api: `curl "https://api.<your-domain>/v2/apps" -H "Authorization: Bearer <token>" { "total_results": 18, "total_pages": 1, "prev_url": null, "next_url": null, "resources": [ { "metadata": { "guid": "..... }`
(Optional) Configure to use CID	CID is extracted from Opsman API, If want to reconsile PCF Services and PCF Nodes with VMWare vms and AWS vms, an Opsman API account wit minimal restricted view privelldge should be configured: To generate the user with read rights for Ops manager API: 1. using uaac tool to connect to ops manager FQDN *uaac target "https://<OPSMAN_FQDN>/uaa" --skip-ssl-validation* 2. login as admin or any users with rights to create users *uaac token owner get* Client ID: opsman Client secret:<press enter> User name: <username> Password: <password> ----------------------------------- Unknown key: Max-Age = 172800 Successfully fetched token via owner password grant. Target: https://opsman.pcf.moviri-integrations.com/uaa Context: admin, from client opsman ------------------------------------- 3. Create a user with a password, for example opstsco *uaac user add opstsco -p <enter a password for this user> --emails example@example.com* ------------------------------------- user account successfully added ------------------------------------- 4. Add opsman.restrcted_view groups if not exists uaac group add opsman.restricted_view 5. Add the user opstsco to the group *uaac member add opsman.restricted_view opstsco* 6. Run the following curl command to get the access token for user opstsco, password is the one we created on Step3 above; Save the access token from the response Or for testing purposes, just run uaac command to get the access token: *uaac token client get opstsco* *uaac context* Alternatively to step 3,4,5,6 we can use client Id and secret, following the steps: The same as step 1 above The same as step 2 above Run the following command to create a client id and password uaac client add <clientID> --authorized_grant_types client_credentials --authorities opsman.restricted_view --secret <ClientSecret> Run the following command to test if there’s access token *curl -XPOST -H "Application/json" --data "client_id=<clientID>&client_secret=<clientSecret>&grant_type=client_credentials" "https://opsman.pcf.moviri-integrations.com/uaa/oauth/token" -k* Test the curl commands that is used in the connector: 1. Run the following curl command, use the access token grabbed from step1 *curl "https://<OPSMAN_FQDN>/api/v0/deployed/products" -H "Authorization: Bearer <Access Token>" -k* 2. Run the following curl command, use the access token grabbed from step1, Product_Id here is the deployment guid. *curl "https://<OPSMAN_FQDN>/api/v0/deployed/products/<Product_Id>/status" -H "Authorization: Bearer <Access Token>" -k*

Step II. Configure the ETL

In the console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
On the ETL tasks page, click Add > Add ETL. The Add ETL page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, Connection, and PCF Connection, PCF Extraction Filters
On the Run Configuration tab, select Moviri - PivotalCF Extractor from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You edit this field to customize the name.

Click the Entity catalog tab, and select one of the following options:

Shared Entity Catalog:
- From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
Private Entity Catalog: Select if this is the only ETL that extracts data from the Pivotal CF resources.

Click the Connection tab, and configure the following properties:

Property Name	Value Type	Required?	Default	Advanced	Description
PCF Connection
Pivotal CloudFoundry domain	String	yes		no	The domain of your Pivotal CLoud Foundry. Eg: examplepcf.com
Pivotal CloudFoundry username	String	yes		no	The username of the user has the correct access right
Pivotal CloudFoundry password	String	yes		no	The password of the user has the correct access right
Proxy server host	String	no		no	If proxy is used, filling this proxy server's host. Only "http" is supported.
Proxy server port	Number	no		no	If proxy is used, filling this proxy server's port number.
Proxy server user	String	no		no	if proxy is used, filling this proxy server's user name.
Proxy server password	String	no		no	If proxy is used, filling this proxy server's user password
PCF Extraction Filter
Data resolution	Option	yes	1 hour	no	Data resolution of the improted data. Options are 5 minutes, 15 minutes, and 1 hour
Select only Applications on the following organizations, use GUID (; separated)	text	no		no	Application whitelist. Show only applications from the following organizations. Use organization GUID. Separate each organizations GUID using ";".
Select only Applications on the following organizations, use name (; separated)	text	no		yes	Application whitelist. Show only applications from the following organizations. Use organization Name. Separate each organizations name using ";".
Select only Applications in the following spaces, use GUID (; separated)	text	no		no	Application whitelist. Show only applications from the following spaces. Use space GUID. separate each space GUID using ";".
Select only VMs in the following deployments (; separated)	text	no		no	Virtual machine whitelist. Show only virtual machine's from the following deployments, separate each deployment using ";".
Exclude any Applications on the following organizations, use GUID (; separated)	text	no		no	Application blacklist. Exclude only applications from the following organizations. Use organization GUID. Separate each organizations GUID using ";".
Exclude any Applications on the following organizations, use name(; separated)	text	no		yes	Application blacklist. Exclude only applications from the following organizations. Use organization Name. Separate each organizations name using ";".
Exclude any Applications in the following spaces, use GUID (; separated)	text	no		no	Application blacklist. Exclude only applications from the following spaces. Use Space GUID. Separate each space GUID using ";".
Exclude any VMs in the following deployments (; separated)	text	no		no	Virtual machine blacklist. Exclude only virtual machines from the following deployments. Separate each deployment using ";".
Import vm list, leave empty if import all	selection	no	empty	yes	Select which types of virtual machines you want to import. If want to import all, leave this field empty.
Do you want to reconcile vms with other entities using CID?	Option	no	no	no	If selected to "yes", the ETL will reconsile using cid to reconcile PCF nodes and PCF Services, and importing them as "generic"
FQDN of opsman API (format http/https://host:[port])	String	no		yes	Opsman API FQDN Address, this property will be visible if set "yes" to "Do you want to reconcile vms with other entities using CID?"
Token generating method (User or Client)	Selection	no	ClientId	yes	Select password or clientid as authentication method. This property will be visible if set "Yes" to Do you want to reconcile vms with other entities using CID?
Username of the Opsman API user	String	no		yes	Username of the Opsman API user if the token generation method is set to "user"
Password of the Opsman API password	String	no		yes	Password of the Opsman API user if the token generation method is set to "user"
ClientId of the Opsman API Client	String	No		yes	Client Id of the Opsman API if the token generation method is set to "client credential"
Client Credential of the Opsman API Client	String	No		yes	Client credential of the Opsman API if the token generation method is set to "client credential"

Hidden properties: Pivotal CloudFoundry ETL provides extra hidden properties and they can be changed manually from edit configuration page.

To change the default polling period (1 minute ) for Applications, Organizations, and Spaces data, set a hidden property "extract.pcf.app.serve.interval" and set the value to desired seconds. For example, if want to change default polling interval from 1 minute to 5 minute, set the value to 300.
To change the default polling period (1 minute ) for Bosh deployment and its nodes data, set a hidden property "extract.pcf.bosh.serve.interval" and set the value to desired seconds. For example, if want to change default polling interval from 1 minute to 5 minute, set the value to 300.

All the other generic properties are documented here.

The ETL tasks page shows the details of the newly configured Pivotal Cloudfoundry ETL.

Step III. Run the ETL

After you configure the ETL, you can run it to collect data. You can run the ETL in the following modes:

A. Simulation mode: Only validates connection to the data source, does not collect data. Use this mode when you want to run the ETL for the first time or after you make any changes to the ETL configuration.

B. Production mode: Collects data from the data source.

Running the ETL in simulation mode

To run the ETL in the simulation mode:

In the console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
On the ETL tasks page, click the ETL. The ETL details are displayed.
In the Run configurations table, click Edit to modify the ETL configuration settings.
On the Run configuration tab, ensure that the Execute in simulation mode option is set to Yes, and click Save.
Click Run active configuration. A confirmation message about the ETL run job submission is displayed.
On the ETL tasks page, check the ETL run status in the Last exit column.
OK Indicates that the ETL ran without any error. You are ready to run the ETL in the production mode.
If the ETL run status is Warning, Error, or Failed:
1. On the ETL tasks page, click in the last column of the ETL name row.
2. Check the log and reconfigure the ETL if required.
3. Run the ETL again.
4. Repeat these steps until the ETL run status changes to OK.

Running the ETL in the production mode

You can run the ETL manually when required or schedule it to run at a specified time. As of version 20.02.00.005 the integration delays the hierarchy to delay for six hours helping to reduce the load on the loader step.

Running the ETL manually

On the ETL tasks page, click the ETL. The ETL details are displayed.
In the Run configurations table, click Edit to modify the ETL configuration settings. The Edit run configuration page is displayed.
On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
To run the ETL immediately, click Run active configuration. A confirmation message about the ETL run job submission is displayed.
When the ETL is run, it collects data from the source and transfers it to the database.

Here's screenshot of ETL page after set it up and run completely.

Step IV. Verify Data Collection

Verify that the ETL ran successfully and check whether the Pivotal CF data is refreshed in the Workspace.

To verify whether the ETL ran successfully:

In the console, click Administration > ETL and System Tasks > ETL tasks.
In the Last exec time column corresponding to the ETL name, verify that the current date and time are displayed.

To verify that the Pivotal CF data is refreshed:

In the console, click Workspace.
Expand (Domain name) > Systems > PivotalCF> Instances.
In the left pane, verify that the hierarchy displays the new and updated Pivotal CF instances.
Click a Pivotal CF entity, and click the Metrics tab in the right pane.
Check if the Last Activity column in the Configuration metrics and Performance metrics tables displays the current date.

Pivotal CF Workspace

Details

Entities

Show Entities

Supported Entities

We extracting two parts from the same Pivotal Cloud Foundry domain, component virtual machines and applications. The component virtual machine parts organized by Pivotal Cloud Foundry foundation (deployment), and are categorized into two parts - Diego Nodes (Pivotal Cloud Foundry Diego component), and Services (other component VMs). The applications components are aligned and structured according to Pivotal Cloud Foundry application management.

PCF - Domain: PCF domain

PCF - Organization: PCF Ogranizations of each domain

PCF - Space: PCF spaces of each oragnziation

PCF - Applicaiton: PCF applications under each space

PCF - Foundation: PCF Application Service (PAS) deployment for each domain and Isolation Segment deployment for each domain

PCF - Diego Node: PCF's diego component virtual machines, including:

diego cell
diego brain
diego database (bbs)
router
tcp-router
isolated diego cell
isolated router

PCF - Service: Other PCF's component virtual machines besides the diego node, including but not limit to:

backup_restore
clock_global
cloud_controller

doppler
loggregator_traffic_controller
mysql
nats
uaa
isolated haproxy

Hierarchy

Show Hierarchy

The connector can replicate relationships and logical dependencies among these entities as they are found configured within the Pivotal Cloud Foundry domain.

hierarchy change

By default the hierarchy will be updated every 6 hours, starting 0:00 of the day.

The integration uses the following hierarchy:

A PCF - Domain is attached to the root hierarchy
Each PCF - Domain contains PCF - Organizations and PCF - Foundations
Each PCF - Organization contains PCF - Space, and each PCF - Space contains PCF - Applications
Each PCF - Foundations contains PCF - Diego Nodes and PCF - Services

The following image shows a sample hierarchy.

Configuration and Performance Metrics Mapping

Metric Mapping

Data Source	Data Source Entity	Data Source Metric	TSCO Entity	TSCO Metric	Scaling Factor
CloudFoundry api	organizations	mem_limit	PCF Organization	MEM_LIMIT	1024*1024
CloudFoundry api	organizations	total_routes	PCF Organization	ROUTES	1
CloudFoundry api	organizations	total_services	PCF Organization	SERVICE_NUM	1
CloudFoundry api	apps/stats	mem_quota	PCF Application	MEM_ALLOCATION	1
CloudFoundry api	apps/stats	disk_quota	PCF Application	DISK_ALLOCATION	1
CloudFoundry api	apps/stats	mem	PCF Application	MEM_CONSUMED	1
CloudFoundry api	apps/stats	disk	PCF Application	DISK_USED_SPACE	1
CloudFoundry api	apps/stats	cpu	PCF Application	CPU_UTIL	1
CloudFoundry api	apps	state	PCF Application	APP_STATUS	N/A
CloudFoundry api	apps		PCF Application	APP_INSTANCES	1
Log Cache api	VMs	system.cpu.sys	PCF Diego Node, PCF Service	CPU_UTIL_SYSTEM	0.01
Log Cache api	VMs	system.cpu.user	PCF Diego Node, PCF Service	CPU_UTIL_USER	0.01
Log Cache api	VMs	system.cpu.wait	PCF Diego Node, PCF Service	CPU_UTIL_WAIT	0.01
Log Cache api	diego_cell	CapacityTotalMemory	PCF Diego Node - diego_cell	TOTAL_REAL_MEM	1024*1024
Log Cache api	diego_cell	CapacityTotalContainers	PCF Diego Node - diego_cell	CONTAINER_MAX	1
Log Cache api	diego_cell	CapacityAllocatedDisk	PCF Diego Node - diego_cell	DISK_ALLOCATION	1024*1024
Log Cache api	diego_cell	CapacityTotalDisk	PCF Diego Node - diego_cell	DISK_SIZE	1024*1024
Log Cache api	diego_cell	ContainerCount	PCF Diego Node - diego_cell	CONTAINER_NUM	1
Log Cache api	diego_cell	CapacityAllocatedMemory	PCF Diego Node - diego_cell	MEM_ALLOCATION	1024*1024
Log Cache api	diego_cell	CapacityRemainingDisk	PCF Diego Node - diego_cell	DISK_FREE	1024*1024
Log Cache api	diego_cell	CapacityRemainingMemory	PCF Diego Node - diego_cell	MEM_FREE	1024*1024
Log Cache api	diego_cell	ContainerUsageDisk	PCF Diego Node - diego_cell	DISK_USED_SPACE	1024*1024
Log Cache api	diego_cell, diego_database, diego_brain, tcp_router, gorouter	job	PCF Diego Node	DIEGO_NODE_TYPE	N/A
Log Cache api	VMs	system_cpu_idle	PCF Diego Node, PCF Service	CPU_UTIL_IDLE	0.01
Log Cache api	VMs	system_disk_ephemeral_read_bytes	PCF Diego Node, PCF Service	DISK_EPHEMERAL_READ_RATE	1
Log Cache api	VMs	system_disk_ephemeral_read_time	PCF Diego Node, PCF Service	DISK_EPHEMERAL_READ_RESPONSE_TIME	1
Log Cache api	VMs	system_disk_ephemeral_write_bytes	PCF Diego Node, PCF Service	DISK_EPHEMERAL_WRITE_RATE	1
Log Cache api	VMs	system_disk_ephemeral_write_time	PCF Diego Node, PCF Service	DISK_EPHEMERAL_WRITE_RESPONSE_TIME	1
Log Cache api	VMs	system_disk_system_read_bytes	PCF Diego Node, PCF Service	DISK_READ_RATE	1
Log Cache api	VMs	system_disk_system_read_time	PCF Diego Node, PCF Service	DISK_READ_RESPONSE_TIME	1
Log Cache api	VMs	system_disk_system_io_time	PCF Diego Node, PCF Service	DISK_TOTAL_LATENCY	1
Log Cache api	VMs	system.disk.system.inode_percent	PCF Diego Node, PCF Service	DISK_USED_INODES_PCT	0.01
Log Cache api	VMs	system.disk.system.percent	PCF Diego Node, PCF Service	DISK_USED_SPACE_PCT	0.01
Log Cache api	VMs	system_disk_system_write_time	PCF Diego Node, PCF Service	DISK_WRITE_RESPONSE_TIME	1
Log Cache api	VMs	system.disk.ephemeral.inode_percent	PCF Diego Node, PCF Service	DISK_EPHEMERAL_USED_INODES_PCT	0.01
Log Cache api	VMs	system.disk.ephemeral.percent	PCF Diego Node, PCF Service	DISK_EPHEMERAL_USED_PCT	0.01
Log Cache api	VMs	go_gc_duration_seconds	PCF Diego Node, PCF Service	GC_TIME	1
Log Cache api	VMs	numGoRoutines	PCF Diego Node, PCF Service	GOROUTINES	1
Log Cache api	VMs	HEAPMEM_FREE	PCF Diego Node, PCF Service	HEAPMEM_FREE	1
Log Cache api	VMs	go_memstats_heap_inuse_bytes	PCF Diego Node, PCF Service	HEAPMEM_USED	1
Log Cache api	VMs	system.mem.kb	PCF Diego Node, PCF Service	MEM_CONSUMED	1024
Log Cache api	VMs	system.mem.percent	PCF Diego Node, PCF Service	MEM_UTIL	0.01
Log Cache api	VMs	process_virtual_memory_bytes	PCF Diego Node, PCF Service	MEM_VIRTUAL_ACTIVE	1
Log Cache api	VMs	process_virtual_memory_max_bytes	PCF Diego Node, PCF Service	MEM_VIRTUAL_LIMIT	1
Log Cache api	VMs	system_network_bytes_received	PCF Diego Node, PCF Service	NET_IN_BYTE_RATE	1
Log Cache api	VMs	system_network_error_in	PCF Diego Node, PCF Service	NET_IN_ERROR_RATE	1
Log Cache api	VMs	system_network_packets_received	PCF Diego Node, PCF Service	NET_IN_PKT_RATE	1
Log Cache api	VMs	system_network_bytes_sent	PCF Diego Node, PCF Service	NET_OUT_BYTE_RATE	1
Log Cache api	VMs	system_network_error_out	PCF Diego Node, PCF Service	NET_OUT_ERROR_RATE	1
Log Cache api	VMs	system_network_packets_sent	PCF Diego Node, PCF Service	NET_OUT_PKT_RATE	1
Log Cache api	VMs	process_open_fds	PCF Diego Node, PCF Service	OPEN_FILES	1
Log Cache api	diego_cell, diego_database	RequestsCancelled	PCF Diego Node - diego_cell, diego_database	REQ_CANCELLED	1
Log Cache api	diego_cell, diego_database	RequestsFailed	PCF Diego Node - diego_cell, diego_database	REQ_FAILED	1
Log Cache api	diego_cell, diego_database	RequestsInFlight	PCF Diego Node - diego_cell, diego_database	REQ_FLIGHT	1
Log Cache api	diego_cell, diego_database	RequestsStarted	PCF Diego Node - diego_cell, diego_database	REQ_STARTED	1
Log Cache api	diego_cell, diego_database	RequestsSucceeded	PCF Diego Node - diego_cell, diego_database	REQ_SUCCEEDED	1
Log Cache api	VMs	go_threads	PCF Diego Node, PCF Service	THREAD_COUNT	1
Log Cache api	diego_database	DBQueryDurationMax	PCF Diego Node - diego_database	DB_QUERY_MAX	1
Log Cache api	diego_database	ActiveLocks	PCF Diego Node- diego_database	DB_LOCKS	1
Log Cache api	diego_database	LocksExpired	PCF Diego Node- diego_database	DB_LOCKS_EXPIRED	1
Log Cache api	diego_database	DBQueriesFailed	PCF Diego Node- diego_database	DB_QUERY_FAILED	1
Log Cache api	diego_database	DBQueriesInFlight	PCF Diego Node- diego_database	DB_QUERY_FLIGHT	1
Log Cache api	diego_database	DBQueriesSucceeded	PCF Diego Node- diego_database	DB_QUERY_SUCCEEDED	1
Log Cache api	diego_database	DBQueriesTotal	PCF Diego Node- diego_database	DB_QUERY_TOTAL	1
Log Cache api	diego_database	DBOpenConnections	PCF Diego Node- diego_database	DB_SESSIONS	1
Log Cache api	diego_database	DBWaitDuration	PCF Diego Node- diego_database	DB_WAIT_DURATION	1
Log Cache api	diego_database	ConvergenceLRPDuration	PCF Diego Node- diego_database	LRP_CONVERGE_DURATION	1
Log Cache api	diego_database	LRPsDesired	PCF Diego Node- diego_database	LRP_MAX	1
Log Cache api	diego_database	LRPsRunning	PCF Diego Node- diego_database	LRP_RUNNING	1
Log Cache api	diego_database	LRPsMissing	PCF Diego Node- diego_database	LRP_MISSING	1
Log Cache api	diego_database	LRPsExtra	PCF Diego Node- diego_database	LRP_EXTRA	1
Log Cache api	diego_database	LRPsClaimed	PCF Diego Node- diego_database	LRP_CLAIMED	1
Log Cache api	diego_database	TasksCompleted	PCF Diego Node- diego_database	TASK_COMPLETED	1
Log Cache api	diego_database	TasksPending	PCF Diego Node- diego_database	TASK_PENDING	1
Log Cache api	diego_database	TasksResolving	PCF Diego Node- diego_database	TASK_RESOLVING	1
Log Cache api	diego_database	TasksRunning	PCF Diego Node- diego_database	TASK_RUNNING	1
Log Cache api	diego_database	TasksFailed	PCF Diego Node- diego_database	TASK_FAILED	1
Log Cache api	diego_database	TasksStarted	PCF Diego Node- diego_database	TASK_STARTED	1
Log Cache api	diego_database	TasksSucceeded	PCF Diego Node- diego_database	TASK_SUCCEEDED	1
Log Cache api	router	total_dropped_messages	PCF Diego Node- router	DROPPED_MESSAGES_NUM	1
Log Cache api	uaa	vitals.jvm.non-heap.committed	PCF Service - uaa	NONHEAPMEM_COMMITTED	1
Log Cache api	uaa	requests.global.completed.count	PCF Service - uaa	RESPONSE_TIME	1
Log Cache api	uaa	vitals.jvm.heap.max	PCF Service - uaa	HEAPMEM_MAX	1
Log Cache api	tcprouter	numCPUs	PCF Diego Node - tcprouter	CPU_NUM	1

Derived Metrics

Data Source	Data Source Entity	Data Source Metric	Target Entity	Target Metric	Aggregation type
TSCO	PCF Application	MEM_CONSUMED	PCF Organization	MEM_CONSUMED	SUM
TSCO	PCF Application	DISK_USED_SPACE	PCF Organization	DISK_USED_SPACE	SUM
TSCO	PCF Applicaiton	CPU_UTIL	PCF Organization	CPU_UTIL	SUM
TSCO	PCF Application	APP_INSTANCES	PCF Organization	APP_INSTANCES	SUM
TSCO	PCF Application	MEM_CONSUMED	PCF SPACE	MEM_CONSUMED	SUM
TSCO	PCF Application	DISK_USED_SPACE	PCF SPACE	DISK_USED_SPACE	SUM
TSCO	PCF Applicaiton	CPU_UTIL	PCF SPACE	CPU_UTIL	SUM
TSCO	PCF Application	APP_INSTANCES	PCF SPACE	APP_INSTANCES	SUM
TSCO	PCF Application	MEM_USED/TOTAL_REAL_MEM	PCF Application	MEM_UTIL	AVG
TSCO	PCF Application	DISK_USED_SPACE/DISK_SIZE	PCF Application	DISK_USED_SPACE_PCT	AVG
TSCO	PCF Diego Node, PCF Service	SWAP_SPACE_USED/SWAP_SPACE_UTIL	PCF Diego Node, PCF Service	SWAP_SPACE_TOT	AVG
TSCO	PCF Diego Node, PCF Service	MEM_CONSUMED/MEM_UTIL	PCF Diego Node, PCF Service	TOTAL_REAL_MEM	AVG
TSCO	PCF Deigo Node, PCF Service	1- CPU_UTIL_IDLE	PCF Diego Node, PCF Service	CPU_UTIL	AVG
TSCO	PCF Diego Node, PCF Service	HEAPMEM_USED/(HEAPMEM_FREE+ HEAPMEM_USED)	PCF Diego Node, PCF Service	HEAPMEM_UTIL	AVG
TSCO	PCF Diego Node - diego_cell	SWAP_SPACE_USED/SWAP_SPACE_UTIL	PCF Foundation	SWAP_SPACE_TOT	SUM
TSCO	PCF Diego Node - diego_cell	MEM_CONSUMED/MEM_UTIL	PCF Foundation	TOTAL_REAL_MEM	SUM
TSCO	PCF Diego Node - diego_cell	1- CPU_UTIL_IDLE	PCF Foundation	CPU_UTIL	SUM
TSCO	PCF Diego Node - diego_cell	HEAPMEM_USED/(HEAPMEM_FREE+ HEAPMEM_USED)	PCF Foundation	HEAPMEM_UTIL	SUM
TSCO	PCF Diego Node, PCF Service		PCF Foundation	NODE_NUM	COUNT
TSCO	PCF Diego Node		PCF Foundation	DIEGO_CELLS	COUNT
TSCO	PCF Service		PCF Foundation	DIEGO_NODES	COUNT
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_SYSTEM	PCF Foundation	CPU_UTIL_SYSTEM	SUM
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_USER	PCF Foundation	CPU_UTIL_USER	SUM
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_WAIT	PCF Foundation	CPU_UTIL_WAIT	SUM
TSCO	PCF Diego Node - diego_cell	TOTAL_REAL_MEM	PCF Foundation	TOTAL_REAL_MEM	SUM
TSCO	PCF Diego Node - diego_cell	CONTAINER_MAX	PCF Foundation	CONTAINER_MAX	SUM
TSCO	PCF Diego Node - diego_cell	DISK_ALLOCATION	PCF Foundation	DISK_ALLOCATION	SUM
TSCO	PCF Diego Node - diego_cell	DISK_SIZE	PCF Foundation	DISK_SIZE	SUM
TSCO	PCF Diego Node - diego_cell	CONTAINER_NUM	PCF Foundation	CONTAINER_NUM	SUM
TSCO	PCF Diego Node - diego_cell	MEM_ALLOCATION	PCF Foundation	MEM_ALLOCATION	SUM
TSCO	PCF Diego Node - diego_cell	DISK_FREE	PCF Foundation	DISK_FREE	SUM
TSCO	PCF Diego Node - diego_cell	MEM_FREE	PCF Foundation	MEM_FREE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_USED_SPACE	PCF Foundation	DISK_USED_SPACE	SUM
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_IDLE	PCF Foundation	CPU_UTIL_IDLE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_READ_RATE	PCF Foundation	DISK_EPHEMERAL_READ_RATE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_READ_RESPONSE_TIME	PCF Foundation	DISK_EPHEMERAL_READ_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_WRITE_RATE	PCF Foundation	DISK_EPHEMERAL_WRITE_RATE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_WRITE_RESPONSE_TIME	PCF Foundation	DISK_EPHEMERAL_WRITE_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_READ_RATE	PCF Foundation	DISK_READ_RATE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_READ_RESPONSE_TIME	PCF Foundation	DISK_READ_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_TOTAL_LATENCY	PCF Foundation	DISK_TOTAL_LATENCY	SUM
TSCO	PCF Diego Node - diego_cell	DISK_USED_INODES_PCT	PCF Foundation	DISK_USED_INODES_PCT	SUM
TSCO	PCF Diego Node - diego_cell	DISK_USED_SPACE_PCT	PCF Foundation	DISK_USED_SPACE_PCT	SUM
TSCO	PCF Diego Node - diego_cell	DISK_WRITE_RESPONSE_TIME	PCF Foundation	DISK_WRITE_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_USED_INODES_PCT	PCF Foundation	DISK_EPHEMERAL_USED_INODES_PCT	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_USED_PCT	PCF Foundation	DISK_EPHEMERAL_USED_PCT	SUM
TSCO	PCF Diego Node - diego_cell	GC_TIME	PCF Foundation	GC_TIME	SUM
TSCO	PCF Diego Node - diego_cell	GOROUTINES	PCF Foundation	GOROUTINES	SUM
TSCO	PCF Diego Node - diego_cell	HEAPMEM_FREE	PCF Foundation	HEAPMEM_FREE	SUM
TSCO	PCF Diego Node - diego_cell	HEAPMEM_USED	PCF Foundation	HEAPMEM_USED	SUM
TSCO	PCF Diego Node - diego_cell	MEM_CONSUMED	PCF Foundation	MEM_CONSUMED	SUM
TSCO	PCF Diego Node - diego_cell	MEM_UTIL	PCF Foundation	MEM_UTIL	SUM
TSCO	PCF Diego Node - diego_cell	MEM_VIRTUAL_ACTIVE	PCF Foundation	MEM_VIRTUAL_ACTIVE	SUM
TSCO	PCF Diego Node - diego_cell	MEM_VIRTUAL_LIMIT	PCF Foundation	MEM_VIRTUAL_LIMIT	SUM
TSCO	PCF Diego Node - diego_cell	NET_IN_BYTE_RATE	PCF Foundation	NET_IN_BYTE_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_IN_ERROR_RATE	PCF Foundation	NET_IN_ERROR_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_IN_PKT_RATE	PCF Foundation	NET_IN_PKT_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_OUT_BYTE_RATE	PCF Foundation	NET_OUT_BYTE_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_OUT_ERROR_RATE	PCF Foundation	NET_OUT_ERROR_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_OUT_PKT_RATE	PCF Foundation	NET_OUT_PKT_RATE	SUM
TSCO	PCF Diego Node - diego_cell	OPEN_FILES	PCF Foundation	OPEN_FILES	SUM
TSCO	PCF Diego Node - diego_cell	REQ_CANCELLED	PCF Foundation	REQ_CANCELLED	SUM
TSCO	PCF Diego Node - diego_cell	REQ_FAILED	PCF Foundation	REQ_FAILED	SUM
TSCO	PCF Diego Node - diego_cell	REQ_FLIGHT	PCF Foundation	REQ_FLIGHT	SUM
TSCO	PCF Diego Node - diego_cell	REQ_STARTED	PCF Foundation	REQ_STARTED	SUM
TSCO	PCF Diego Node - diego_cell	REQ_SUCCEEDED	PCF Foundation	REQ_SUCCEEDED	SUM
TSCO	PCF Diego Node - diego_cell	THREAD_COUNT	PCF Foundation	THREAD_COUNT	SUM
TSCO	PCF Diego Node - diego_cell	SWAP_SPACE_USED/SWAP_SPACE_UTIL	PCF Domain	SWAP_SPACE_TOT	SUM
TSCO	PCF Diego Node - diego_cell	MEM_CONSUMED/MEM_UTIL	PCF Domain	TOTAL_REAL_MEM	SUM
TSCO	PCF Diego Node - diego_cell	1- CPU_UTIL_IDLE	PCF Domain	CPU_UTIL	SUM
TSCO	PCF Diego Node - diego_cell	HEAPMEM_USED/(HEAPMEM_FREE+ HEAPMEM_USED)	PCF Domain	HEAPMEM_UTIL	SUM
TSCO	PCF Diego Node, PCF Services		PCF Domain	NODE_NUM	COUNT
TSCO	PCF Diego Node		PCF Domain	DIEGO_CELLS	COUNT
TSCO	PCF Services		PCF Domain	DIEGO_NODES	COUNT
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_SYSTEM	PCF Domain	CPU_UTIL_SYSTEM	SUM
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_USER	PCF Domain	CPU_UTIL_USER	SUM
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_WAIT	PCF Domain	CPU_UTIL_WAIT	SUM
TSCO	PCF Diego Node - diego_cell	TOTAL_REAL_MEM	PCF Domain	TOTAL_REAL_MEM	SUM
TSCO	PCF Diego Node - diego_cell	CONTAINER_MAX	PCF Domain	CONTAINER_MAX	SUM
TSCO	PCF Diego Node - diego_cell	DISK_ALLOCATION	PCF Domain	DISK_ALLOCATION	SUM
TSCO	PCF Diego Node - diego_cell	DISK_SIZE	PCF Domain	DISK_SIZE	SUM
TSCO	PCF Diego Node - diego_cell	CONTAINER_NUM	PCF Domain	CONTAINER_NUM	SUM
TSCO	PCF Diego Node - diego_cell	MEM_ALLOCATION	PCF Domain	MEM_ALLOCATION	SUM
TSCO	PCF Diego Node - diego_cell	DISK_FREE	PCF Domain	DISK_FREE	SUM
TSCO	PCF Diego Node - diego_cell	MEM_FREE	PCF Domain	MEM_FREE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_USED_SPACE	PCF Domain	DISK_USED_SPACE	SUM
TSCO	PCF Diego Node - diego_cell	CPU_UTIL_IDLE	PCF Domain	CPU_UTIL_IDLE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_READ_RATE	PCF Domain	DISK_EPHEMERAL_READ_RATE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_READ_RESPONSE_TIME	PCF Domain	DISK_EPHEMERAL_READ_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_WRITE_RATE	PCF Domain	DISK_EPHEMERAL_WRITE_RATE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_WRITE_RESPONSE_TIME	PCF Domain	DISK_EPHEMERAL_WRITE_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_READ_RATE	PCF Domain	DISK_READ_RATE	SUM
TSCO	PCF Diego Node - diego_cell	DISK_READ_RESPONSE_TIME	PCF Domain	DISK_READ_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_TOTAL_LATENCY	PCF Domain	DISK_TOTAL_LATENCY	SUM
TSCO	PCF Diego Node - diego_cell	DISK_USED_INODES_PCT	PCF Domain	DISK_USED_INODES_PCT	SUM
TSCO	PCF Diego Node - diego_cell	DISK_USED_SPACE_PCT	PCF Domain	DISK_USED_SPACE_PCT	SUM
TSCO	PCF Diego Node - diego_cell	DISK_WRITE_RESPONSE_TIME	PCF Domain	DISK_WRITE_RESPONSE_TIME	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_USED_INODES_PCT	PCF Domain	DISK_EPHEMERAL_USED_INODES_PCT	SUM
TSCO	PCF Diego Node - diego_cell	DISK_EPHEMERAL_USED_PCT	PCF Domain	DISK_EPHEMERAL_USED_PCT	SUM
TSCO	PCF Diego Node - diego_cell	GC_TIME	PCF Domain	GC_TIME	SUM
TSCO	PCF Diego Node - diego_cell	GOROUTINES	PCF Domain	GOROUTINES	SUM
TSCO	PCF Diego Node - diego_cell	HEAPMEM_FREE	PCF Domain	HEAPMEM_FREE	SUM
TSCO	PCF Diego Node - diego_cell	HEAPMEM_USED	PCF Domain	HEAPMEM_USED	SUM
TSCO	PCF Diego Node - diego_cell	MEM_CONSUMED	PCF Domain	MEM_CONSUMED	SUM
TSCO	PCF Diego Node - diego_cell	MEM_UTIL	PCF Domain	MEM_UTIL	SUM
TSCO	PCF Diego Node - diego_cell	MEM_VIRTUAL_ACTIVE	PCF Domain	MEM_VIRTUAL_ACTIVE	SUM
TSCO	PCF Diego Node - diego_cell	MEM_VIRTUAL_LIMIT	PCF Domain	MEM_VIRTUAL_LIMIT	SUM
TSCO	PCF Diego Node - diego_cell	NET_IN_BYTE_RATE	PCF Domain	NET_IN_BYTE_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_IN_ERROR_RATE	PCF Domain	NET_IN_ERROR_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_IN_PKT_RATE	PCF Domain	NET_IN_PKT_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_OUT_BYTE_RATE	PCF Domain	NET_OUT_BYTE_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_OUT_ERROR_RATE	PCF Domain	NET_OUT_ERROR_RATE	SUM
TSCO	PCF Diego Node - diego_cell	NET_OUT_PKT_RATE	PCF Domain	NET_OUT_PKT_RATE	SUM
TSCO	PCF Diego Node - diego_cell	OPEN_FILES	PCF Domain	OPEN_FILES	SUM
TSCO	PCF Diego Node - diego_cell	REQ_CANCELLED	PCF Domain	REQ_CANCELLED	SUM
TSCO	PCF Diego Node - diego_cell	REQ_FAILED	PCF Domain	REQ_FAILED	SUM
TSCO	PCF Diego Node - diego_cell	REQ_FLIGHT	PCF Domain	REQ_FLIGHT	SUM
TSCO	PCF Diego Node - diego_cell	REQ_STARTED	PCF Domain	REQ_STARTED	SUM
TSCO	PCF Diego Node - diego_cell	REQ_SUCCEEDED	PCF Domain	REQ_SUCCEEDED	SUM
TSCO	PCF Diego Node - diego_cell	THREAD_COUNT	PCF Domain	THREAD_COUNT	SUM
TSCO	PCF Space		PCF Domain	SPACES	COUNT
TSCO	PCF Organization		PCF Domain	ORGANIZATIONS	COUNT
TSCO	PCF Applications		PCF Domain	APPLICATIONS	COUNT
TSCO	PCF Applications	APP_INSTANCES	PCF Domain	APP_INSTANCES	SUM

Lookup Field Considerations

Lookup Consideration

Entity Type	Strong Lookup Field	Others
PCF - Application	DOMAIN&&GUID&&ORG&&SPACE&&TYPE	WeakLoopup:GUID
PCF - Space	DOMAIN&&GUID&&ORG&&TYPE	WeakLoopup:GUID
PCF - Organization	DOMAIN&&GUID&&TYPE	WeakLoopup:GUID
PCF - Diego Node	DOMAIN&&DEPLOYMENT&&NAME&&TYPE	WeakLookup: Name, Compatibility: Name
PCF - Service	DOMAIN&&DEPLOYMENT&&NAME&&TYPE	WeakLookup: Name, Compatibility: Name
PCF - Deployment	DOMAIN&&NAME&&TYPE	WeakLookup: Name
PCF - Domain	NAME&&TYPE	WeakLookup: Name

Moviri Integrator for TrueSight Capacity Optimization - Pivotal Cloud Foundry

Create the user:

Test the user's access right:

Now, Use the same token to test the access for cloud foundry api:

Test the curl commands that is used in the connector:

Running the ETL in simulation mode

Running the ETL in the production mode

Running the ETL manually

Supported Entities

Comments