Moviri - Pivotal Cloud Foundry Extractor
"Moviri Integrator for BMC Helix Continuous Optimization – PivotalCF" is an additional component of BMC Helix Continuous Optimization product. It allows extracting data from the Pivotal Cloud Foundry a unified, multi-cloud platform to manage enterprise applications at scale. Relevant capacity metrics imported into BMC Helix Continuous Optimization, which provides advanced analytics over the extracted data in the form of an interactive dashboard, the Pivotal Cloud Foundry View.
The integration supports the extraction of both performance and configuration data across the different components of the Pivotal Cloud Foundry Application Service (PAS) and can configurable via parameters that allow entity filtering and many other settings. The connector can also replicate relationships and logical dependencies among entities such as applications, organizations, and virtual machines.
For the virtual machines under the Pivotal Cloud Foundry Application Service and Isolation Segment, Pivotal CF connector extracts component virtual machines. In the Pivotal CF extractor, we categorized them into two parts, the Diego Node, and PCF Services. Diego Nodes, categorizes the component virtual machines contributed to Pivotal Cloud Foundry Diego component, and other component VMs are counted as PCF Services.
For the applications under the Pivotal Cloud Foundry domain, Pivotal CF connector extracts both system default applications, as long as the customized applications, and arrange them based on the hierarchy of organization, space and applications, in aligned with the Pivotal Cloud Foundry application manager.
This documentation is targeted at BMC Helix Continuous Optimization administrators, in charge of configuring and monitoring the integration between BMC Helix Continuous Optimization and Pivotal Cloud Foundry.
Moviri Integrator for BMC Helix Continuous Optimization - PivotalCF is compatible with BMC Helix Continuous Optimization 19.11 and onward.
Step I. Complete the preconfiguration tasks
Step IV. Verify the data collection
N
| Steps | Details |
|---|---|
| Check if the required API version is supported | |
| Generate access to Pivotal Cloudfoundry | |
| (Optional) Configure to use CID | CID is extracted from Opsman API, If want to reconsile PCF Services and PCF Nodes with VMWare vms and AWS vms, an Opsman API account wit minimal restricted view privelldge should be configured: To generate the user with read rights for Ops manager API: 1. using uaac tool to connect to ops manager FQDN uaac target "https://<OPSMAN_FQDN>/uaa" --skip-ssl-validation 2. login as admin or any users with rights to create users uaac token owner get Client ID: opsman Client secret:<press enter> User name: <username> Password: <password> ----------------------------------- Unknown key: Max-Age = 172800 Successfully fetched token via owner password grant. Target: https://opsman.pcf.moviri-integrations.com/uaa Context: admin, from client opsman ------------------------------------- 3. Create a user with a password, for example opstsco uaac user add opstsco -p <enter a password for this user> --emails example@example.com ------------------------------------- user account successfully added ------------------------------------- 4. Add opsman.restrcted_view groups if not exists uaac group add opsman.restricted_view 5. Add the user opstsco to the group uaac member add opsman.restricted_view opstsco 6. Run the following curl command to get the access token for user opstsco, password is the one we created on Step3 above; Save the access token from the response Or for testing purposes, just run uaac command to get the access token: uaac token client get opstsco uaac context Alternatively to step 3,4,5,6 we can use client Id and secret, following the steps:
uaac client add <clientID> --authorized_grant_types client_credentials --authorities opsman.restricted_view --secret <ClientSecret>
curl -XPOST -H "Application/json" --data "client_id=<clientID>&client_secret=<clientSecret>&grant_type=client_credentials" "https://opsman.pcf.moviri-integrations.com/uaa/oauth/token" -k Test the curl commands that is used in the connector:1. Run the following curl command, use the access token grabbed from step1 curl "https://<OPSMAN_FQDN>/api/v0/deployed/products" -H "Authorization: Bearer <Access Token>" -k 2. Run the following curl command, use the access token grabbed from step1, Product_Id here is the deployment guid. curl "https://<OPSMAN_FQDN>/api/v0/deployed/products/<Product_Id>/status" -H "Authorization: Bearer <Access Token>" -k |
- In the console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
- On the ETL tasks page, click Add > Add ETL. The Add ETL page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, Connection, and PCF Connection, PCF Extraction Filters
- On the Run Configuration tab, select Moviri - PivotalCF Extractor from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You edit this field to customize the name.
- Click the Entity catalog tab, and select one of the following options:
- Shared Entity Catalog:
- From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
- Private Entity Catalog: Select if this is the only ETL that extracts data from the Pivotal CF resources.
Click the Connection tab, and configure the following properties:
Property Name
Value Type
Required?
Default
Advanced Description
PCF Connection
Pivotal CloudFoundry domain
String yes no The domain of your Pivotal CLoud Foundry. Eg: examplepcf.com Pivotal CloudFoundry username String yes no The username of the user has the correct access right Pivotal CloudFoundry password String yes no The password of the user has the correct access right Proxy server host String no no If proxy is used, filling this proxy server's host. Only "http" is supported. Proxy server port Number no no If proxy is used, filling this proxy server's port number. Proxy server user String no no if proxy is used, filling this proxy server's user name. Proxy server password String no no If proxy is used, filling this proxy server's user password PCF Extraction Filter
Data resolution
Option yes 1 hour no Data resolution of the improted data. Options are 5 minutes, 15 minutes, and 1 hour Select only Applications on the following organizations, use GUID (; separated) text no no Application whitelist. Show only applications from the following organizations. Use organization GUID. Separate each organizations GUID using ";". Select only Applications on the following organizations, use name (; separated) text no yes Application whitelist. Show only applications from the following organizations. Use organization names. Separate each organizations name using ";". Select only Applications in the following spaces, use GUID (; separated) text no no Application whitelist. Show only applications from the following spaces. Use space GUID. separate each space GUID using ";". Select only VMs in the following deployments (; separated) text no no Virtual machine whitelist. Show only virtual machine's from the following deployments, separate each deployment using ";". Exclude any Applications on the following organizations, use GUID (; separated) text no no Application blacklist. Exclude only applications from the following organizations. Use organization GUID. Separate each organizations GUID using ";". Exclude any Applications on the following organizations, use name (; separated) text no yes Application blacklist. Exclude only applications from the following organizations. Use organization names. Separate each organizations name using ";". Exclude any Applications in the following spaces, use GUID (; separated) text no no Application blacklist. Exclude only applications from the following spaces. Use Space GUID. Separate each space GUID using ";". Exclude any VMs in the following deployments (; separated) text no no Virtual machine blacklist. Exclude only virtual machines from the following deployments. Separate each deployment using ";". Import vm list, leave empty if import all selection no empty yes Select which types of virtual machines you want to import. If want to import all, leave this field empty. Do you want to reconcile vms with other entities using CID? Option no no no If selected to "yes", the ETL will reconsile using cid to reconcile PCF nodes and PCF Services, and importing them as "generic" FQDN of opsman API (format http/https://host:[port]) String no yes Opsman API FQDN Address, this property will be visible if set "yes" to "Do you want to reconcile vms with other entities using CID?" Token generating method (User or Client) Selection no ClientId yes Select password or clientid as authentication method. This property will be visible if set "Yes" to Do you want to reconcile vms with other entities using CID? Username of the Opsman API user String no yes Username of the Opsman API user if the token generation method is set to "user" Password of the Opsman API password String no yes Password of the Opsman API user if the token generation method is set to "user" ClientId of the Opsman API Client String No yes Client Id of the Opsman API if the token generation method is set to "client credential" Client Credential of the Opsman API Client String No yes Client credential of the Opsman API if the token generation method is set to "client credential" Hidden properties: Pivotal CloudFoundry ETL provides extra hidden properties and they can be changed manually from edit configuration page.
- To change the default polling period (1 minute ) for Applications, Organizations, and Spaces data, set a hidden property "extract.pcf.app.serve.interval" and set the value to desired seconds. For example, if want to change default polling interval from 1 minute to 5 minute, set the value to 300.
- To change the default polling period (1 minute ) for Bosh deployment and its nodes data, set a hidden property "extract.pcf.bosh.serve.interval" and set the value to desired seconds. For example, if want to change default polling interval from 1 minute to 5 minute, set the value to 300.
All the other generic properties are documented here.
The ETL tasks page shows the details of the newly configured Pivotal Cloudfoundry ETL.
- Shared Entity Catalog:
After you configure the ETL, you can run it to collect data. You can run the ETL in the following modes:
A. Simulation mode: Only validates connection to the data source, does not collect data. Use this mode when you want to run the ETL for the first time or after you make any changes to the ETL configuration.
B. Production mode: Collects data from the data source.
Running the ETL in simulation mode
To run the ETL in the simulation mode:
- In the console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
- On the ETL tasks page, click the ETL. The ETL details are displayed.
- In the Run configurations table, click Edit
to modify the ETL configuration settings. - On the Run configuration tab, ensure that the Execute in simulation mode option is set to Yes, and click Save.
- Click Run active configuration. A confirmation message about the ETL run job submission is displayed.
- On the ETL tasks page, check the ETL run status in the Last exit column.
OK Indicates that the ETL ran without any error. You are ready to run the ETL in the production mode. - If the ETL run status is Warning, Error, or Failed:
- On the ETL tasks page, click
in the last column of the ETL name row. - Check the log and reconfigure the ETL if required.
- Run the ETL again.
- Repeat these steps until the ETL run status changes to OK.
- On the ETL tasks page, click
Running the ETL in the production mode
You can run the ETL manually when required or schedule it to run at a specified time. As of version 20.02.00.005, the integration delays the hierarchy to delay for six hours helping to reduce the load on the loader step.
Running the ETL manually
- On the ETL tasks page, click the ETL. The ETL details are displayed.
- In the Run configurations table, click Edit
to modify the ETL configuration settings. The Edit run configuration page is displayed. - On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
- To run the ETL immediately, click Run active configuration. A confirmation message about the ETL run job submission is displayed.
When the ETL is run, it collects data from the source and transfers it to the database.
Here's screenshot of ETL page after set it up and run completely.
Verify that the ETL ran successfully and check whether the Pivotal CF data is refreshed in the Workspace.
To verify whether the ETL ran successfully:
- In the console, click Administration > ETL and System Tasks > ETL tasks.
- In the Last exec time column corresponding to the ETL name, verify that the current date and time are displayed.
- In the console, click Workspace.
- Expand (Domain name) > Systems > PivotalCF> Instances.
- In the left pane, verify that the hierarchy displays the new and updated Pivotal CF instances.
- Click a Pivotal CF entity, and click the Metrics tab in the right pane.
- Check if the Last Activity column in the Configuration metrics and Performance metrics tables displays the current date.
| Pivotal CF Workspace | Details |
|---|---|
| Entities | |
| Hierarchy | The connector can replicate relationships and logical dependencies among these entities as they are found configured within the Pivotal Cloud Foundry cluster. hierarchy change By default the hierarchy will be updated every 6 hours, starting 0:00 of the day.
The following image shows a sample hierarchy.
|
| Configuration and Performance Metrics Mapping | |
| Derived Metrics | |
| Lookup Field Considerations |
Comments
Log in or register to comment.