Google Cloud Platform connector

Use the Google Cloud Platform connector to collect the cost and usage data of your compute engines that are provisioned in the Google Cloud Platform (GCP) cloud. BMC Helix Cloud Cost uses this data to provide cost estimations and optimize your cloud costs.

All communication between BMC Helix Cloud Cost and Google Cloud Platform is secure over HTTPS. The connector uses the following SDKs and methods to collect data from GCP:

  • BigQuery Client SDK Open link
  • CreateQueryStream Method Open link
  • Storage client SDK Open link
  • Download Method Open link

Information

Due to the lag in the billing cycle (from the provider) you might not see 100% accurate cost and usage data for the current and the previous day. As a result, the complete and correct data for the current day will be available after 48 hours in BMC Helix Cloud Cost.

License utilization

A product license gets consumed when the connector is used to collect data from the following asset types:

  • Google Compute Engine
  • Google Cloud Bigtable

  • Google Cloud Datastore

  • Google Cloud Spanner

  • Google Cloud SQL

  • Google Cloud Kubernetes Engine 

  • Google Cloud Dataproc

  • Google Cloud Redis

Collecting data by using the Google Cloud Platform connector

To collect data by using the Google Cloud Platform connector, do the following tasks:

I. Complete the preconfiguration tasks.

II. Configure the connector.

Step I. Complete the preconfiguration tasks

To collect data by using the GCP connector, do the following tasks:

StepDetails

Subscribe to the Google Cloud Platform.

Important: Ensure that you create a billing account in US dollar only. The connector does not support other currency formats.

  1. Log in to the Google Cloud Platform console with your Google account credentials.
  2. Select your email preferences, accept the terms of service, and click Agree and continue.
  3.  Sign up for a free trial or apply for a paid subscription.

Create projects to manage virtual machine instances, billing, and Cloud Platform services.

All the Google Cloud Platform resources, such as Google Compute Engine virtual machines, Google Cloud Storage buckets, and Google App Engine instances are grouped under a project. Projects are required for using all Cloud Platform services, managing APIs, enabling billing, adding and removing collaborators, and managing permissions.

  1. Log in to the Google Cloud Platform console with your Google account credentials.
  2. On the title bar, next to Google Cloud Platform, click Select a project.

    If you have already created projects and updated any of them, the name of the last project that you worked on is displayed besides Google Cloud Platform. In this scenario, click the project name.

  3. Click New project. In the Project name field, specify a name for your project. The project ID is generated automatically.
  4. Click Create.
  5. On the upper-left corner of the page, click Select a project or the project name next to Google Cloud Platform. Your recently created project name is displayed in the list.
  6. Select the project, and click Open.
    The project dashboard is displayed, and you can start working in the project.

Create a service account to authenticate applications that run on your virtual machine instances to access other Google Cloud Platform services. 

If you want to retrieve data from multiple projects by using a single connector, create a separate service account key file for each project.

A service account acts as an identity for an instance or an application that runs on a virtual machine. You can use service accounts to create instances and other resources. A single instance can belong to a single service account only, and you can change the service account for the instance.

  1. Log in to the Google Cloud Platform console with your Google account credentials.
  2. From the title bar, select the project for which you want to create the service account.
  3. From the left navigation pane, click IAM & admin > Service accounts.
  4. Click Create Service Account.
  5. On the Create service account page, complete these steps:
    1. Specify a name for the service account.
      The service account ID is generated automatically when you enter the service account name.
    2. (Optional) Specify the service account description.
    3. Click Create.
    4. On Service Account Permissions page, select Compute Engine > Compute Viewer. This role provides read-only access to Compute Engine resources.
  6. Click Add Another Role, and select Monitoring > Monitoring Viewer. This role provides read-only access to monitoring and configuration data. Click Continue.
  7. Click Create Key, and retain the default selection as JSON for the key type.
  8. Click Create.
    The settings are saved, and a confirmation message is displayed. The private key for the account is saved in a service account key json file and downloaded to your computer.
    Repeat this procedure for each new Google Cloud Project for which you want to collect the performance data.

    Ensure that you save this key file securely. If this file is lost or tampered, you need to create a new key.

Create a billing account.

A billing account is a payment profile that contains payment-related settings, such as payment method and a billing address. Each project is attached to a billing account, and any costs that are incurred for the project are added to the billing account. Multiple projects can be attached to a single billing account.

  1. Log in to the Google Cloud Console with your Google account credentials.
  2. In the left navigation pane, click Billing > Manage billing accounts.
  3. If this is your first billing account, complete the following steps. Otherwise, go to step 4.
    1. Click Add billing account.
    2. On the Try Cloud Platform for free page, specify your country, accept the terms of service, and click Agree and continue.
    3. On the customer information page, add the details, such as account type, tax status, name and contact, and the credit card number.
    4. Click Start my free trial.
      If the specified details such as card number are correct, a confirmation message about the generation of the billing account is displayed.
  4. If you already have some existing billing accounts and want to create a new billing account, complete these steps:
    1. Click Create account.
    2. In the Name field, specify a name for the billing account, and click Continue.
    3. Verify your country and currency, and click Confirm.
    4. On the payments profile page, enter your billing information, such as account type, tax status, name and address, credit card details, and click Submit and enable billing.
    5. On the verification page, enter the 3-letter CVV number that is on the back of your card, and click Continue.
      The billing account is created and is added to the list of existing billing accounts.
Add the email addresses of the service accounts for the projects (projects that are linked to the billing account) to the billing account, and assign the Billing Account Viewer role to each service account that you want to use for the connector.
  1. Log in to the Google Cloud Console with your Google account credentials.
  2. In the left navigation pane, click Billing.
  3. Click ALL BILLING ACCOUNTS and then select the billing account that is linked to your projects.
  4. In the right pane, click SHOW INFO PANEL and do the following:
    1. In the Add members field, add the email addresses of the service accounts that you have created for each project in earlier steps.
    2. From the Select a role list, select Billing > Billing Account Viewer.
    3. Click Add.
      The settings are saved, and a confirmation message is displayed.
Link projects to a billing account so that the costs that are incurred for the projects are added to the billing account. You can link all your projects to a single billing account or can link each project to a separate billing account.
  1. Log in to the Google Cloud Console with your Google account credentials.
  2. On the upper-left corner of the page, click Select a project or the project name next to Google Cloud Platform, and select a project from the list.
  3. In the left navigation pane, click Billing > Link a billing account.
  4. From the Billing account list, select the required billing account, and click SET ACCOUNT.
    You are directed to the billing summary page that displays the current status of remaining credits, and a list of projects that are attached to the billing account.
    If a billing account is not available, you are prompted to create the billing account.

Create a BigQuery dataset

Datasets are top-level containers that are used to organize and control access to your tables and views. A table or view must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery.

  1. Log in to the Google Cloud Console with your Google account credentials.
  2. On the upper-left corner of the page, click Select a project or project name besides Google Cloud Platform, and select the project that you have set up to contain your dataset.
  3. In the left navigation pane, click BigQuery.
  4. In the BigQuery navigation pane, select your project.
  5. Click Create dataset.
  6. On the Create Dataset page, complete the following steps:
    1. Enter a Dataset ID.

      Dataset ID must be used as an input in the Billing Dataset Name field in the connector configuration.

    2. Select a Data location.

      The data location specifies the region where your data is stored. All tables within this dataset will share this location. When creating a dataset, the Default location is the US multi-region. After you create the dataset, the location cannot be changed

    3. Select the Default table expiration.

      For data that needs to be preserved, select Never. If you enter a specific number of days, any new table created in this dataset will be automatically deleted after the specified number of days of creation. If you delete any exported data (such as Cloud Billing data records), GCP cannot backfill the deleted records.

    4. Select the Encryption option. For Cloud Billing export, select Google-managed key.
    5. Click Create dataset.
Enable Cloud Billing export to the BigQuery dataset so that the connector can collect the billing details of your project resources.
  1. Log in to the Google Cloud Console with your Google account credentials.
  2. In the left navigation pane, click Billing.
  3. If you have more than one Cloud Billing account, do one of the following:

    • To manage Cloud Billing for the current Cloud project, select Go to linked billing account.
    • To locate a different Cloud Billing account, select Manage billing accounts and choose the account for which you want to enable Cloud Billing data export to BigQuery.
  4. In the Billing navigation menu, select Billing export.
  5. Select the BigQuery export tab.
  6. Enable the Daily cost detail type of data export.
    1. Click Edit settings.
    2. From the Project list, select the project that you have set up to contain your BigQuery dataset.

      The selected project is used to store the exported Cloud Billing data in the BigQuery dataset. The exported Cloud Billing data includes usage/cost data for all Cloud projects paid for by the same Cloud Billing account.

    3. From the Billing export dataset list, select the dataset that you have set up to contain your exported Cloud Billing data.
    4. Click Save.
Enable the billing APIs to collect the billing and usage data.
  1. Log in to the Google Cloud Console with your Google account credentials.
  2. In the left navigation pane, click APIs and services Dashboard. A list of APIs that are enabled is displayed.
  3. Verify that the Google Cloud Billing API is enabled.
  4. If the APIs are disabled, perform these steps to enable them:
    1. Click Enable APIs and Services.
    2. Search for the required APIs, select them, and click Enable.
      The APIs are added to the list of enabled APIs.

      If your setup is behind a firewall, ensure that the connector can access the following API endpoints:

      1. Billing: cloudbilling.googleapis.com
      2. Stackdriver logging: logging.googleapis.com
      3. Stackdriver monitoring: monitoring.googleapis.com
      4. Cloud Resource manager: cloudresourcemanager.googleapis.com
      5. Compute engine: compute.googleapis.com
      6. Authentication: oauth2.example.comwww.googleapis.com
      7. BigQuery: bigquery.googleapis.com
      8. BigQuery Data Transfer: bigquerydatatransfer.googleapis.com

      Additionally, if you encounter issues with API requests due to 1e100.net domain, include it in the firewall allow rules.

Step II. Configure the connector

You must configure the connector to connect to Google Cloud Platform for collecting the cost data of GCP resources.

To configure the connector:

  1. In the BMC Helix Cloud Cost dashboard, navigate to Connectors Add a Connector > and select GCP Cloud Connector from the cloud based connectors.
  2. On the Configure Connector page, configure the following properties:

    PropertyDescription
    Connector nameA unique name for the connector.
    Select the type of data that you want to collect
    • Security & Compliance: Collect resource usage data to evaluate it for compliance and security. This option is available only if you are licensed to use  BMC Helix Cloud Security .
    • Billing information: Collect billing information to manage and optimize cost
      Note: Collection of cost data is supported only by exporting the GCP billing data to a CSV file.
    Client emailEnter the client email.
    Private key

    Enter the private key.

    Private key in the JSON file:
    "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG........p/ZkxmI5fg\n-----END PRIVATE KEY-----\n"

    Input required in BMC Helix Cloud Cost:
    -----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG........p/ZkxmI5fg\n-----END PRIVATE KEY-----\n

    Billing Account IDEnter the billing account ID.
    Billing Project IDEnter the project ID that is associated to your billing account ID.
    Billing Dataset NameEnter the name of the dataset that you have created for your project.
  3. Collection mode: By default, the data collection cycle is set to On Demand collection. You can select an appropriate unit of time (days, minutes, hours) to schedule the data collection frequency along with event driven collection cycle where the collection is triggered when an event is identified in the selected account.
  4. On the Select Policies page, select the policies that you want to import from the policy library. This option is available only if you are licensed to use BMC Helix Cloud Security. For more information, see Managing policies Open link .
  5. Click Continue. A confirmation message about the request for data collection processing is displayed.
    The Manage Connectors page shows the details of the newly configured GCP Cloud Connector.

Step III. Verify data collection

Verify that the connector ran successfully and check whether GCP data is refreshed on the Dashboard.

To verify whether the connector ran successfully:

  1. On the Manage Connectors page, the state of the newly configured connector is updated to Running.
    When you run the connector for the first time, the connector recovers data for the past 6 months. The data collection begins immediately but depending on the number of resources in your environment the data is displayed after some time in BMC Helix Cloud Cost.
  2. On the BMC Helix Cloud Cost dashboard, the GCP connector tab is displayed.
  3. Select the GCP tab from the Dashboard.
  4. In the Summary tab, verify that the total cost, historical cost, and total resources are displayed. Currently, the GCP connector does not support generation of recommendations. Cost details at resource level are available only for the Compute Engine service. Cost details for all other services are calculated at individual service level. If multiple resources possess the same Stock Keeping Unit (SKU), cost of all such resources is aggregated. 
  5. Resource pool information is not available by default. You must create a resource pool to view the resource pool details like name, resource count, budget, actual cost, and the projected cost. To create a resource pool, click Resource Pools.
  6. In the Accounts tab, verify that the account details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed for the account that is configured for this connector.
  7. In the Services tab, verify that the service details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed.
  8. In the Explore Bill tab, verify that the resource name, actual cost, resource type, region, account name, and the service name are displayed.



Was this page helpful? Yes No Submitting... Thank you

Comments