Google Cloud

This article describes how to setup Exivity to report on Google Cloud consumption.

Prerequisites

This tutorial assumes that you have a Billing Account for your Google Cloud Projects in place. In the event this is not the case please follow the steps in this tutorial before proceeding.

Setting up BigQuery

Exivity will leverage the GCP BigQuery service. This will enable the export of detailed Google Cloud billing data to a BigQuery dataset, giving Exivity the capability to query the billing data in a more granular manner than with the standard file export (which is going to be deprecated by Google).

Before proceeding further, make sure your user has the Billing Account Administrator and the BigQuery User roles associated

First and foremost, Exivity will create a dataset containing the billing table which will be periodically queried. Please follow these instructions:

1. Go to the BigQuery service page.

2. Select the project that will contain your dataset in the project drop down.

3. Click CREATE DATASET.

   1. Enter a Dataset ID

   2. Select Data location (region)

   3. Set Default table expiration to Never

   4. Set Encryption to Google-managed key

4. Finally, click Create dataset

5. Note the Project, BigQueryProject and Table, these parameters will be used by Exivity on a later stage.

Once you have the dataset created in BigQuery you need to enable Cloud Billing export to BigQuery:

1. Sign in to the Cloud Billing Console, select Organization and main Billing Account.

2. Go to the Billing Export tab.

3. Click Edit Settings to enable the export.

   1. Select the project where you previously created the BigQuery dataset.

   2. Select the specific dataset from the Billing export dataset list.

   3. Click Save.

Creating Exivity Service Account

Exivity requires a GCP Service Account with the BigQuery User role associated to retrieve the billing data. Please follow this manual to create a Service Account, this manual to associate a Private Key with the service account, and finally this manual to associate the role with the service account.

Make sure to write down the Mail address/service account and its associated Private Key, as these parameters will be required for the Exivity Data Extractor

Configuring the Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Google Cloud, please click Google_Cloud from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to Variables tab and fill the parameters:

• Hostname: Default endpoint for Google BigQuery with version

• Private: Provide the RSA private key PEM format

IMPORTANT make sure to replace \n with ${NEWLINE} in the RSA Private key using a text editor, before pasting the key in the Exivity Extractor field

• Email: Mail address/service account associated with the private key, obtained in the Creating Exivity Service Account section.

• Project: Main GCP Project name.

• BigQuery project: GCP BigQuery Billing Project, obtained in the Setting up BigQuery section.

• BigQuery table: GCP BigQuery Billing Table, obtained in the Setting up BigQuery section.

Finally, click on Update.

Example of GCP Extractor Variables

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to start collecting consumption data.

• to_date is the end date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure a Transformer

Once you have successfully run your Google Cloud Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button.

Select the Google Cloud Transformer that you just created, go to the Run Tab and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

When you click the Run Now button, you should get a successful result.

Create your Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Select the column(s) by which you would like to break down the costs (you can start with only project_name as a test). Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date selector shown when preparing the report.

Once this is done you should be able to run any of the Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.