Pre-requisites

This tutorial assumes that you have CUR (Cost and Usage Report) set up in your AWS environment. In the event that this is not the case please follow the steps in Turning on the AWS Cost and Usage Report before proceeding.

Please note that we also need the following requisites:

• S3 bucket where the CUR reports are placed

• The AWS region of the S3 bucket

• The path where the reports are placed inside the S3 bucket

To get this requirements you can go to the Amazon S3 service - Buckets and note the name and the region of our CUR S3 bucket

Once inside the CUR bucket, you must find the path where the monthly reports reside, please navigate the bucket until you find a folder like this one and note the path:

Obtaining the API Credentials

We have to create an IAM User that has read access to the CUR bucket. Please follow these steps:

• Go to IAM - Policies and click on Create policy.

• In the next screen: Service select S3, Actions select Read, Resources select Specific and in bucket click on Add ARN.

• In the next screen, type the name of your bucket followed by /* in Bucket name (example: billing-bucket/*) and click on Add.

• Click on Review Policy.

• Give a Name and a Description to the policy and finally click on Create policy.

Now, we will attach the policy to a user:

• Go to IAM – Users – Add user.

• Add an User name and select Access type: Programmatic access.

• Click on Attach existing Policies directly and filter by the name of the policy you just created.

• Select the policy and click on Next:Tags.

• Add a Key-Value pair for the Tags and click on Next:Review.

• Click on Create User.

• In the next screen click on Download .csv we will use this credentials to populate our Exivity Extractor in the next section.

• Finally click on Close.

Configure Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For AWS, please click AWS_CUR_Extractor from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to first tab: Variables

• Bucket: Type the name of your S3 CUR bucket.

• AWS region: Name of your S3 CUR bucket region code you can find them here (column Region).

• Access key: You can find it in the csv that you have downloaded in the previous section.

• Secret key: You can find it in the csv that you have downloaded in the previous section.

• Cur report: Type the name of your CUR report.

• Bucket Directory: Type the path that you got in the previous section (if there is more than one folder separate them with /).

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires one parameters in yyyMMdd format:

• from_date is the date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your AWS CUR Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button. Select the AWS CUR Transformer and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

Create Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

When deploying the Azure EA for Exivity, some configuration is required within your Azure EA environment. The following process must be completed in order to report on Azure EA consumption:

1. Create an Access Key and Secret in your Azure EA portal

2. Configure the Azure EA Extractor

3. Configure your Azure EA Transformer

4. Create a Report definition

Creating an Access Key & Secret

In order for Exivity to authenticate with Azure EA, you will need to create an access key and secret in the Azure EA Portal. Also you will need to find your enrollment number. To do this, login to your Azure EA Portal on and navigate to the Reports menu:

Just under the Windows logo in the left menu section you will find your enrollment number which you will need to provide later when configuring the Extractor. To create the Access Key & Secret, click on Download Usage and then on API Access Key. This brings you to the menu where you can manage your Access Keys and corresponding secret which you will need in order to configure the data Extractor.

Configure Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Azure EA:

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, go to first tab: Variables

Fill in all variables in above screenshot, and feel free to encrypt any sensitive data using the lock symbol on the right.

Once you've filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyyMMdd format:

• from_date is the date for which you wish to collect consumption data

• to_date should be the date immediately following from_date

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your Azure EA Blob Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

When deploying the Azure CSP for Exivity, some configuration is required within your Microsoft Cloud Solution Provider Potal. The following process must be completed in order to report on Azure CSP consumption:

1. Create a Partner Center Web Application

2. Configure Extractors for Azure CSP Usage, Billing & Ratecard

3. Configure Transformers

4. Create your Report

5. Create your Workflows

It is necessary to create independent Extractors/Transformers for the Usage and Billing, the Usage Extractors will be retrieving the data on a daily basis giving an estimation of your daily costs. The Billing Extractor will consolidate the rates based on the blended costs per service for the billing period.

Create a Partner Center Web Application

Perform the following to create the Azure AD configured to access the Partner Center API

• Browse to Partner Center, , and login using credentials that have admin agent and global admin privileges

• Click on the following Dashboard –> Account Settings –> App Management

• Click Add key to create a new Application Key for you App ID that can be used with Exivity.

• Go to the Billing section.

• Open the last month's invoice in pdf format.

• Take note of your billing period

Configure Extractors for Azure CSP Usage, Billing & Ratecard

Go into the Exivity GUI and browse to Data Sources -> Extractors. Then click on Create Extractor and you should get a list of templates. Unfold Azure CSP and pick the usage template:

After selecting the template, click the green Create button on the bottom right. Now make sure to give the new Extractor a name in field at the top:

Now click again on the green Create button at the bottom right. Then click on the Variables menu item:

Now make sure to fill in your Client ID, Secret and your onmicrosoft.com domain. When required, you can encrypt security sensitive field using the lock button on the right of each field. Once you are filling in these details, click the Update button.

Now test the Extractor by going into the Run tab and providing a from and to date like in this example:

Now click the Run Now button and confirm it works as expected:

Create a second extractor using the template Azure_CSP_Invoice_Extractor and give it a name.

This extractor uses the same variables in the Variables menu item as the previous extractor. You can now test the extractor by going to the Run tab, this script uses 3 arguments:

• Positive offset: Starting on 0, it will retrieve the most recent invoice, a 1 will retrieve the previous report, etc.

• Year of the report: Year of the report you want to retrieve.

• Starting day of the billing period: If your billing period goes from 22nd to 21st, the input will be 22.

After filling the arguments you can test the extractor by clicking in Run Now.

Finally, follow the same steps for the Azure Rate Card Extractor, this extractor does not need any arguments.

Configure Transformers

Once you have successfully run your Azure CSP Usage, Billing & Rate Card Extractors, you can create the Transformers templates via Data Sources -> Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. You will need to create two separate transformers using these two templates:

The Azure_CSP_Daily-Usage Transformer will transform the daily usage data and the Azure_CSP_End-of-Month Transformer will consolidate the usage with the final blended rates.

Make any changes that you feel necessary and then select the Run tab to execute it for a single day as a test.Make sure that when running the Transformer you select custom range in the drop-down menu labelled Run for and select the same day as for which you have extracted consumption data in the previous step.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Create your Workflows

You may want to automate the CSP ETL process, you can achieve it by leveraging on the Exivity's Workflow capabilities. You will create two Workflows, one will run on a daily basis calculating the usage of your CSP subscriptions and the other one will run on a monthly basis to consolidate the service rates.

Start by browsing to Administration -> Workflows in the Exivity GUI and click on the +Create button.

Fill the Name and Description fields, in the SCHEDULES section configure the workflow to run on a daily basis at a convenient time.

In the STEPS section you can create as many steps as needed by adding them with the + button. For the first daily workflow, a minimum of 4 steps are required, two steps for the Usage and Ratecard extractors, one for the Transformer and one for the Report. Make sure to input the right FROM and TO date offsets. Click on Update to finish the creation of your first Workflow.

Create a second Workflow, fill the Name, and Description fields and configure the SCHEDULES section to run on a monthly basis, preferably 2 to 3 days after your month's billing period has finished. For the monthly workflow, a minimum of 3 steps are required, one step for the Billing extractor, one for the Transformer and one for the Report. Make sure to input the right FROM and TO date offsets (to cover the entire billing period) and arguments. Click on Update to finish the creating of the second Workflow.

Introduction

When deploying the Azure Stack for Exivity, some configuration is required within your Azure Stack environment and a lookup file needs to be created. The following process must be completed in order to report on Azure Stack consumption:

1. Create an Exivity Enterprise Application in your Azure AD for authentication

2. Configure a rate card lookup file

3. Configure an Extractor

4. Configure a Transformer

5. Create your Report

Creating an Enterprise Application

In order for Exivity to authenticate with Azure Stack, you will need to create an application in the Azure AD where you have registered your Azure Stack management node:

When you create this application in your Azure AD make sure it has (at least) the Reader Role in your Default Provider Subscription:

Create a rate card

%EXIVITY_HOME_PATH%\system\extracted\AzureStack\rates\azure_stack_example_rates.csv

Configure Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Azure Stack:

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, go to first tab: Variables

Fill in all required variables marked within the red box in above screenshot. If you don't know some of the required GUIDs, most of these can be obtained by browsing to the Azure Stack management node URL:

• https://adminmanagement.**<your.domain.com>**/metadata/endpoints?api-version=2015-01-01

Another way to obtain some of this information is using the Diagnostics button in your management portal:

When you click the Show Diagnostics link, it should download a JSON file containing most of the parameters you'll need, such as Provider GUID, Audience GUID etc.

Once you've filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyyMMdd format:

• from_date is the date for which you wish to collect consumption data

• to_date should be the date immediately following from_date

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your Azure Stack Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

When deploying the vCenter Extractor template for Exivity, it is required to configure a user with approperiate permissions. Additionally, the following process must be completed in order to report on vCenter consumption:

1. Create Exivity vCenter user

2. Configure an Extractor

3. Configure a Transformer

4. Create your Report

Exivity supports Out of the Box integration with vCenter version 6.5 and higher. For version before 6.5, please contact support@exivity.com.

Create Exivity vCenter user

Exivity needs a user with a reader role in order to retrieve consumption data. Please follow this procedure:

• In your vCenter Configuration Manager go to Configuration > Local Users and Groups > Users.

• Create a new User, insert username and password.

• Take note of the username and password, they will be used later on to configure Exivity

• Click on Create.

• Under Users and Groups, click Add, select the previously created user.

• On the Select Users and Groups dialog, click Add, and then click OK.

• Add a Reader role in the dropdown.

• Finally click on OK.

Configure an Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. To create a new Extractor based off the vCenter template follow these steps:

• Provide a name for the Extractor in the Name field above

• Pick vCenter 6.5 (VM Inventory REST API) template from the list

• Click the Create button.

Once you've created the Extractor, go to theVariables tab:

Fill in all required variables with the values that you gathered in the previous step. You have the option to encrypt a variable in case it contains sensitive information (i.e. password) by clicking the lock icon on the right of each variable field. When finished, click Update.

Once you've filled in all details, go to the Run tab to execute the Extractor clicking on Run Now:

If the variables are correct and your vCenter is reachable for Exivity, you should get a successful result.

Configure a Transformer

Once you have successfully run your vCenter Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day (today) as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select your vCenter dataset, and your preferred Reporting Columns to break down the report (we recommend only cluster_name for the default report). When you are ready, click on Create.

Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Pre-requisites

This tutorial assumes that you have a Billing Account for your Google Cloud Projects in place. In the event this is not the case please follow the steps in this tutorial before proceeding.

Setting up BigQuery

Exivity will leverage on the GCP BigQuery service. This will enable the export of detailed Google Cloud billing data to a BigQuery dataset, giving Exivity the capability to query the billing data in a more granular manner than with the standard file export (which is going to be deprecated by Google).

First and foremost, Exivity will create a dataset containing the billing table which will be periodically queried. Please follow these instructions:

1. Go to the BigQuery service page.

2. Select the project that will contain your dataset in the project drop down.

3. Click CREATE DATASET.

   1. Enter a Dataset ID

   2. Select Data location (region)

   3. Set Default table expiration to Never

   4. Set Encryption to Google-managed key

4. Finally, click Create dataset

5. Note the Project, BigQueryProject and Table, these parameters will be used by Exivity on a later stage.

Once you have the dataset created in BigQuery you need to enable Cloud Billing export to BigQuery:

1. Sign in to the Cloud Billing Console, select Organization and main Billing Account.

2. Go to the Billling Export tab.

3. Click Edit Settings to enable the export.

   1. Select the project where you previously created the BigQuery dataset.

   2. Select the specific dataset from the Billing export dataset list.

   3. Click Save.

Creating Exivity Service Account

Exivity requires a GCP Service Account with the BigQuery User role asscociated in order to retrieve the billing data. Please follow this manual to create a Service Account, this manual to associate a Private Key with the service account, and finally this manual to associate the role with the service account.

Configuring the Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Google Cloud, please click Google_Cloud from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to Variables tab and fill the parameters:

• Hostname: Default endpoint for Google BigQuery with version

• Private: Provide the RSA private key PEM format

• Email: Mail address / service account associated with the private key, obtained in the Creating Exivity Service Account section.

• Project: Main GCP Project name.

• BigQuery project: GCP BigQuery Billing Project, obtained in the Setting up BigQuery section.

• BigQuery table: GCP BigQuery Billing Table, obtained in the Setting up BigQuery section.

Finally, click on Update.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to start collecting consumption data.

• to_date is the end date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure a Transformer

Once you have successfully run your Google Cloud Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button.

Select the Google Cloud Transformer that you just created, go to the Run Tab and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

When you click the Run Now button, you should get a successful result.

Create your Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

When deploying the vCloud Extractor template for Exivity, some input and configuration is required from your vCloud environment. The following process must be completed in order to report on vCloud consumption:

1. Create Exivity vCloud user (vCloud < 9.1)

2. Create Exivity vCloud user (vCloud >= 9.1)

3. Configure an Extractor

4. Configure a Transformer

5. Create your Report

Create Exivity vCloud user (vCloud < 9.1)

For environments with a vCloud previous to 9.1 Exivity needs a user with the sysadmin role on the system ORG. Please follow this procedure:

• Click the Administration tab and click Users in the left panel.

• Click New, fill the required details.

• Note the username and password.

• Finally, click OK.

Create Exivity vCloud user (vCloud >= 9.1)

For environments with vCloud version 9.1 or higher, you can create a user with more fine grained permissions. Exivity needs a user with a reader role on the system ORG. Please follow this procedure:

First you will need to create a new custom role for the Exivity user:

• From the main menu, select Administration.

• In the left panel, under Access Control, click Roles

• Click New.

• Enter a name and, optionally, a description for the new role.

• Select the rights that you want to associate with the role. Exivity needs all the View Rights in the different tabs and also Perform Administrator Queries under the General tab.

• Click Save.

Once you have the Role created, you need to setup a new user and assign the Role previously created:

• On the VMware Cloud Service toolbar, click the VMware Cloud Services icon and select Identity & Access Management.

• Click Add Users.

• On the Active Users tab, fill the details of the user you want to add to the system organization.

• In the Role in organization text box, assign the role previously created.

Configure an Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For vCloud:

• Pick vCloud_Extractor_AdminVM from the list

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, next go to theVariables tab:

Fill in all required variables with the values that you gathered in the previous step. You have the option to encrypt them. Click on Update.

Once you've filled in all details, go to the Run tab to execute the Extractor clicking on Run Now:

Configure a Transformer

Once you have successfully run your vCloud Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day (today) as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions.

Select your vCloud dataset, and your preferred Reporting Columns to break down the report (we recommend only Org_name and VDC for the default report). When you are ready, click on Create.

Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Pre-requisites

This tutorial assumes that you have CUR (Cost and Usage Report) set up in your AWS environment. In the event that this is not the case please follow the steps in Turning on the AWS Cost and Usage Report before proceeding.

Please note that in order to deploy this solution the S3 bucket to which CUR reports are written must reside in one of the following AWS regions:

• Northern Virginia

• Ohio

• Oregon

• Mumbai

• Seoul

• Singapore

• Sydney

• Tokyo

• Frankfurt

• Ireland

• London

At this point in time, only the previous regions have all the necessary services deployed.

Introduction

This tutorial shows how to build a serverless solution for querying the AWS CUR Report using Exivity. This solution makes use of AWS serverless services such as Lambda and Athena, as well as other commonly used services such as S3, CloudFormation, and API Gateway. The following topics will be covered:

1. Solution Overview

2. Launching the CloudFormation Template

3. Creating the Lambda function and API Gateway

4. Configuring an Extractor

5. Configuring a Transformer

6. Creating your Report

Solution Overview

The Billing and Cost Management service writes your AWS Cost and Usage report to the S3 bucket that you designated when setting up the service. These files can be written on either an hourly or daily basis.

The CloudFormation template that accompanies this tutorial builds a Serverless environment containing a Lambda function which reads a CUR file, processes it and writes the resulting report to an output S3 bucket. The output data object has a prefix structure of "year=current-year" and "month=current-month". For example, if a file is written 13/09/2018 then the Lambda function outputs an object called "bucket-name/year=2018/month=09/file_name".

The next step in the template is to translate this processed report into Athena so that it can be queried. The following diagram shows the steps involved in the process:

Afterwards, we will create a Lambda function to query the Athena database, returning a URL with the results of the query in CSV format. We will also create an API EndPoint with the AWS API Gateway service, which is used by Exivity to retrieve the data.

Launching the CloudFormation template

In order to deploy this solution successfully the following information is required:

1. The name of your AWS Cost and Usage report.

2. The name of the S3 bucket in which the reports are currently stored.

Firstly, launch the CloudFormation template that builds all the serverless components that facilitate running queries against your billing data. When doing this, ensure that you choose the same AWS Region within which your CUR S3 bucket is located.

• Ireland

• Ohio

• Oregon

• Northern Virginia

• Mumbai

• Seoul

• Singapore

• Sydney

• Tokyo

• Frankfurt

• London

Now follow the instructions in the CloudFormation wizard, using the following options, and then choose Create.

• For CostnUsageReport, type the name of your AWS Cost and Usage report.

• For S3BucketName, type a unique name to be given to a new S3 bucket which will contain the processed reports.

• For s3CURBucket, type the name of the bucket into which your current reports are written.

While your stack is building, a page similar to the following is displayed.

When the Status column shows CREATE_COMPLETE, you have successfully created four new Lambda functions and an S3 bucket into which your transformed bills will be stored.

Once you have successfully built your CloudFormation stack, you can create a Lambda trigger that points to the new S3 bucket. This means that every time a new file is added to, or and existing file is modified in, the S3 bucket the action will trigger the lambda function.

Create this trigger using the following steps:

• Open the Lambda console.

• Choose Functions, and select the aws-cost-n-usage-main-lambda-fn-A Lambda function (note: do not click the check box beside it).

• There should be no existing triggers. Choose Trigger, Add trigger.

• For Trigger type (the box with dotted lines), choose S3.

• Select the S3 bucket withing which your CUR reports are stored.

• For Event type, choose Object Created (All) and check Enable trigger.

• Click Submit.

The database and table are not created until your function runs for the first time. Once this has been done, Athena will contain the database and table.

Athena stores query results in S3 automatically. Each query that you run has a results file in CSV format and a metadata file (*.csv.metadata) that includes header information such as column type, etc.

Testing (Optional)

Once you have successfully added the trigger to the S3 bucket in which the Billing and Cost Management services writes your CUR reports, test the configuration using the following steps.

• In the S3 path to which AWS writes your AWS Cost and Usage Billing reports, open the folder with your billing reports, open the folder with your billing reports. There will be either a set of folders or a single folder with a date range naming format.

• Open the folder with the data range for the current month. In this folder, there is a metadata file that can be found at the bottom of the folder. It has a JSON extension and holds the S3 key for the latest report.

• Download the metadata file. Ensure that the name of the file on your machine is the same as the version stored on your S3 bucket.

• Upload the metadata file to the same S3 path from which you downloaded it. This triggers the Lambda function aws-cost-n-usage-main-lmbda-fn-A.

• In the S3 bucket that you created to hold your processed files, choose the "year=" folder and then the "month=" folder that corresponds to the current month. You should see the transformed file there, with the time stamp that indicated that it was just written.

Creating the Lambda function and API Gateway

To automate this process a CloudFormation template will be provided. This template will create an IAM role and Policy in order that our API can invoke Lambda functions. Then it will create a Lambda function with the capabilities of querying our previously created Athena Serverless DB, and save the output in a S3 bucket in .csv format, (this output will be later retrieved by Exivity). Finally, it will deploy an API Gateway allowing us to create an endpoint for our Lambda function, this is the endpoint that the Exivity extractor will consume. Make sure to launch the CloudFormation template in the same region that you have deployed the previous one.

Let's start by downloading the CloudFormation template (you only need to choose one of the formats, both are supported by AWS):

Then follow the next steps:

• Go to the CloudFormation console.

• Choose Create Stack.

• Choose Upload a template to Amazon S3.

• Select from your computer the template that you have downloaded.

• Follow the CloudFormation wizard - Add a Name to the Stack and select I acknowledge that AWS CloudFormation might create IAM resources with custom names in the last step.

• Once the stack is created you should see an CREATE_COMPLETE message.

• Click on Output to take a note of your endpoint (you will need to input this in the Exivity extractor).

Next, we will associate an API Gateway trigger to our Lambda function:

• Go to the Lambda console.

• Choose the QueryAthena2 function.

• Under Add Triggers select API gateway. You should see an image like the following:

• Click on API Gateway figure to configure it.

• On API select QueryAthena2.

• On Deployment Stage select v1.

• On Security select Open.

• Choose Add.

• Choose Save.

You should see a screen like this:

Finally, we will deploy the API Gateway:

• Go to the API Gateway console.

• Choose QueryAthena2.

• In the Resources section, click on the ANY method.

• In Actions, choose Delete Method.

• Click on Delete.

• In the Resources section, choose Actions.

• Click on Deploy API

• In Deployment Stage select V1.

• Add a Deployment Description.

• Choose Deploy.

Securing the API Gateway

Initially, the created API endpoint is public and as such is vulnerable to the possibility of misuse or denial-of-service attacks. To prevent this, associate an API Key with the endpoint as per the following steps:

• Inside the API Gateway dashboard, select the QueryAthena2 API

• In Resources, select Method Request

• In Settings, change API Key Required to True

• Click on Actions and choose Deploy API to effect the change

• In Deployment Stage, select v1 and click on Deploy

• Go to the API Keys section

• Click on Actions and select Create API Key

• In Name write ExivityAPIKey

• Click on Save

• Copy the API Key, as this will be required by the Exivity configuration

• Go to Usage Plan

• Click on Create.

• In Name write ExivityUsagePlan

• In the Throttling Section, change Rate to 100 and Burst to 10

• In the Quota Section, change it to 50000 requests per Month

• Click on Next

• Click on Add API Stage

• In API, select QueryAthena2 and in Stage select v1

• Confirm the changes and click on Next

• Click on Add API Key to Usage Plan

• Select ExivityAPIKey, confirm the changes

• Click on Done

The API Key is now required to access the API endpoint thus adding a layer of security to mitigate unauthorized access attempts.

Configure Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For AWS, please click AWS_CUR_Extractor from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to first tab: Variables

• In the Bucket variable specify the name of the S3 bucket where the .csv with the output of the query will be saved (The S3BucketName previously specified when launching the CloudFormation template).

• In the Api endpoint variable specify the API endpoint previously created plus the route /QueryAthena.

• In the DBname variable specify the name of your DB, you can find it in the Athena main Dashboard.

• In the Tablename variable specify the name of the table inside your DB, you can find it in the Athena main Dashboard.

• In the API_Key variable specify the API Key that we have created in the Securing API Gateway Section.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to collect consumption data.

• to_date should be the date immediately following from_date.

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your AWS CUR Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button. Select the AWS CUR Transformer and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

Create Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.