Exivity can be deployment on a single node, or on multiple nodes for HA and load balancing. This guide walks you trough the steps to install some of the Exivity components on different nodes.

Multi Node System Architecture

The following diagram outlines the various components that can be deployed on seperate nodes:

• PostgreSQL DB

  • Apart from installing PostgreSQL from the installer, Exivity is compatible with any PostreSQL compatible database engine (PostgreSQL on Linux, Amazon RDS, CockroachDB, etc) in order to achieve High Availability

• API/Backend

  • The API and backend components can be installed on any Windows Server. To achieve HA, it is possible to leverage HA techniques from the underlying hypervisor. Or optionally it is possible to deploy Exivity on an Active/Passive Windows cluster

• Web/UI

  • The Exivity Web application and front end facing component can be deployed on as many instances as you like. In case multiple Web UI instances are required, this will require a load balancer in front of the Exivity Web application. Please note that the load balancer component is not shipped with the Exivity installer. However, Exivity is compatible with most load balancer software suits available (i.e. Nginx, Cisco, F5, AWS ELB, etc)

Deploying a PostgreSQL node

In case you prefer to install PostgreSQL on Windows, Exivity recommends to install the PostgreSQL role together with the API/Scheduler backend components during the installation wizard.

Depoying an API/backend

In order to deploy a API/Backend node, the following steps needs to be executed manually or automatically thru the use of the silent installer CLI interface.

After starting the installer, click Next and provide a valid license key when asked. After clicking Next again, the component screen will be shown:

Ensure to deselect the Web Service component.

Click Next to continue. Then chose a folder for the Exivity program files, and afterwards select a folder for the Exivity home files.

Provide a custom administrator username and password, or leave the default:

Now specify a remote PostgreSQL database instance, or select to install the PostgreSQL database locally on the API/backend node:

When you are finished configuring your PostgreSQL database settings, click the Install button to start the installation process.

Once the installation is finished, ensure to check 'Start the Exivity Windows Services' to start the Exivity services after clicking Finish.

Silently install API/Backend node

The below example will silently install an Exivity API/Backend node while using a remote PostgreSQL database instance:

Exivity_{version}_setup.exe /S /PGUSER=exivityadmin /PGPASSWORD=S3cret!123 /PGHOST=db.exivity.local /PGPORT=5432 /PGDB=exivitydb /PSQL_INSTALLED=0 /SCHEDULER_INSTALLED=1 /API_INSTALLED=1 /WEB_INSTALLED=0

Deploying a Web/UI node

In order to deploy a Web/UI only node, the following steps needs to be executed manually or automatically thru the use of the silent installer CLI interface.

After starting the installer, click Next and provide a valid license key when asked. After clicking Next again, the component screen will be shown:

Click Next to continue. Then chose a folder for the Exivity program files, and afterwards select a folder for the Exivity home files.

In the following screen, it will be required to specify the remote host and port of your Exivity API node:

Once the installation is finished, ensure to check 'Start the Exivity Windows Services' to start the Exivity services after clicking Finish.

Silent install Web/UI node

The below example will silently install an Exivity Web/UI node:

Exivity_{version}_setup.exe /S /EXIVITY_PROGRAM_PATH=C:\Exivity\Program /EXIVITY_HOME_PATH=D:\Exivity\home /PSQL_INSTALLED=0 /SCHEDULER_INSTALLED=0 /API_INSTALLED=0 /WEB_INSTALLED=1 /PROXIMITYHOST=remote.api.local /PROXIMITYPORT=443

Exivity can be easily deployed on any system running Windows Server 2016 or later.

Single Node System Architecture

In order to deploy a single node Exivity system, be aware of the following system architecture:

All components can be installed on a single node. However, even in a single node deployment, it might be desirable to run a the PostgreSQL database on a diffent system. Apart from installing PostgreSQL from the Exivity installer, ths Exivity software is also compatible with any PostreSQL compatible database engine (PostgreSQL on Linux, Amazon RDS, CockroachDB, etc) in order to achieve High Availability or Load Balancing.

Deploying Exivity on a Single Node

After starting the Exivity installer and providing a valid license, ensure to have all components selected and click Next:

Then make sure to provide a valid path for the Exivity program files:

And select a folder for the Exivity home files:

Provide a custom administrator username and password, or leave the default:

Then provide the details for the PostgreSQL database. The installer will configure a local PostgreSQL database by default:

In case you prefer to use a remote PostgreSQL database system, ensure to deselect 'Install Local PSQL Engine' and provide your remove PSQL server credentials:

When you are finished configuring your PostgreSQL database settings, click the Install button to start the installation process.

Once the installation is finished, ensure to check 'Start the Exivity Windows Services' to start the Exivity services after clicking Finish.

Executing a Single Node Silent Installation

Follow below steps to execute an unintended setup using the silent installer flags supported by the setup program.

1 - installing Exivity on a single node with custom PostgreSQL data path

The below example will silently install Exivity with mostly default settings, except having the Exivity program, home and PGDATA in seperate locations:

Exivity_{version}_setup.exe /S /EXIVITY_PROGRAM_PATH=C:\Exivity\Program /EXIVITY_HOME_PATH=D:\Exivity\home /PGDATA=E:\PostgreSQL\PGDATA 

2 - installing Exivity on a single node using a remote PostgreSQL database

The below example will silently install Exivity while using a remote PostgreSQL database:

Exivity_{version}_setup.exe /S /EXIVITY_PROGRAM_PATH=C:\Exivity\Program /EXIVITY_HOME_PATH=D:\Exivity\home /PGUSER=exivityadmin /PGPASSWORD=S3cret!123 /PGHOST=db.exivity.local /PGPORT=5432 /PGDB=exivitydb /PSQL_INSTALLED=0

Exivity can be installed in your on-premises data center using the provided installer. You can automatically deploy it using the silent installation command line options or execute it as an interactive installer.

To install Exivity, you'll need the following:

1. A system that complies with the Exivity minimal system requirements

2. The Exivity software installation executable

3. A valid Exivity license key

Interactive installation

To install Exivity interactively, execute the provided setup executable. Then follow the instructions depending on your intended architecture:

Silent Installation

To execute a silent installation the following command line parameters are supported:

Example to install using all defaults:

<setup>.exe /S

Example to install using custom path and Exivity admin user:

<setup>.exe /S /EXIVITY_PROGRAM_PATH="C:\Program Files\Exivity\program" /EXIVITY_HOME_PATH=D:\Exivity\home /ADMIN_USER=master /ADMIN_PASSWORD=P@ssword

Updating minor versions

Updating your installation of Exivity to a more recent minor and/or bugfix release (i.e. 3.x.x --> 3.x.x) is straightforward and may be done by installing the new version over the top of the old.

Manually update

Execute the setup executable. It will detect the installed version of Exivity, and will automatically upgrade when you click Next

Silent update

When executing <setup>.exe /S, your existing installation will be automatically upgraded.

Upgrading major versions

Installing a valid SSL certificate

Exivity comes as standard with an untrusted self-signed SSL certificate. It is therefore highly recommended to replace the default certificate with an official one, signed by your Certificate Authority. To install a signed certificate, follow this procedure:

• Use the openssl.exe executable to generate a valid key file on the Exivity server by executing the following command:

openssl.exe genrsa -out exivity.key 4096

• Run the following command to create a certificate signing request file:

openssl.exe req -new -key exivity.key -out exivity.csr -addext "subjectAltName = DNS:example.com"

• You will be asked to enter general information like company name, city, etc. It is important to include the FQDN of the Exivity server when asked for Common Name (e.g. server FQDN or YOUR name) []:

• The generated CSR file should be sent to your Certificate Authority. After processing by your CA, you should receive back a .crt file. Rename this file to webcertificate.crt and rename your exivity.key to webcertificate.key and copy it to the directory %EXIVITY_PROGRAM_PATH%\server\nginx\conf. This should overwrite the existing .key and .crt files.

• Restart the Exivity Web Service Windows service to activate your signed certificate.

Configuring a separate web server portal

In some environments it may be desirable to separate the webserver from the backend components. This can be achieved by installing two separate Exivity instances. One instance could be placed in a DMZ and the second instance would then typically be deployed within a local network as shown in the following diagram:

Make sure to replace HOSTNAME_BACKEND_PORTAL with the actual hostname or IP address of the system that serves as your Exivity Backend Portal.

After completing your installation, you should now be able to access your Exivity User Portal.

Using an Internet proxy when extracting data

In cases where Exivity instance requires internet connectivity (i.e. to obtain Azure or AWS consumption data), and your network infrastructure requires use of a proxy server, it is necessary to configure a system environment variable.

Right click on This PC in an Explorer Window and click on Properties:

Then go to Advanced System Settings, then click the Environment Variables button:

Now add a new System Variable with the name ALL_PROXY and fill in the address of your proxy server as the value for this variable:

In case you do not want to use the proxy for certain address or domains, it is also possible to add an additional variable NO_PROXY:

After confirming the change, make sure to restart both the Exivity Windows Services.

Increase limit of filesize upload in the API

Enable PDF report exports

Depending on your installation method chrome.exe should be installed in one of those directories:

• C:\Program Files\Google\Chrome\Application

• C:\Users\{username}\AppData\Local\Google\Chrome\Application

Restart the Exivity API service after making these changes.

Introduction

Apart from installing Exivity in any on premise environment, Exivity can also be deployed from the Azure Market Place (AMP). Deploying Exivity on AMP is straight forward, and can be finised within a few minutes via your Azure Portal.

Azure Marketplace Offering

Once you've selected the Exivity offering, you should be presented with the following screen:

After clicking the Create button, you will be redirected to the VM deployment wizard

Deployment Wizard

1. Fill in a Windows user/pass and pick your deployment Resource Group:

1. You may select any additional options, but none are required for running Exivity succesfully, so you may skip this page simply by clicking the OK button:

1. Review the summary and click Create to deploy your Exivity VM:

This may take a few minutes. You may review the status of the Virtual Machine in your VM list:

Write down the Public IP address once it is available. Optionally you may configure a custom DNS name to have an easy way to connect.

Connecting to your Exivity instance

You can logon to your Exivity instance with RDP, but after deployment you should be able to connect to your instance using the public IP address or DNS name of your Exivity instance considering the following default URL:

• https://<Your_Public_IP>:8001

The default admin username is admin with password exivity.

Next steps

Exivity is a metering and billing software solution for public and private cloud environments that allows you to report on cloud consumption from any IT resource. Exivity enables you to apply your MSP/CSP business rules and makes any type of Pay-as-you-Go model work. It also facilitates internal charge-back and show-back requirements for Enterprise IT.

These things are done by extracting IT consumption data from various endpoints and then mapping this data to meaningful customer-specific information such as services, customer IDs, names and contracts.

There are four main steps involved in a successful deployment:

1. Extract

2. Transform

3. Report

4. Integrate (optional)

Extract

The Extract step defines your data sources such as:

• APIs that return usage data, service catalogue, rate card, customer/subscriber lists and similarly available records from public or private clouds

• APIs or ODBC queries that return contracts, customer names, IDs and other contextual lookup data from CMDB / CRM systems

• Flat files on disk in CSV, JSON or XML format

• Other HTTP/S sources

Exivity provides a rich scripting interface via its Unified Scriptable Extractor (USE) component which facilitates integration with almost any data source. For most of the big cloud platforms we provide template extractor scripts as part of the product. Additionally you can also write your own USE scripts from scratch in order to integrate with custom data sources.

Transform

This is done using the Transcript component, which executes user-definable scripts (termed tasks) in order to produce a meaningful set of data suitable for reporting against. Often this data will feed a consolidated bill of IT based on the various different consumed services.

Transcript also allows you to define and populate services and rates, either of which may be passed through from cloud data, defined as custom offerings or a mixture of the two.

Report

Exivity provides a modern responsive User Interface, that allows you to 'slice and dice' the processed data in any way you choose. Multiple Report Definitions can be created with ease which allow you to graphically and textually display both cost and usage statistics.

Integrate

We think that Exivity should be part of your automation landscape, where it can provide (for example) line items that can be digested by your ERP and/or invoicing system. Therefore we consider Integrate as the logical final step for any deployment where it is useful.

To this end we offer an open and fully featured REST API. Our GUI uses our own API for all back-end processes meaning that all textual data shown in the Exivity GUI is also obtainable via our API.

Pre-requisites

Please note that we also need the following requisites:

• S3 bucket where the CUR reports are placed

• The AWS region of the S3 bucket

• The path where the reports are placed inside the S3 bucket

To get this requirements you can go to the Amazon S3 service - Buckets and note the name and the region of our CUR S3 bucket

Once inside the CUR bucket, you must find the path where the monthly reports reside, please navigate the bucket until you find a folder like this one and note the path:

Obtaining the API Credentials

We have to create an IAM User that has read access to the CUR bucket. Please follow these steps:

• Go to IAM - Policies and click on Create policy.

• In the next screen: Service select S3, Actions select Read, Resources select Specific and in bucket click on Add ARN.

• In the next screen, type the name of your bucket followed by /* in Bucket name (example: billing-bucket/*) and click on Add.

• Click on Review Policy.

• Give a Name and a Description to the policy and finally click on Create policy.

Now, we will attach the policy to a user:

• Go to IAM – Users – Add user.

• Add an User name and select Access type: Programmatic access.

• Click on Attach existing Policies directly and filter by the name of the policy you just created.

• Select the policy and click on Next:Tags.

• Add a Key-Value pair for the Tags and click on Next:Review.

• Click on Create User.

• In the next screen click on Download .csv we will use this credentials to populate our Exivity Extractor in the next section.

• Finally click on Close.

Configure Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For AWS, please click AWS_CUR_Extractor from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to first tab: Variables

• Bucket: Type the name of your S3 CUR bucket.

• Access key: You can find it in the csv that you have downloaded in the previous section.

• Secret key: You can find it in the csv that you have downloaded in the previous section.

• Cur report: Type the name of your CUR report.

• Bucket Directory: Type the path that you got in the previous section (if there is more than one folder separate them with /).

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires one parameters in yyyMMdd format:

• from_date is the date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your AWS CUR Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button. Select the AWS CUR Transformer and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

Create Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

1. Create an Exivity Enterprise Application in your Azure AD for authentication

2. Configure a rate card lookup file

3. Configure an Extractor

4. Configure a Transformer

5. Create your Report

Creating an Enterprise Application

In order for Exivity to authenticate with Azure Stack, you will need to create an application in the Azure AD where you have registered your Azure Stack management node:

When you create this application in your Azure AD make sure it has (at least) the Reader Role in your Default Provider Subscription:

Create a rate card

%EXIVITY_HOME_PATH%\system\extracted\AzureStack\rates\azure_stack_example_rates.csv

Configure Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Azure Stack:

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, go to first tab: Variables

Fill in all required variables marked within the red box in above screenshot. If you don't know some of the required GUIDs, most of these can be obtained by browsing to the Azure Stack management node URL:

• https://adminmanagement.**<your.domain.com>**/metadata/endpoints?api-version=2015-01-01

Another way to obtain some of this information is using the Diagnostics button in your management portal:

When you click the Show Diagnostics link, it should download a JSON file containing most of the parameters you'll need, such as Provider GUID, Audience GUID etc.

Once you've filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyyMMdd format:

• from_date is the date for which you wish to collect consumption data

• to_date should be the date immediately following from_date

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your Azure Stack Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

An introduction to Services

In Exivity Services can be anything that corresponds to a SKU or sellable item from your Service Catalogue. It should relate to a consumption record (or multiple records) from your extracted data sources.

For example: with most public cloud providers, the provider defines the chargeable items that are shown on the end of month invoice. However, when working through a Managed Services Provider, a Cloud Services Provider or a System Integrator, additional services can be sold on top of those. Potentially, you may want to apply an uplift to the rate or charge a fixed amount of money every month for a certain service. Different scenario's are possible here, it all depends on your business logic.

Terminology

A service is a named item with associated rates and/or costs used to calculate a charge that appears on a report, where rates represent revenue and costs represent overheads.

When discussing services and their related charges a number of terms are required. Exivity uses the following terminology in this regard:

Creating service definitions

If the task does not complete successfully then the service definitions cached in memory are discarded, the expectation being that the task will be re-run after the error condition that caused it to fail has been rectified and the services will be written to the global database at that time.

Types of charges

There are different types of charge that can be associated with a service. Collectively these influence the total charge(s) shown on the report and Exivity supports the following charge types as described in the Terminology table above:

• unit rate

• fixed price

• COGS rate

• fixed COGS

At least one of these charge types must be associated with a service definition. Multiple combinations of the charge types may also be used.

Once the resulting charge has been calculated based on the charge types, it may be further modified through the application of adjustments, proration and minimum commit (all of which are detailed later in this article).

Charge intervals

In order to calculate the charge(s) associated with usage of a service Exivity needs to know the period of time for which each payment is valid. For example a Virtual Machine may have a daily cost associated with it, in which case using it multiple times in a single day counts as a single unit of consumption whereas Network Bandwidth may be chargeable per Gigabyte and each gigabyte transferred is charged as it occurs.

The charge interval (also termed simply interval) for a service can be one of the following:

• individually - the charge for a service is applied every time a unit of the service is consumed, with no regard for a charging interval

• daily - the charge is applied once per day

• monthly - the charge is applied once per calendar month

Charge models

Monthly services may be charged in one of two ways:

Peak

For each day of the month, a 'candidate' charge is calculated using Quantity * Unit Rate . The monthly charge will reflect the day of the month which resulted in the highest charge.

In the event that multiple days share the same highest charge then that charge will be associated with the first of those days seen, unless a subsequent day in that set has a higher quantity, in which case the charge will be associated with that subsequent day.

Average

The average unit rate for those days where usage was seen in the month is calculated, and multiplied by the average quantity for each day in the month. When calculating the average quantity, any days for which there was no consumption are factored in as having a quantity of 0.

Minimum commit

The minimum commit is the minimum number of units of consumption that are charged every interval, or (in the case of services with an interval of individually) every time the service is used. If fewer units than the minimum commit are actually consumed then the service will be charged as if the minimum commit number of units had been used.

Proration

After the charge for usage of a monthly service has been determined, it may be prorated by modifying that charge based on the frequency of the usage.

This process will reduce the charge based on the number of days within the month that the service was used. For example if consumption of a service with a monthly charge interval was only seen for 15 days within a 30 day calendar month then the final charge will be 1/2 of the monthly charge.

Service definitions

A service definition comprises two categories of information:

1. The service - Metadata describing fixed attributes of the service such as its name, description, group, interval, proration and charge type(s)

2. The rate revision - Information detailing the charge type(s) associated with the service (the rate, fixed price, COGS rate and fixed COGS values) and additional information detailing the date(s) for which those values should be applied

The following tables summarize the data members that comprise each of these categories:

Service attributes

Rate revision attributes

The rate_col, fixed_price_col, cogs_col and fixed_cogs_col fields are used when the specific value to use is derived at report-time from the usage data, as opposed to explicitly being included in the rate revision itself.

A service may have any number of associated rate revisions so long as they have different effective_date or minimum commit values. This means that a service can have different charges applied depending on the date that the report is to be generated for, or depending on the specific values in the columns used by a report.

A service may use neither, either or both of rate and fixed_price, and neither or one of cogs and fixed_cogs. At least one of rate, fixed_price, cogs or fixed_cogs is required, but cogs and fixed_cogs may not both be used.

Relative quantity discounts can cause negative charges with prorated monthly services

Description

When configuring a prorated monthly service, and then applying a relative quantity discount (adjustment), in certain cases the total charge can be less than 0 resulting in a credit. Although this might be an unusual configuration and might not always happen, it is important to be aware of this behaviour.

Status

This issue is pending. (internal reference: EXVT-1337)

Pre-requisites

Please note that in order to deploy this solution the S3 bucket to which CUR reports are written must reside in one of the following AWS regions:

• Northern Virginia

• Ohio

• Oregon

• Mumbai

• Seoul

• Singapore

• Sydney

• Tokyo

• Frankfurt

• Ireland

• London

At this point in time, only the previous regions have all the necessary services deployed.

Introduction

This tutorial shows how to build a serverless solution for querying the AWS CUR Report using Exivity. This solution makes use of AWS serverless services such as Lambda and Athena, as well as other commonly used services such as S3, CloudFormation, and API Gateway. The following topics will be covered:

1. Solution Overview

2. Launching the CloudFormation Template

3. Creating the Lambda function and API Gateway

4. Configuring an Extractor

5. Configuring a Transformer

6. Creating your Report

Solution Overview

The Billing and Cost Management service writes your AWS Cost and Usage report to the S3 bucket that you designated when setting up the service. These files can be written on either an hourly or daily basis.

The CloudFormation template that accompanies this tutorial builds a Serverless environment containing a Lambda function which reads a CUR file, processes it and writes the resulting report to an output S3 bucket. The output data object has a prefix structure of "year=current-year" and "month=current-month". For example, if a file is written 13/09/2018 then the Lambda function outputs an object called "bucket-name/year=2018/month=09/file_name".

The next step in the template is to translate this processed report into Athena so that it can be queried. The following diagram shows the steps involved in the process:

Afterwards, we will create a Lambda function to query the Athena database, returning a URL with the results of the query in CSV format. We will also create an API EndPoint with the AWS API Gateway service, which is used by Exivity to retrieve the data.

Launching the CloudFormation template

In order to deploy this solution successfully the following information is required:

1. The name of your AWS Cost and Usage report.

2. The name of the S3 bucket in which the reports are currently stored.

Firstly, launch the CloudFormation template that builds all the serverless components that facilitate running queries against your billing data. When doing this, ensure that you choose the same AWS Region within which your CUR S3 bucket is located.

Now follow the instructions in the CloudFormation wizard, using the following options, and then choose Create.

• For CostnUsageReport, type the name of your AWS Cost and Usage report.

• For S3BucketName, type a unique name to be given to a new S3 bucket which will contain the processed reports.

• For s3CURBucket, type the name of the bucket into which your current reports are written.

While your stack is building, a page similar to the following is displayed.

When the Status column shows CREATE_COMPLETE, you have successfully created four new Lambda functions and an S3 bucket into which your transformed bills will be stored.

Once you have successfully built your CloudFormation stack, you can create a Lambda trigger that points to the new S3 bucket. This means that every time a new file is added to, or and existing file is modified in, the S3 bucket the action will trigger the lambda function.

Create this trigger using the following steps:

• Open the Lambda console.

• Choose Functions, and select the aws-cost-n-usage-main-lambda-fn-A Lambda function (note: do not click the check box beside it).

• There should be no existing triggers. Choose Trigger, Add trigger.

• For Trigger type (the box with dotted lines), choose S3.

• Select the S3 bucket withing which your CUR reports are stored.

• For Event type, choose Object Created (All) and check Enable trigger.

• Click Submit.

The database and table are not created until your function runs for the first time. Once this has been done, Athena will contain the database and table.

Athena stores query results in S3 automatically. Each query that you run has a results file in CSV format and a metadata file (*.csv.metadata) that includes header information such as column type, etc.

Testing (Optional)

Once you have successfully added the trigger to the S3 bucket in which the Billing and Cost Management services writes your CUR reports, test the configuration using the following steps.

• In the S3 path to which AWS writes your AWS Cost and Usage Billing reports, open the folder with your billing reports, open the folder with your billing reports. There will be either a set of folders or a single folder with a date range naming format.

• Open the folder with the data range for the current month. In this folder, there is a metadata file that can be found at the bottom of the folder. It has a JSON extension and holds the S3 key for the latest report.

• Download the metadata file. Ensure that the name of the file on your machine is the same as the version stored on your S3 bucket.

• Upload the metadata file to the same S3 path from which you downloaded it. This triggers the Lambda function aws-cost-n-usage-main-lmbda-fn-A.

• In the S3 bucket that you created to hold your processed files, choose the "year=" folder and then the "month=" folder that corresponds to the current month. You should see the transformed file there, with the time stamp that indicated that it was just written.

Creating the Lambda function and API Gateway

To automate this process a CloudFormation template will be provided. This template will create an IAM role and Policy in order that our API can invoke Lambda functions. Then it will create a Lambda function with the capabilities of querying our previously created Athena Serverless DB, and save the output in a S3 bucket in .csv format, (this output will be later retrieved by Exivity). Finally, it will deploy an API Gateway allowing us to create an endpoint for our Lambda function, this is the endpoint that the Exivity extractor will consume. Make sure to launch the CloudFormation template in the same region that you have deployed the previous one.

Let's start by downloading the CloudFormation template (you only need to choose one of the formats, both are supported by AWS):

Then follow the next steps:

• Go to the CloudFormation console.

• Choose Create Stack.

• Choose Upload a template to Amazon S3.

• Select from your computer the template that you have downloaded.

• Follow the CloudFormation wizard - Add a Name to the Stack and select I acknowledge that AWS CloudFormation might create IAM resources with custom names in the last step.

• Once the stack is created you should see an CREATE_COMPLETE message.

• Click on Output to take a note of your endpoint (you will need to input this in the Exivity extractor).

Next, we will associate an API Gateway trigger to our Lambda function:

• Go to the Lambda console.

• Choose the QueryAthena2 function.

• Under Add Triggers select API gateway. You should see an image like the following:

• Click on API Gateway figure to configure it.

• On API select QueryAthena2.

• On Deployment Stage select v1.

• On Security select Open.

• Choose Add.

• Choose Save.

You should see a screen like this:

Finally, we will deploy the API Gateway:

• Go to the API Gateway console.

• Choose QueryAthena2.

• In the Resources section, click on the ANY method.

• In Actions, choose Delete Method.

• Click on Delete.

• In the Resources section, choose Actions.

• Click on Deploy API

• In Deployment Stage select V1.

• Add a Deployment Description.

• Choose Deploy.

Securing the API Gateway

Initially, the created API endpoint is public and as such is vulnerable to the possibility of misuse or denial-of-service attacks. To prevent this, associate an API Key with the endpoint as per the following steps:

• Inside the API Gateway dashboard, select the QueryAthena2 API

• In Resources, select Method Request

• In Settings, change API Key Required to True

• Click on Actions and choose Deploy API to effect the change

• In Deployment Stage, select v1 and click on Deploy

• Go to the API Keys section

• Click on Actions and select Create API Key

• In Name write ExivityAPIKey

• Click on Save

• Copy the API Key, as this will be required by the Exivity configuration

• Go to Usage Plan

• Click on Create.

• In Name write ExivityUsagePlan

• In the Throttling Section, change Rate to 100 and Burst to 10

• In the Quota Section, change it to 50000 requests per Month

• Click on Next

• Click on Add API Stage

• In API, select QueryAthena2 and in Stage select v1

• Confirm the changes and click on Next

• Click on Add API Key to Usage Plan

• Select ExivityAPIKey, confirm the changes

• Click on Done

The API Key is now required to access the API endpoint thus adding a layer of security to mitigate unauthorized access attempts.

Configure Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For AWS, please click AWS_CUR_Extractor from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to first tab: Variables

• In the Bucket variable specify the name of the S3 bucket where the .csv with the output of the query will be saved (The S3BucketName previously specified when launching the CloudFormation template).

• In the Api endpoint variable specify the API endpoint previously created plus the route /QueryAthena.

• In the DBname variable specify the name of your DB, you can find it in the Athena main Dashboard.

• In the Tablename variable specify the name of the table inside your DB, you can find it in the Athena main Dashboard.

• In the API_Key variable specify the API Key that we have created in the Securing API Gateway Section.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to collect consumption data.

• to_date should be the date immediately following from_date.

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your AWS CUR Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button. Select the AWS CUR Transformer and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

Create Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

1. Create an Access Key and Secret in your Azure EA portal

2. Configure the Azure EA Extractor

3. Configure your Azure EA Transformer

4. Create a Report definition

Creating an Access Key & Secret

Just under the Windows logo in the left menu section you will find your enrollment number which you will need to provide later when configuring the Extractor. To create the Access Key & Secret, click on Download Usage and then on API Access Key. This brings you to the menu where you can manage your Access Keys and corresponding secret which you will need in order to configure the data Extractor.

Configure Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Azure EA:

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, go to first tab: Variables

Fill in all variables in above screenshot, and feel free to encrypt any sensitive data using the lock symbol on the right.

Once you've filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyyMMdd format:

• from_date is the date for which you wish to collect consumption data

• to_date should be the date immediately following from_date

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your Azure EA Blob Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Pre-requisites

Setting up BigQuery

First and foremost, Exivity will create a dataset containing the billing table which will be periodically queried. Please follow these instructions:

2. Select the project that will contain your dataset in the project drop down.

3. Click CREATE DATASET.

   1. Enter a Dataset ID

   2. Select Data location (region)

   3. Set Default table expiration to Never

   4. Set Encryption to Google-managed key

4. Finally, click Create dataset

5. Note the Project, BigQueryProject and Table, these parameters will be used by Exivity on a later stage.

Once you have the dataset created in BigQuery you need to enable Cloud Billing export to BigQuery:

2. Go to the Billling Export tab.

3. Click Edit Settings to enable the export.

   1. Select the project where you previously created the BigQuery dataset.

   2. Select the specific dataset from the Billing export dataset list.

   3. Click Save.

Creating Exivity Service Account

Configuring the Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Google Cloud, please click Google_Cloud from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to Variables tab and fill the parameters:

• Hostname: Default endpoint for Google BigQuery with version

• Private: Provide the RSA private key PEM format

• Email: Mail address / service account associated with the private key, obtained in the Creating Exivity Service Account section.

• Project: Main GCP Project name.

• BigQuery project: GCP BigQuery Billing Project, obtained in the Setting up BigQuery section.

• BigQuery table: GCP BigQuery Billing Table, obtained in the Setting up BigQuery section.

Finally, click on Update.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to start collecting consumption data.

• to_date is the end date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure a Transformer

Once you have successfully run your Google Cloud Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button.

Select the Google Cloud Transformer that you just created, go to the Run Tab and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

When you click the Run Now button, you should get a successful result.

Create your Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

The interface allows you to do the following:

• Develop extractors

• Create transformer (ETL) tasks

• Configure report definitions

• Run graphical usage & costs reports

• Run textual usage & costs reports

• Manage users & roles

• General configuration

• White labeling

More features are added on a regular basis.

Introduction

1. Create Exivity vCenter user

2. Configure an Extractor

3. Configure a Transformer

4. Create your Report

Exivity supports Out of the Box integration with vCenter version 6.5 and higher. For version before 6.5, please contact support@exivity.com.

Create Exivity vCenter user

Exivity needs a user with a reader role in order to retrieve consumption data. Please follow this procedure:

• In your vCenter Configuration Manager go to Configuration > Local Users and Groups > Users.

• Create a new User, insert username and password.

• Take note of the username and password, they will be used later on to configure Exivity

• Click on Create.

• Under Users and Groups, click Add, select the previously created user.

• On the Select Users and Groups dialog, click Add, and then click OK.

• Add a Reader role in the dropdown.

• Finally click on OK.

Configure an Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. To create a new Extractor based off the vCenter template follow these steps:

• Provide a name for the Extractor in the Name field above

• Pick vCenter 6.5 (VM Inventory REST API) template from the list

• Click the Create button.

Once you've created the Extractor, go to theVariables tab:

Fill in all required variables with the values that you gathered in the previous step. You have the option to encrypt a variable in case it contains sensitive information (i.e. password) by clicking the lock icon on the right of each variable field. When finished, click Update.

Once you've filled in all details, go to the Run tab to execute the Extractor clicking on Run Now:

If the variables are correct and your vCenter is reachable for Exivity, you should get a successful result.

Configure a Transformer

Once you have successfully run your vCenter Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day (today) as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select your vCenter dataset, and your preferred Reporting Columns to break down the report (we recommend only cluster_name for the default report). When you are ready, click on Create.

Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

PostgreSQL

Windows Services

In version 2.x.x there were only two Exivity Services installed:

With Exivity version 3.x.x, assuming all components are installed on a single host system, there will be total of 4 different services:

In case you were using a service account in Exivity version 2.x.x for the Exivity Scheduling Service and the Exivity Web Service, you will have to reconfigure this service account for both services, as well as the Exivity API Service. In most cases, the Exivity Database Service may continue to run un the Local System account.

Default TCP ports

In v2.x.x, the default port for the Exivity GUI was 8001 and 8002 for the Proximity API. Both services were already available through port 443 (the default port used for https traffic, which means clients don't have to explicitly specify the port) and in v3.x.x this will be used by default:

Default Security Settings

As of Exivity version 3 more strict security settings are applied by default. These can be found under Administration > Settings. One important item which should be considered when upgrading a multi node environment, is the use of CORS. It is required to list all possible front end UI nodes in the CORS origins field:

Transformer changes

@FILE_EXISTS and @FILE_EMPTY

Previously, it would only check for files in the system and exported folders within the Exivity home directory, and if a specified path + filename did not start with system/ or exported/ then these would be prepended automatically before the check as done.

This behaviour has been changed in the following ways in v3.x.x:

• Any path + filename within the Exivity home directory can now be checked

• path + filenames are now accepted and treated as being relative to the Exivity home directory

• The folders system/ and exported/ are no longer automatically prepended

Consider a file somefile.csv in the %EXIVITY_HOME_PATH%/exported folder. Previously with version 2.x.x a user could check for the existence of this file using the following example:

In version 3 it is required to include the entire path relative to the %EXIVITY_HOME_PATH% :

Extractor changes

HTTP server certificate validation

The default behaviour of the HTTP subsystem in USE was changed to fully validate server SSL certificates, which may cause some USE scripts to fail. This typically applies to data Extractors, which are connecting to on premise data sources that use self signed SSL certificated. In version 2.x.x, the default behavior was to ignore these certificate errors. You can identify these errors in your Extractor logs:

Extractor & Transformer Schedules

GUI changes

There are some minor changes in the GUI which are not backwards-compatible:

• We removed the Excel export options from the reports. They were using the CSV format under the hood (i.e. they never actually produced valid Excel worksheets). In the future, we plan to implement proper Excel export formats for the reports, including a full summary report Excel export.

Changes to data processing for reports

Due to changes to the processing of reports, when making changes to either services, rates or adjustments, associated reports should be prepared. In v2.x.x this was already required when making changes to services or rates. Since v3.x.x this will be also required when making changes to adjustments. We've planned further improvements making this more transparent (i.e. handling preparation of reports automatically in the background).

API changes

Normalised date/time data in responses

Some endpoints were returning dates and timestamps in different formats. This has been normalised in such a way that all responses use the same serialisation for dates and timestamps:

• A date is always represented as ISO-8601 string: "yyyy-mm-dd", e.g. "2020-01-29"

• A date/time is always represented as ISO-8601 string in UTC: "yyyy-mm-ddThh:mm:ssZ", e.g. "2020-01-29T11:26:52Z". Note that the Z suffix denotes the UTC time standard.

This affects the responses (attributes) for the following group of API endpoints:

• /v1/audit

  • created_at

• /v1/budgetrevisions

  • effective_from

• /v1/dsets

  • earliest_rdf

  • latest_rdf

  • rdf_detail.created

  • rdf_detail.updated

• /v1/extractors

  • last_modified

• /v1/log

  • created

  • lines.date

• /v1/reports

  • data_status.first_date

  • data_status.last_date

  • data_status.status.date

• /v1/services

  • created_at

  • updated_at

• /v1/transformers

  • last_modified

• /v1/workflows

  • created_at

  • updated_at

• /v1/workflowschedules

  • start_time

  • next_run

Changed functionality

/v1/reports/{id}/run endpoint

• The JSON format in the response is simplified. See examples below.

This is an example response from v2.x.x:

This is an example response from v3.x.x:

Remove deprecated functionality

/v1/usergroups endpoint

Removed deprecated permission aliases.

• upload_files (use manage_files instead)

• manage_configuration (use manage_settings instead)

• manage_system (use manage_settings instead)

/v1/reports/{id}/run endpoint

Functionality deprecated in v2.x.x has been removed. If you were relying on any of the following functionality, please use the suggested replacement instead:

• The pdf/invoice option for the format query parameter has been removed. Please use pdf/summary instead.

• The invoice_options query parameter has been removed in favour of summary_options.

/v1/configuration endpoint

• Configuration keys prefixed with INVOICE_ are replaced by respective keys prefixed with SUMMARY_.

/v1/workflowsteplogs endpoint

Also relevant to other endpoints including workflowsteplogs .

• Removed the last_log attribute. Include the last log by specifying the query parameters include=steplogs&related[steplogs][limit]=1&related[steplogs][sort]=-start_ts.

• Removed the timestamp attribute. Use start_timestamp instead.

• Removed the message attribute. Use a combination of status and output instead.

/v1/file endpoint

Only applicable to POST requests to this endpoint.

• The filename in the response from this endpoint will no longer include the import/ prefix to better align for other requests in this endpoint. See example below.

/v1/usergroups endpoint

The following permissions has been removed in favour of their new counterparts:

• UPLOAD_FILES has become MANAGE_FILES

• MANAGE_CONFIGURATION has become MANAGE_SETTINGS

• MANAGE_SYSTEM has become MANAGE_SETTINGS

Install / upgrade

Release feed

Changelog

v3.4.3

March 22, 2021

New features

v3.4.2

February 24, 2021

Bug fixes

• Fixed an issue with SAML user account access provisioning Previously, user account access provisioning would grant access to a matching account, including all its child accounts. With this release, the SAML user account access provisioning only grants a user access to the first matching account (i.e. with the lowest depth) in the account hierarchy.

• Fixed an issue where reports could become invalid for certain user account permission configurations

• Revolved a race condition related to preparing a report On rare occasions the actions of multiple users administering accounts and reports simultaneously could result in duplicate charges in reports. This has now been fixed.

• Fixed a very rare issue with minimum commit Resolved an issue whereby it was possible on extremely rare occasions for the minimum commit calculations applied to one service to be incorrectly applied also to other services in a report.

v3.4.1

December 08, 2020

Bug fixes

• Fixed an issue with PDF export of the summary report In certain environments, the PDF export feature was broken. This issue has now been resolved.

• Default SAML user group re-applied when a user logs in An issue has been resolved which caused the default user group to be re-applied when existing users were logging in through a SAML Identity Provider.

v3.4.0

November 17, 2020

New features

• Executing a Reports now requires a valid license

Bug fixes

• Fixed a minor issue with the calendar widget

• Fixed a minor issue with the SAML SLS endpoint

• Fixed an issue with custom rates and subscriptions

  Sometimes a subscription would not reflect changes in custom rates for a service. This has now been fixed.

v3.3.0

November 02, 2020

New features

• Improved user provisioning (beta) Added options to provision user attributes from a SAML Identity Provider/AD response. It's possible to provision the users display name, username and email address.

• Added claims based account access provisioning (beta) Added options to provision users permission levels using attributes from a SAML Identity Provider/AD. It's possible to provision both the usergroup and user account access (by matching either the account key or metadata value to a SAML/AD response attribute).

Bug fixes

• Change to log filenames for extractors and transformers When scheduling extractors and transformers with workflows, the selected environment is now part of the log filename to distinguish their log files when they are running at the same time.

• Fixed opening curly brace detection When opening curly brace wasn't preceded by space, sometimes it wasn't properly processed. It has been fixed

• Improved error message on lookups screen Fixed an issue where a user might get a confusing error message when saving a lookup file.

• Fixes initializing directories in home directory There was a bug when the home directories were initially created and two directories were wrongly created in the /home directory.

• Improved support for UTF-8 in usernames

v3.2.7

October 09, 2020

New features

• Added support for the Safari web browser

  Exivity now supports the Safari web browser

• Added ability to set custom escape character in Transformer Previewer Added a dropdown to the transformer previewer where the client can select which escape character to use during Transformer preview mode

Bug fixes

• Fixed an issue with the Dataset Manager Resolved regression issues regarding the display of RDF dates

• Fixed an issue with report filters There was a bug if a user accidentally submits a string instead of a number on filtering 'parent_account_id' with report. This is now being fixed

• Fixed an issue with displaying workflow start times in the workflow list The time column of the workflow list omitted the hour at which the workflow would run for daily and monthly workflows

• Resolve global variables in "uri" statement In some statements (i.e. uri) global variables weren't properly resolved. This has now been fixed

• Fixed an issue with COGS charges for services with an average monthly charge model COGS charges for monthly services that used the average charge model could be slightly lower than they should be, as the COGS rate for the first day seen was not factored into the rate averaging. This has now been fixed

• Fixed an issue when managing rates An aggregation_level error could occur when updating rates. This has now been resolved.

v3.2.5

September 30, 2020

Bug fixes

• Improved error message when creating invalid revision When creating a duplicate rate revision for the same effective date, an appropriate error message is now shown

• Improved error message for LDAP Added a more meaningful message when unable to connect to an LDAP server

• Service Category names must be unique It was possible to have duplicate service category names. This could lead to confusion and has now been resolved. The Service Category name must now be unique

• Resolved a timezone offset issue for Workflows Resolved an issue where timezones were incorrectly applied when creating workflows

• Resolved an issue with the Reports menu Fixed an issue where the report screen would crash after selecting multiple reports

v3.2.4

September 9, 2020

Bug fixes

• Increased the boundaries of some USE arithmetical operations

  The operators +=, -=, *= and /= were limited to a 32-bit range and this could cause an overflow in certain real-world applications. The range has now been increased to match that of the other arithmentical operations which is based on the range of a 64-bit floating point value: 1.7E +/- 308 (up to 15 digits).

• Fixed an issue where the CSV export of the instance report could fail The CSV export of the instance report did not work when the usage column was included and some of the instances did not have any usage data

• One-off subscriptions should ignore the charge_day field Only recurring subscriptions require the "charge day" field, but it was required for all subscriptions. This has now been fixed

• Fixed a decimal precision issue with the timeline graph The Y-axis of the timeline sometimes displayed long floats. To resolve this the precision has been fixed to two decimals

• Fix for "space" as thousand separator It is possible to select different symbols as a thousand separator for large numbers shown to users. One of the options, space (" "), was not saving correctly. This has now been resolved

• Fix error handling transformer Fixed an issue where it was possible that a failed transformer error was not processed properly which prevented the transformer from giving the user feedback

• Added index to speed-up report preparation During report preparation, Edify executes several queries to the adj_lookup table. An index has been adding to this table to improve performance

v3.2.1

September 1, 2020

• Quantity and Charge Adjustment are now handled in strict order As of this release, quantity adjustments are applied first and before any charge related adjustments. This in turn enables the ordering of individual adjustment policies.

• Implemented translations settings (Beta) Implemented an automatic translation feature for Dutch, German, and French. This can be configured on a system or user level. Currently, this is released as a beta feature.

• Delete old Nginx log files Old Nginx log files will now be deleted by the garbage collector. This will help reduce disk space.

• Improve user error page Updated the error page to improve user experience. Customer logo will now be shown here, if it has been set.

• Adjustment API endpoint will now accept order In the past, there was no way to reorder adjustments. Users had to delete and recreate them to do this. This has now been solved. The order field now takes in an order number. If this is not used, the current functionality will still work, using the creating date instead.

• Reports can now include adjustment name The adjustment name can now be added to a report. This is useful for report users that do not have access to view the whole adjustment.

• Support in UI for changing the order in which Adjustments are applied Since it is possible to create multiple Adjustments for a single account, they may affect each other. It is therefore desirable to have the ability to control the order in which they are applied. As of this version, it is possible to change the order in which Adjustments are applied.

Bug fixes

• Fixed an issue when partial preparing manual services When partial prepare kicks in, it previously skipped manual services. This has now been resolve.

• Fixed SAML ACS invalid schema error Some users were experiencing an Invalid Schema bug when accessing the /v1/auth/saml/acs endpoint. This has now been fixed.

• Fixes "Nginx Log File" not found error The location of some log files could not be found. This has been resolved.

• Improved invalid token handling Changed how this error is logged. It will now be treated as a notice instead of a warning.

• Fixed an issue with running transformers

  In release 3.2.0 there was a problem running transformers from the GUI. This has now been fixed.

v3.1.5

July 21, 2020

Bug fixes

• Premature validation warning

  Resolved an issue where a user could get a premature validation error for certain input fields.

• Fixed an issue with the budget report

  With certain budget configurations, the budget report would display a message instead of the report even if the budget configuration was valid. This has now been resolved.

• Improved user message when execution time is exceeded

  Improved the message when the execution time of an Extractor, Transformer or Report is being exceeded

• Fixed an issue with budget leaf account distribution When configuring a budget for a budget leaf, the distribution setting will be forced to be of the type "shared"

• Fixed an issue when deleting services Previously it wasn't possible to delete a service if it still had any associated rates. This behavior has now been corrected so that services including its relationships will be deleted

• Fixed an issue with the Garbage collector exits code The Garbage collector would previously always return an exit code of 259 when invoked manually. This has been resolved

• Fixed an issue where subscriptions could show up on more than one report In some cases, a subscription (created for an account associated with a specific report) could be shown when a different report was run. This has now been fixed

• Fixed graph image export When exporting a graph chart (i.e. Pie, Bar, Line) in any of the reports (Accounts, Services Instances) the file download would not start. This has now been resolved

• Restored indicators for account access list Pencil-like indicators will inform the user where nested accounts have been selected. This functionality was unintentionally removed from the user management screen but has been placed back

• Horizon performance improvements Implemented two seperate changes to improve Horizon budget execution performance. (1) budget gets validated only once after a budget configuration change and (2) added a new database index to improve database query execution

• Always show decimal values Resolved an issue where some values would be displayed using their scientific notation

• Resolved issue when deleting services Fixed an issue where the user would get an error when making multiple delete requests for services

v3.1.2

July 3, 2020

New features

• Ability to skip database backup when updating When updating the software, the user can now skip the PostgreSQL database backup to improve update speed

Bug fixes

• Fixed an issue with the webproxy.conf proxy_pass URL

  When updating the software, the proxy_pass URL for the webproxy.conf NGINX configuration was always reverted back to https://127.0.0.1:8002. This has now been resolved

v3.1.1

June 28, 2020

New features

• Improved auditing when managing accounts manually Added a number of new audit points when performing manual account management

• Auditing token creation correctly logs client IP address when API is behind a proxy server

• Updating user profile information now requires providing the current password

• Implemented current password validation A user is now required to enter his/her current password before any changes to its profile are applied

• Added logging audit entries for metadata, service subscriptions, and budgets

Bug fixes

• Fixed an issue where some users could select budgets even if it would not contain any details

• Fixed an issue which sometimes caused the workflows screen to load for a long time A resource-intensive operation that ran whenever the workflow page is visited got removed

• getCUPRs function ported to PSQL getCUPRs function was changed from using SQLite global DB to PSQL global DB

• Mitigated a potential security issue (internal reference EXVT-3773)

• Fixed an issue were sometimes log files would appear malformed in the Transformer run tab

• Reverted the way a user deletes an RDF

• Fixed form behavior in rates screen

• Fixed "other" option behavior at services

• Fix report filter When switching reports, some of the filters were not reset and appeared unused while they actually held a value not related to the currently selected report. This made the reports appear empty. This has been resolved.

• When the API runs out of memory (possible for very large reports) it will respond with a descriptive message

• Incorrect display of minimum_commit_delta_charge and minimum_commit_delta_quantity When a minimum commit quantity was set for a service rate revision, the resulting charges could show up incorrectly by having the quantity and minimum commit delta swapped on the reports

v3.0.5

May 26, 2020

Bug fixes

• Fixed issue at /lookups where in specific cases an error was undefined

• Fixed issue where user was shown an incorrect validation warning When providing a number with more then one decimal, the user interface would show an error message. This has been resolved.

• Added cache check in partial preparation There was an issue that partial preparation code didn't check for cache table presence before analysis, which caused execution errors, and it has been fixed.

• Fixed escape option behaviour in 'import' There was an error that '\"' sequence was always treated as escaped quoted even when 'escape' option switched off (default), it has been fixed.

• Rate revision date never changes Transcript was trying to change rate revision date when it detects an attempt to create new revision with the same attributes but earlier date, which conflicted with database constraint, so it was removed in order to avoid execution errors.

• Fixed issue where line breaks were not shown When providing an address for the summary report with multiple new lines, only the first would be shown. This has been resolved.

• Fixed undefined tool tip at workflows The was a column which showed a tool tip with undefined as value. This has been resolved.

• Select a single date for transformer run After introduction of a new calendar, the single date selection was missing. This has now been resolved.

• Services and categories at subscriptions are now alphabetically sorted

• Fixed an issue where dataset columns were missing After repreparing a report, the columns associated with a dataset were not visible. This has been resolved.

• Fixed visibility newly created dataset When running a transcript, newly created datasets were only visible after refresh. This has been resolved.

• Show red underline when incorrect value for textarea input

• Show correct validation values when saving environment

  When saving an environment, we will now show the correct error messages for missing values.

v3.0.4

April 30, 2020

New features

• New feature: Workflows widget When Beta features are enabled, a new Workflow traffic-light widget will be shown on the main Dashboard page.

• Added more information to the summary report The summary report now contains the following additional columns: service_key, account keys (between 1 and 5 inclusive depending on the report), start_date, end_date. These columns are also included in CSV exports.

• Improved quoting in exported CSVs To avoid potential complications with Excel, any cell values in CSVs exported from Exivity that are not numbers and begin with any of the characters =,+,- or @ are now preceded with a single quote.

• A new setting has been added to allow users to stay logged in. If disabled (default), users need to log in each time they open Exivity in their browser.

• Administrators can now set the token lifetime. When the token lifetime expires, users need to log in again.

• Added ability for users to log out of all devices.

• Mitigated a potential security issue (internal reference EXVT-3457)

• Mitigated a potential security issue (internal reference: EXVT-3455)

• Mitigated a potential security issue (internal reference EXVT-3270)

• Implemented a new charge model for monthly services The charge for a monthly services may now be based on the usage of a specific day in the month.

• Improvements in report performance Reports are now pre-loaded in the database which speeds up report loading times in the GUI.

• Added support for LDAP authentication LDAP authentication was available as beta feature already, and is now generally available. A guide will be added to our documentation soon. Configuration options are available in the Settings screen (Single sign-on tab).

Bug fixes

• Fixed an issue with manage_metadata_definitions Granting the "Manage Metadata" permission is now possible when creating a security group

• Fixed an edge case for incorrect net value on accounts table Whenever cogs or charge had 0 as value the net value wouldn't be displayed correctly on the details table of the accounts report.

• Fixed an issue when creating a manual service Manual per unit cogs type services will not require cogs column

• Fixed invalid format issue metadata lists Previously empty lines we're persisted as list option. Options are now trimmed and validated before persisting.

• The API now requires additional attributes for certain service types

• Fixed a small issue with the metadata selector in the report management screen

• Removed toolbar from PDF export The Summary report PDF export included the toolbar on the top of every page of the PDF document. This has now been removed.

• Fixed an Extractor XML parsing issue USE could previously fail when iterating over an empty XML node inside a foreach loop. This has now been resolved.

• Fixed modified display issue in Accounts Overview When switching from Account Name to Account Key in the Accounts Overview screen, the modified state / pencil icon disappeared previously. This has now been resolved.

• Fixed an issue when scrolling in the Accounts Overview When having a large amount of Accounts in a report definition, the Accounts Overview screen could generate cosmetic glitches when scrolling

• Fixed a Transformer issue with skip_corrupted_records When import option skip_corrupted_records was set, import could fail if last column in the record is missing closing quote. This has been resolved.

• Fixed a Transformer issue with aggregate When using the aggregate max function, Transcript could pick the wrong value.

• Fixed a false warning about invalid COGS type in the logfile When preparing reports, in some cases a lot of warnings would appear in the logfile that state: Invalid cogs_type (0) in database for service ID nnn where "nnn" is a service ID. This was a false warning and could be ignored but looked concerning and could lead to larger logfiles. This issue has now been fixed.

• Removed "remainder" option from lowest level accounts when setting budgets When a budget is set for an account at the deepest report level, the "remainder" option is no longer shown, as there are no sibling accounts to share the budget with.

• Fixed an issue where (harmless) SQL errors could be logged when deleting services When deleting services, SQL errors could be present in the logfile for days where reports that reference those services had not been prepared. This has now been fixed.

The Accounts report provides the ability to drill down into your metered based IT consumption costs. The way you've created your Report, determines on what values you can zoom into.

Once you've logged into the system, go to "Reports" > 'Accounts'. Here are a few key parameters you can use to define how your report is generated:

Date Selection

The date selector is important to limit the scope of data you're focusing on.

First select the date range you are interested in. This can be a single month, a 3-month time period, half a year, a full year, or a custom date range.

Report

Your Exivity solution can have more then one report configured. If this is the case, you will need to select the appropriate report containing the data you wish to examine. An end user will only see reports listed that they have permissions to, and the first of those is automatically selected.

Drill Down, Services & Report Depth

After selecting a date range and report you can start drilling down into your data in a number of ways:

1. Moving your mouse over one of the accounts will reveal a toolbar as shown above. For each account, you have the option to click on the Drilldown control in that toolbar. This will do the following:

   1. Descends one level deeper into the report

   2. Updates your view of the data to reflect that deeper level

   3. Sets the 'Parent' filter to the account you selected to drill down on

• automatic per unit

• automatic per interval

• automatic per unit & interval

• manual per unit

• manual per interval

• manual per unit & interval

By default each service has a global rate configured, which will be applied to all accounts that consume this service. However, it is possible to use customer specific rates by overriding a service rate for one ore multiple accounts.

Edit global rate for a manual service

A manual service can have up to 3 rate values that can be changed: the unit rate, the interval money value and the COGS rate. To change these values, go the 'Services' > 'Rates' screen and click on the service name for which you want to change the global rate value:

To change the rate values of this service, consider the following:

1. Effective date is the date from when this rate is applied to the service. A service can have one or multiple revisions. You may add new rate revisions by using the Add Revision button. Existing rate revision dates can be changed using the Change Date option

2. The Per Unit rate value is the value that the service charged for, every (portion of) configured interval service. In this example, if this would be a daily service that is charged 1 euro per Gigabyte of database usage, and each day a 100 GB database is consumed, a value of € 100 will be charged per day (and € 3100 if used for entire month of December)

3. The Per Interval charge is applied every occurrence of the consumed service, regardless of the total consumed quantity of that service. In the previous example at 2, this would mean every day a value of 110 euro will be charged for a 100 GB database: 100 euro because of the standard rate + 10 euro for the fixed interval charge. Considering the same consumption for the entire month of December, the total charge for that month will be 31 x € 110 = € 3410

4. It is possible to configure a COGS rate for this service. This is applied the same way as the Per Unit rate

5. To delete an invalid or wrong revision, use the Remove Revision button. Do bear in mind you cannot delete the last rate revision for a service

7. If you are planning to make more changes to other services in the same report definition, use the Save Revision > Without Preparing option. This will avoid running the re-preparation several times, and allows you to start the re-preparation only after you've made all of the required rate changes.

Exivity enables you to create account specific rate adjustment policies. An adjustment policy allows you to apply a discount or a premium using one of these modifiers:

1. a certain amount of money (i.e. $ 100)

2. a certain quantity (i.e. 100 GB/hours)

3. a percentage (i.e. 10%)

This Adjustment can then be applied to a single service, multiple different services, or one or more service categories.

Create an Adjustment Policy

To create a new adjustment policy for an account, follow these steps:

1. From the menu on the left, select 'Services' > 'Adjustments'

2. Then select the Account from the list of accounts for which you want to create an adjustment policy

3. After selecting the account, click 'Add Policy', and provide a meaningful name for your policy in right screen where it says 'Adjustment name'

4. Provide the Start date, by selecting the initial month when this adjustment policy is applied

5. Provide the End date, by selecting the month when this adjustment policy will be discontinued. This is optional, since an adjustment policy can be applied permanently.

6. Select which Service or Service Category this policy is applied to. You are able to select multiple using the check boxes that are provided.

7. Select a Type for this adjustment. This can be either a Discount or a Premium

8. Select the Target, meaning: is this Adjustment targeting the total Charge or the total Quantity of the selected service(s)?

9. Select the Difference setting, to indicate an Absolute value (i.e. 100 units, or 100 dollars) or a Relative value (such as 10%)

10. Lastly provide the Adjustment value. In the example shown in the image above, there is a value of '10' provided in the Amount field, which will adjust the total charge with -10% given the provided parameters.

11. When you're done, click the Add Policy button. Your changes are now applied to all charge related reports.

Once logged into the system, go to Reports > Summary. From here you're able to generate different kinds of detailed costs reports, which can be used for billing, chargeback and showback.

Filters and Reporting Depth

Once you have selected your date range and chosen which report you to activate will be presented with the summary cost report as shown above. By default it will show all consumed services for an account. As accounts are heirarchical, this view will include consumed services for all children of the selected account. Therefore it is important that you select an appropriate Depth when running this report.

Grouping and display of Instances

The Summary report has a few options that you can turn on and off. These options allow you to tune the amount of detail shown and the grouping applied to it:

If you want to include a detailed grouping of Services and Service Categories, then ensure that the Services checkbox is enabled. The same goes for including Instance level information. The latter enables you to view resource level consumption data, such as the Virtual Machine hostname, Container or User Name.

It is possible to define a budget on any level within your organisation. This enables different audiences to monitor costs across different clouds. It also enables any business owner to set thresholds informing customers, departments or project owners when they are reaching their configured budget.

Create a budget

In order to create a budget, navigate to Accounts > Budgets. Then click the Create button to create a new budget. In this menu, a couple of items are presented:

Global options

Global options apply on the entire budget, and aply to the following items:

• Interval: determine whether the budget is applied Monthly, Quarterly or Yearly

• Apply to: a configured budget is by default applied to the total Charge of the configured Interval. It is however also possible to create a budget which is applied to the Cost of Cogs (COGS) instead

Revisions

A budget configuration can potentially change year over year, and therefore it is possible to create different budget revisions. Each revision can have the following settings applied:

• Revision start date: the start Month, Quarter or Year for this budget revision

• Filter by: typically a budget is applied to a single or multiple Accounts. It is however possible to add additional filtering on Service or Service Category. By applying Service based filtering, it is possible to limit the scope for a configured budget

Accounts

When a budget is created, it is possible to set a budget money amount for one or multiple accounts. In case each account for which a budget is set, also has 1 or multiple levels of child accounts, it is possible to control how the budget 'trickles down' the organisational structure:

• Account selection: it is required to select an Account from any level in your Report Definition. It is the possible to set a budget value (i.e. $100000) in the grey box next to the Account.

  • The same applies to any Child Accounts for which you may set/overwrite a budget. You may add a Child Account to the list, by click the green button left of the account name

    • NOTE: a child account may also be excluded from a budget, by clicking the Exclude checkbox right of the name of the Account

• Remainder: using the Remainder drop-down it is possible to control the distribution of the budget towards child accounts. The two options to pick by default are:

  • even: each child account will get an even amount of budget. Example :Consider a top level account 'ACME Corp' with a monthly budget set of $100.000. When 'ACME Corp' has 10 child 'Business Unit' accounts , each of these 'Business Units' will get an even amount of the budget: In this case this will be $10.000.

  • shared: when the distribution of the remainer is set to shared, the consumption of child accounts is ignored. As long as the total spendings of all child accounts does not go beyond the configured budget. Example: Consider a top level account 'ACME Corp' with a monthly budget set of $100.000. When 'ACME Corp' has 10 child 'Business Unit' accounts , each of these 'Business Units' combined should not use more then $100.000.

  • none: the option to not distribute any remainders is only applicable when overriding the budget percentage for each child account. This means it is required set a distinct budget percentage manually for each child account.

Changing a budget

An existing budget can be changed by navigating to Accounts > Budgets, and then clicking the budget which you want to change.

Changing an existing budget revision

Once a budget has been saved, you will be unable to change the start date unless you edit the Budget Revision:

Once you have enabled edit mode for an existing Budget Revision, you will be able to change the start date:

After making these changes you will need to save these by click the blue checkbox on the right. You can also cancel your changed using the blue x-sign, or delete the revision by clicking the red recycle bin.

Adding a new budget revision

Similar to changing a Budget Revision, it is possible to add a new revision that holds a different start date with revised budget plan. To do this, you will need to click the green + sign:

The 'Data pipelines' menu allows an admin of the Exivity solution to manage USE 'Extractors'. USE has its own language reference, which is fully covered in a separate chapter of this documentation.

Creating Extractors

To create a new USE Extractor, follow these steps:

1. From the menu on the left, select "Data pipelines" > 'Extractors'

2. To create a new USE Extractor where you want pull usage or lookup data from, click the 'Add Extractors' button

4. Provide a meaningful name for your USE Extractor. In the above example we're creating an USE Extractor for VMware vCenter 6.5 and higher. Therefore we call this USE Extractor : 'vCenter 6.5'

5. When you're done creating your USE Extractor, click the 'Insert' at the bottom of the screen

Edit and Delete Extractors

When you want to change or delete an existing USE Extractor, first select one from the list of USE Extractors that you want to change:

1. After you select your USE Extractor, you can change its variable values at the 'Variables' tab.

2. At the "Editor" tab you can make more advanced changes or delete the original USE-script. Such as:

   • changing existing API calls

   • changing csv output format

Run and Schedule Extractors

To test your USE Extractor, you can execute or schedule it directly from the Glass interface:

1. After you have selected the USE Extractor that you would like to run, click to the 'Run' tab next to the 'Editor' tab

2. Most Extractors require one or more parameters, usually in a date format such as 20171231. In this example, the USE Extractor requires two parameters: a from and to date

3. When you've provided the required run parameters, click 'Run Now' to execute the USE Extractor. After the USE Extractor has completed running, you will receive some success or failed message, after which you might need to make additional changes to your USE Extractor

4. Once you're happy with your output, you can schedule the USE Extractor via the 'Schedule' tab, which is located next to the 'Run' tab at the top of the screen.

5. USE Extractors can be scheduled to run once a day at a specific time. Also you should provide a from and (optionally) to date, which are provided by using an offset value. For example, if you want to use the day before yesterday as a from date, you should use the down pointing arrows on the right, to select a value of -2. If the to date should always correspond with yesterdays date, you should provide a value there of -1.

6. If your Use Extractor requires additional parameters, you may provide these as well in the 'Schedule with these arguments' text field.

7. When you're done with the schedule configuration, you may click the 'Schedule' button. In case you want to change or remove this schedule afterwards, click the 'Unschedule' button.

USE scripts are stored in <basedir>/system/config/use, and are ASCII files which can be created with any editor. Both UNIX and Windows end-of-line formats are supported but in certain circumstances they may be automatically converted to UNIX end-of-line format.

Statements

Quotes and escapes

By default, a space, tab or newline will mark the end of a word in a USE script. To include whitespace in a word (for example to create a variable with a space in it) then double quotes - " - or an escape - \ - must be used to prevent the parser from interpreting the space as an end of word marker. Unless within double quotes, to specify a literal tab or space character it must be escaped by preceding it with a backslash character - \.

Examples:

"This quoted string is treated as a single word"  
var myname = "Eddy Deegan"  
This\ is\ treated\ as\ a\ single\ word
"The character \" is used for quoting"

The following table summarises the behaviour:

Comments

Comments in a USE script start with a # character that is either of

• the first character of a line

• the first character in a word

Comments always end at the end of the line they were started on

# This is a comment
set http_header "Content-Type: application/x-www-form-urlencoded"     # This is a comment
var usage#1 = Usage1   # The '#' in 'usage#1' does not start a comment

Variables

Overview

USE scripts often make use of variables. Variables have a name and a value. When a variable name is encountered on any given line during execution of the script, the name is replaced with the value before the line is executed.

To reference a variable, the name should be preceded with ${ and followed by }. For example to access the value of a variable called username, it should be written as ${username}.

The length (in characters) of a variable can be determined by appending .LENGTH to the variable name when referencing it. Thus if a variable called result has a value of success then ${result.LENGTH} will be replaced with 7.

Creation

Encryption

Publishing to the user interface

Variables may be exposed in the GUI by prefixing their declaration with the word public as follows:

public var username = username
public encrypt var password = something_secret

Any variable so marked may be edited using a form in the GUI before the script is executed. If a public variable is followed by a comment on the same line, then the GUI will display that comment for reference. If there is no comment on the same line, then the line before the variable declaration is checked, and if it starts with a comment then this is used. Both variants are shown in the example below:

public var username = login_user  # Set this to your username
# Set this to your password
public var password = "<please fill this in>"

Named buffers

A named buffer (also termed a response buffer) contains data retrieved from an external source, such as an HTTP or ODBC request. Buffers are created with the buffer statement.

Once created, a buffer can be referenced by enclosing its name in { and } as follows:

# Example of buffer creation
buffer token = http POST "https://login.windows.net/acme/oauth2/token"

# Examples of referencing a buffer
save {token} as "extracted\token.data"
discard {token}

• Buffer names may be up to 31 characters in length

• Up to 128 buffers may exist simultaneously

• Up to 2Gb of data can be stored in any given buffer (memory permitting)

Extracting data with Parslets

Parslets are used to extract data from from the contents of a named buffer.

To create a subscription, first browse to the Accounts > Subscriptions menu. In the Subscriptions menu it is now first required to select a 'leaf account' (meaning: an account at the deepest level of a report definition). Once this account has been selected, a new Subscription can be created:

When creating a new subscription, it is mandatory to first give it a name and select the Service Category and Service where this Subscription applies to:

Once a Service has been selected, depending on the type of Service, you are either free to fill in a customer Subscription Rate and optional Cost. If this service has already rates configured in Exivity, it will automatically show and use those:

Next the Interval section will be required to fill in. First provide the Subscription Type, which can be either a Recurring subscription or a One-off transaction. In the case of a Recurring subscription, it is required to provide an Interval of Daily, Monthly or Yearly. This determines how often the Subscription is being charged. Then the Subscription Start date needs to be set, which determines the initial charge date. Also an optional End date be configured. If none is provided, it will be charged until the Subscription is removed, or changed later to include an End date.

When creating a Monthly or Yearly Subscription, it is also required to select a Charge day. This determines the day of the month when the Subscription is being charged. In case of a Yearly subscription, it will also be required to specify a Charge month, to indicate which month of the year the Subscription will be charged:

Click the Create button to create this subscription. In case you are creating a Subscription with a historical Start date, it will then be required to use the Create - with preparing button:

In case you are creating multiple Subscriptions, it is advisable to stack these creations to avoid having to execute Create - With preparing multiple times in a row.

Introduction

Extraction is the process by which USE (Unified Scriptable Extractor) retrieves data from external locations. The following types of data source are supported:

USE script

A USE script is required for USE to operate. Further information can be found via the links below:

An introductory overview of the scripting language:

A reference guide for the USE scripting language:

How to parse XML and JSON data

Template scripts that can be used as starting points for common data sources:

Our customers and Solution Architects have created several useful subroutines over the last few years. We are keeping a small library of the most useful ones on our public docs. You can find them listed on this page.

The basename statement is used to extract the filename portion of a path + filename string