Exivity is a metering and billing software solution for public and private cloud environments that allows you to report on cloud consumption from any IT resource. Exivity enables you to apply your MSP/CSP business rules and makes any type of Pay-as-you-Go model work. It also facilitates internal charge-back and show-back requirements for Enterprise IT.

These things are done by extracting IT consumption data from various endpoints and then mapping this data to meaningful customer-specific information such as services, customer IDs, names, and contracts.

There are four main steps involved in a successful deployment:

1. Extract

2. Transform

3. Report

4. Integrate (optional)

Extract

The Extract step defines your data sources such as:

• APIs that return usage data, service catalogue, rate card, customer/subscriber lists, and similarly available records from public or private clouds

• APIs or ODBC queries that return contracts, customer names, IDs, and other contextual lookup data from CMDB / CRM systems

• Flat files on disk in CSV, JSON, or XML format

• Other HTTP/S sources

Exivity provides a rich scripting interface via its Unified Scriptable Extractor (USE) component which facilitates integration with almost any data source. For most of the big cloud platforms, we provide template extractor scripts as part of the product. Additionally, you can also write your own USE scripts from scratch in order to integrate with custom data sources.

Transform

The Transform step provides a powerful ETL framework for processing extracted data. Using it you can merge consumption metrics, contract details, customer information, custom metadata, service definitions or any other imported information to produce an enriched and/or normalised result.

This is done using the Transcript component, which executes user-definable scripts (termed tasks) in order to produce a meaningful set of data suitable for reporting against. Often this data will feed a consolidated bill of IT based on the various different consumed services.

Transcript also allows you to define and populate services and rates, either of which may be passed through from cloud data, defined as custom offerings, or a mixture of the two.

Report

Exivity provides a modern responsive User Interface, that allows you to 'slice and dice' the processed data in any way you choose. Multiple Report Definitions can be created with ease which allows you to graphically and textually display both cost and usage statistics.

Integrate

We think that Exivity should be part of your automation landscape, where it can provide (for example) line items that can be digested by your ERP and/or invoicing system. Therefore we consider Integrate as the logical final step for any deployment where it is useful.

To this end, we offer an open and fully-featured REST API. Our GUI uses our own API for all back-end processes meaning that all textual data shown in the Exivity GUI is also obtainable via our API.

Detailed information about the REST API can be found at https://api.exivity.com

Exivity can be easily deployed on any system running Windows Server 2016 or later.

Single Node System Architecture

In order to deploy a single node Exivity system, be aware of the following system architecture:

All components can be installed on a single node. However, even in a single node deployment, it might be desirable to run the PostgreSQL database and RabbitMQ system on a different system. Apart from installing PostgreSQL from the Exivity installer, the Exivity software is also compatible with any PostreSQL compatible database engine (PostgreSQL on Linux, Amazon RDS, CockroachDB, etc) in order to achieve High Availability or Load Balancing. The same goes for the RabbitMQ component.

Deploying Exivity on a Single Node

After starting the Exivity installer and providing a valid license, ensure to have all components selected and click Next:

Then make sure to provide a valid path for the Exivity program files:

And select a folder for the Exivity home files:

Provide a custom administrator username and password, or leave the default:

Then provide the details for the PostgreSQL database. The installer will configure a local PostgreSQL database by default:

In case you prefer to use a remote PostgreSQL database system, ensure to deselect 'Install Local PSQL Engine' and provide your PSQL server credentials:

When you are finished configuring your PostgreSQL database settings, click the Next button to configure RabbitMQ. To use a remote RabbitMQ instance, deselect the 'Install Local RabbitMQ Engine' and provide the appropriate hostname, username, password, vhost and TCP port. In case you require TLS/SSL towards your RabbitMQ instance, select that checkbox as well:

Once the installation is finished, ensure to check 'Start the Exivity Windows Services' to start the Exivity services after clicking Finish.

Executing a Single Node Silent Installation

Follow the below steps to execute an unintended setup using the silent installer flags supported by the setup program.

1 - installing Exivity on a single node with a custom PostgreSQL data path

The below example will silently install Exivity with mostly default settings, except having the Exivity program, home, and PGDATA in separate locations:

Exivity_{version}_setup.exe /S /EXIVITY_PROGRAM_PATH=C:\Exivity\Program /EXIVITY_HOME_PATH=D:\Exivity\home /PGDATA=E:\PostgreSQL\PGDATA 

2 - installing Exivity on a single node using a remote PostgreSQL database and RabbitMQ

The below example will silently install Exivity while using a remote PostgreSQL database:

Exivity_{version}_setup.exe /S /EXIVITY_PROGRAM_PATH=C:\Exivity\Program /EXIVITY_HOME_PATH=D:\Exivity\home /PGUSER=exivityadmin /PGPASSWORD=S3cret!123 /PGHOST=db.exivity.local /PGPORT=5432 /PGDB=exivitydb /PSQL_INSTALLED=0 /MQ_INSTALLED=0 /MQHOST=mq.exivity.local /MQUSER=mqexivity /MQPASSWORD=mqpass /MQVHOST=/exvt /MQSSL=1 

Exivity can be deployed on a single node, or on multiple nodes for HA and load balancing. This guide walks you through the steps to install some of the Exivity components on different nodes.

Multi-Node System Architecture

The following diagram outlines the various components that can be deployed on separate nodes:

For larger environments, it is recommended to deploy each component on separate virtual machine nodes. As displayed in the diagram above, a typical multi-node deployment will consist of the following elements:

• Load Balancer (optional)

  • An optional but recommended 3rd party load balancer may be used to ensure high availability of the Exivity front end web application.

• Exivity Web/UI node(s)

  • 1 or more Exivity Web/UI nodes should be deployed to support the customer facing web application.

• Exivity API/Backend node(s)

  • 1 or more Exivity API/Backend nodes should be deployed to support the customer facing web application.

  • Storage

    • In multi-node environments, it is advisable to provide a shared storage device (i.e. SMB/NFS) that can be accessed by the backend nodes

• PostgreSQL node or cluster

  • Exivity solution relies on a PostgreSQL version 10 (or higher) compliant database engine.

• RabbitMQ node or cluster

  • Exivity relies on a RabbitMQ version 3.8 (or higher) message broker

The platform is designed in such a way that adding and removing Exivity nodes should be relatively straightforward, which compliments thr potential growth or shrinkage of data processing needs.

Deploying a PostgreSQL node

Exivity highly recommends deploying your own PostgreSQL database cluster on Linux (or use a managed PSQL service from Azure, AWS or other vendors). To achieve High Availability, any PostgreSQL compatible cluster manager software may be used. At Exivity we have good experiences with pg_auto_failover, which is an open source PostgreSQL on Linux cluster manager (backed by Microsoft) in case you prefer to self-manage the Exivity PostgreSQL database.

In case you prefer to install PostgreSQL on Windows, Exivity recommends installingthe PostgreSQL role together with the API/Scheduler backend components during the installation wizard.

Deploying a RabbitMQ node

Exivity highly recommends deploying your own RabbitMQ instance on Linux (or use a managed RabbitMQ service from AWS, CloudAMQP, or other vendors).

Deploying a Backend node

In order to deploy a backend node, the following steps need to be executed manually or automatically thru the use of the silent installer CLI interface.

After starting the installer, click Next and provide a valid license key when asked. After clicking Next again, the component screen will be shown:

Ensure to deselect the Web Service . The API Service component can also be excluded, although in some cases it is recommended to have the API and Backend Services running on the same system. Please consult with Exivity support in case you are not certain what to select.

Click Next to continue. Then provide a folder for the Exivity program files, and afterward select a folder for the Exivity home files.

Provide a custom administrator username and password, or leave the default:

Now specify a remote PostgreSQL database instance, or select to install the PostgreSQL database locally on the API/backend node:

When you are finished configuring your PostgreSQL database settings, click the Next button to configure RabbitMQ. To use a remote RabbitMQ instance, deselect the 'Install Local RabbitMQ Engine' and provide the appropriate hostname, username, password, vhost and TCP port. In case you require TLS/SSL towards your RabbitMQ instance, select that checkbox as well:

Once the installation is finished, ensure to check 'Start the Exivity Windows Services' to start the Exivity services after clicking Finish.

Silently install Backend node

The below example will silently install an Exivity API/Backend node while using a remote PostgreSQL database instance as well as a remote RabbitMQ instance using SSL:

start /wait Exivity_{version}_setup.exe /S ^
	/PSQL_INSTALLED=0 ^
		/PGUSER=exivityadmin /PGPASSWORD=S3cret!123 /PGHOST=db.exivity.local /PGPORT=5432 /PGDB=exivitydb ^
	/MQ_INSTALLED=0 ^
		/MQHOST=mq.exivity.local /MQPORT=5671 /MQUSER=exivity /MQPASSWORD=My5pecia1pas9 /MQVHOST=exvt /MQSSL=1 ^
	/WEB_INSTALLED=0 ^
	/JOBMAN_INSTALLED=1 ^
	/SCHEDULER_INSTALLED=1 ^
	/API_INSTALLED=1 ^
	/BACKEND_INSTALLED=1 

Deploying a Web/API node

In order to deploy a Web/UI only node (with optional API), the following steps need to be executed manually or automatically thru the use of the silent installer CLI interface.

After starting the installer, click Next and provide a valid license key when asked. After clicking Next again, the component screen will be shown:

Select the Web Service and optionally also the API Service. In some cases, the API can also be deployed on the front-end node, but if you are not certain, install only the Web Service on the front-end node, and the API Service on the backend node.

Click Next to continue. Then choose a folder for the Exivity program files, and afterward select a folder for the Exivity home files.

In the following screen, it will be required to specify the remote host and port of your Exivity API node:

Once the installation is finished, ensure to check 'Start the Exivity Windows Services' to start the Exivity services after clicking Finish.

Silent install Web/UI node

The below example will silently install an Exivity Web/UI node:

Exivity_{version}_setup.exe /S ^
	/EXIVITY_PROGRAM_PATH=C:\Exivity\Program /EXIVITY_HOME_PATH=D:\Exivity\home ^
	/PSQL_INSTALLED=0 ^
	/MQ_INSTALLED=0 ^
		/MQHOST=mq.exivity.local /MQPORT=5671 /MQUSER=exivity /MQPASSWORD=My5pecia1pas9 /MQVHOST=exvt /MQSSL=1 ^
	/JOBMAN_INSTALLED=0 ^
	/SCHEDULER_INSTALLED=0 ^
	/API_INSTALLED=0 ^
		/PROXIMITYHOST=remote.api.local /PROXIMITYPORT=443 ^
	/WEB_INSTALLED=1

Introduction

Exivity can be deployed from the AWS Market Place allowing you to have a functional Exivity solution in a matter of minutes. This tutorial will get you up and running.

AWS Marketplace Offering

Login to your AWS portal and access the Exivity offering .

1. Click on Continue to Subscribe.

2. Read our Terms and Conditions, and when ready, click on Continue to Configuration.

3. Select the Region where you want to deploy Exivity and click on Continue to Launch.

4. In Choose Action select Launch though EC2 and click on Launch to access the Deployment Wizard.

Deployment Wizard

1. In this section, you can select your VPC Configuration, or leave the default values.When you are done with your configuration click on Next: Add Storage.

2. In the Storage section, two drives are recommended, you can influence the Volume Type and the Size (GiB) parameters. When ready, click on Next: Add Tags.

3. In the Tags section, include the tags that are meaningful to you, as a minimum, a Name tag is recommended. Click on Next: Configure Security Group.

1. In the Security Group section, you can leave the default recommended security group or add more rules if needed. Click on Review and Launch.

2. Review the details and click on Launch, select your preferred Key Pair to connect to the instance.

In a few minutes your instance will be deployed, you can track the progress in your EC2 Dashboard:

Write down the Public IP address / Public DNS and the Instance ID once they are available.

Connecting to your Exivity instance

You can logon to your Exivity instance with RDP, but after deployment you should be able to connect to your instance using the public IP address or DNS name of your Exivity instance considering the following default URL:

• https://<Your_Public_DNS>:8001

The default admin username is admin with password Instance ID.

Next Steps

Introduction

Apart from installing Exivity in any on premise environment, Exivity can also be deployed from the Azure Market Place (AMP). Deploying Exivity on AMP is straight forward, and can be finised within a few minutes via your Azure Portal.

Azure Marketplace Offering

Login to your Azure Portal at and then go to the Marketplace to search for the Exivity offer:

Once you've selected the Exivity offering, you should be presented with the following screen:

After clicking the Create button, you will be redirected to the VM deployment wizard

Deployment Wizard

1. Fill in a Windows user/pass and pick your deployment Resource Group:

1. You may select any additional options, but none are required for running Exivity succesfully, so you may skip this page simply by clicking the OK button:

1. Review the summary and click Create to deploy your Exivity VM:

This may take a few minutes. You may review the status of the Virtual Machine in your VM list:

Write down the Public IP address once it is available. Optionally you may configure a custom DNS name to have an easy way to connect.

Connecting to your Exivity instance

You can logon to your Exivity instance with RDP, but after deployment you should be able to connect to your instance using the public IP address or DNS name of your Exivity instance considering the following default URL:

• https://<Your_Public_IP>:8001

The default admin username is admin with password exivity.

Next steps

Server

Exivity can be installed on any Microsoft Windows 2016 or higher server in your on-premise data center or in the cloud. Depending on the amount of data, Exivity recommends the following system configuration:

* a managed high available and high-performance cluster would be recommended ** consult with Exivity for advisory on appropriate sizing of extra-large environments

Each node can be either a physical or virtual machine. The CPU and RAM requirements provided in the table above are defined per node whereas the Storage requirements are to be considered for the complete UMB deployment as a whole and should be hosted on shared storage (i.e. SMB/NFS). An operating system will need to be pre-installed on every node. Web and Backend nodes require a version of Windows 2016 Standard or higher to be installed. In scenarios where the RabbitMQ and/or PostgreSQL nodes are deployed on dedicated or clustered environments, the operating system guidelines specific to those applications are to be considered.

Client

The Exivity front-end supports the following desktop browsers:

• Google Chrome v59+

• Microsoft Edge v41+ (EdgeHTML 16+ / Blink 80+)

• Opera v46+

• Mozilla Firefox v65+ (support added in Exivity v2.10.0)

• Apple Safari v10.1+ (support added in Exivity v3.2.7)

Introduction

When deploying the vCenter Extractor template for Exivity, it is required to configure a user with approperiate permissions. Additionally, the following process must be completed in order to report on vCenter consumption:

1. Create Exivity vCenter user

2. Configure an Extractor

3. Configure a Transformer

4. Create your Report

Exivity supports Out of the Box integration with vCenter version 6.5 and higher. For version before 6.5, please contact support@exivity.com.

Create Exivity vCenter user

Exivity needs a user with a reader role in order to retrieve consumption data. Please follow this procedure:

• In your vCenter Configuration Manager go to Configuration > Local Users and Groups > Users.

• Create a new User, insert username and password.

• Take note of the username and password, they will be used later on to configure Exivity

• Click on Create.

• Under Users and Groups, click Add, select the previously created user.

• On the Select Users and Groups dialog, click Add, and then click OK.

• Add a Reader role in the dropdown.

• Finally click on OK.

Configure an Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. To create a new Extractor based off the vCenter template follow these steps:

• Provide a name for the Extractor in the Name field above

• Pick vCenter 6.5 (VM Inventory REST API) template from the list

• Click the Create button.

Once you've created the Extractor, go to theVariables tab:

Fill in all required variables with the values that you gathered in the previous step. You have the option to encrypt a variable in case it contains sensitive information (i.e. password) by clicking the lock icon on the right of each variable field. When finished, click Update.

Once you've filled in all details, go to the Run tab to execute the Extractor clicking on Run Now:

If the variables are correct and your vCenter is reachable for Exivity, you should get a successful result.

Configure a Transformer

Once you have successfully run your vCenter Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day (today) as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select your vCenter dataset, and your preferred Reporting Columns to break down the report (we recommend only cluster_name for the default report). When you are ready, click on Create.

Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Pre-requisites

This tutorial assumes that you have CUR (Cost and Usage Report) set up in your AWS environment. In the event that this is not the case please follow the steps in Turning on the AWS Cost and Usage Report before proceeding.

Please note that in order to deploy this solution the S3 bucket to which CUR reports are written must reside in one of the following AWS regions:

• Northern Virginia

• Ohio

• Oregon

• Mumbai

• Seoul

• Singapore

• Sydney

• Tokyo

• Frankfurt

• Ireland

• London

At this point in time, only the previous regions have all the necessary services deployed.

Introduction

This tutorial shows how to build a serverless solution for querying the AWS CUR Report using Exivity. This solution makes use of AWS serverless services such as Lambda and Athena, as well as other commonly used services such as S3, CloudFormation, and API Gateway. The following topics will be covered:

1. Solution Overview

2. Launching the CloudFormation Template

3. Creating the Lambda function and API Gateway

4. Configuring an Extractor

5. Configuring a Transformer

6. Creating your Report

Solution Overview

The Billing and Cost Management service writes your AWS Cost and Usage report to the S3 bucket that you designated when setting up the service. These files can be written on either an hourly or daily basis.

The CloudFormation template that accompanies this tutorial builds a Serverless environment containing a Lambda function which reads a CUR file, processes it and writes the resulting report to an output S3 bucket. The output data object has a prefix structure of "year=current-year" and "month=current-month". For example, if a file is written 13/09/2018 then the Lambda function outputs an object called "bucket-name/year=2018/month=09/file_name".

The next step in the template is to translate this processed report into Athena so that it can be queried. The following diagram shows the steps involved in the process:

Afterwards, we will create a Lambda function to query the Athena database, returning a URL with the results of the query in CSV format. We will also create an API EndPoint with the AWS API Gateway service, which is used by Exivity to retrieve the data.

Launching the CloudFormation template

In order to deploy this solution successfully the following information is required:

1. The name of your AWS Cost and Usage report.

2. The name of the S3 bucket in which the reports are currently stored.

Firstly, launch the CloudFormation template that builds all the serverless components that facilitate running queries against your billing data. When doing this, ensure that you choose the same AWS Region within which your CUR S3 bucket is located.

• Ireland

• Ohio

• Oregon

• Northern Virginia

• Mumbai

• Seoul

• Singapore

• Sydney

• Tokyo

• Frankfurt

• London

Now follow the instructions in the CloudFormation wizard, using the following options, and then choose Create.

• For CostnUsageReport, type the name of your AWS Cost and Usage report.

• For S3BucketName, type a unique name to be given to a new S3 bucket which will contain the processed reports.

• For s3CURBucket, type the name of the bucket into which your current reports are written.

While your stack is building, a page similar to the following is displayed.

When the Status column shows CREATE_COMPLETE, you have successfully created four new Lambda functions and an S3 bucket into which your transformed bills will be stored.

Once you have successfully built your CloudFormation stack, you can create a Lambda trigger that points to the new S3 bucket. This means that every time a new file is added to, or and existing file is modified in, the S3 bucket the action will trigger the lambda function.

Create this trigger using the following steps:

• Open the Lambda console.

• Choose Functions, and select the aws-cost-n-usage-main-lambda-fn-A Lambda function (note: do not click the check box beside it).

• There should be no existing triggers. Choose Trigger, Add trigger.

• For Trigger type (the box with dotted lines), choose S3.

• Select the S3 bucket withing which your CUR reports are stored.

• For Event type, choose Object Created (All) and check Enable trigger.

• Click Submit.

The database and table are not created until your function runs for the first time. Once this has been done, Athena will contain the database and table.

Athena stores query results in S3 automatically. Each query that you run has a results file in CSV format and a metadata file (*.csv.metadata) that includes header information such as column type, etc.

Testing (Optional)

Once you have successfully added the trigger to the S3 bucket in which the Billing and Cost Management services writes your CUR reports, test the configuration using the following steps.

• In the S3 path to which AWS writes your AWS Cost and Usage Billing reports, open the folder with your billing reports, open the folder with your billing reports. There will be either a set of folders or a single folder with a date range naming format.

• Open the folder with the data range for the current month. In this folder, there is a metadata file that can be found at the bottom of the folder. It has a JSON extension and holds the S3 key for the latest report.

• Download the metadata file. Ensure that the name of the file on your machine is the same as the version stored on your S3 bucket.

• Upload the metadata file to the same S3 path from which you downloaded it. This triggers the Lambda function aws-cost-n-usage-main-lmbda-fn-A.

• In the S3 bucket that you created to hold your processed files, choose the "year=" folder and then the "month=" folder that corresponds to the current month. You should see the transformed file there, with the time stamp that indicated that it was just written.

Creating the Lambda function and API Gateway

To automate this process a CloudFormation template will be provided. This template will create an IAM role and Policy in order that our API can invoke Lambda functions. Then it will create a Lambda function with the capabilities of querying our previously created Athena Serverless DB, and save the output in a S3 bucket in .csv format, (this output will be later retrieved by Exivity). Finally, it will deploy an API Gateway allowing us to create an endpoint for our Lambda function, this is the endpoint that the Exivity extractor will consume. Make sure to launch the CloudFormation template in the same region that you have deployed the previous one.

Let's start by downloading the CloudFormation template (you only need to choose one of the formats, both are supported by AWS):

Then follow the next steps:

• Go to the CloudFormation console.

• Choose Create Stack.

• Choose Upload a template to Amazon S3.

• Select from your computer the template that you have downloaded.

• Follow the CloudFormation wizard - Add a Name to the Stack and select I acknowledge that AWS CloudFormation might create IAM resources with custom names in the last step.

• Once the stack is created you should see an CREATE_COMPLETE message.

• Click on Output to take a note of your endpoint (you will need to input this in the Exivity extractor).

Next, we will associate an API Gateway trigger to our Lambda function:

• Go to the Lambda console.

• Choose the QueryAthena2 function.

• Under Add Triggers select API gateway. You should see an image like the following:

• Click on API Gateway figure to configure it.

• On API select QueryAthena2.

• On Deployment Stage select v1.

• On Security select Open.

• Choose Add.

• Choose Save.

You should see a screen like this:

Finally, we will deploy the API Gateway:

• Go to the API Gateway console.

• Choose QueryAthena2.

• In the Resources section, click on the ANY method.

• In Actions, choose Delete Method.

• Click on Delete.

• In the Resources section, choose Actions.

• Click on Deploy API

• In Deployment Stage select V1.

• Add a Deployment Description.

• Choose Deploy.

Securing the API Gateway

Initially, the created API endpoint is public and as such is vulnerable to the possibility of misuse or denial-of-service attacks. To prevent this, associate an API Key with the endpoint as per the following steps:

• Inside the API Gateway dashboard, select the QueryAthena2 API

• In Resources, select Method Request

• In Settings, change API Key Required to True

• Click on Actions and choose Deploy API to effect the change

• In Deployment Stage, select v1 and click on Deploy

• Go to the API Keys section

• Click on Actions and select Create API Key

• In Name write ExivityAPIKey

• Click on Save

• Copy the API Key, as this will be required by the Exivity configuration

• Go to Usage Plan

• Click on Create.

• In Name write ExivityUsagePlan

• In the Throttling Section, change Rate to 100 and Burst to 10

• In the Quota Section, change it to 50000 requests per Month

• Click on Next

• Click on Add API Stage

• In API, select QueryAthena2 and in Stage select v1

• Confirm the changes and click on Next

• Click on Add API Key to Usage Plan

• Select ExivityAPIKey, confirm the changes

• Click on Done

The API Key is now required to access the API endpoint thus adding a layer of security to mitigate unauthorized access attempts.

Configure Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For AWS, please click AWS_CUR_Extractor from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to first tab: Variables

• In the Bucket variable specify the name of the S3 bucket where the .csv with the output of the query will be saved (The S3BucketName previously specified when launching the CloudFormation template).

• In the Api endpoint variable specify the API endpoint previously created plus the route /QueryAthena.

• In the DBname variable specify the name of your DB, you can find it in the Athena main Dashboard.

• In the Tablename variable specify the name of the table inside your DB, you can find it in the Athena main Dashboard.

• In the API_Key variable specify the API Key that we have created in the Securing API Gateway Section.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to collect consumption data.

• to_date should be the date immediately following from_date.

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your AWS CUR Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button. Select the AWS CUR Transformer and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

Create Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

When deploying the vCloud Extractor template for Exivity, some input and configuration is required from your vCloud environment. The following process must be completed in order to report on vCloud consumption:

1. Create Exivity vCloud user (vCloud < 9.1)

2. Create Exivity vCloud user (vCloud >= 9.1)

3. Configure an Extractor

4. Configure a Transformer

5. Create your Report

Create Exivity vCloud user (vCloud < 9.1)

For environments with a vCloud previous to 9.1 Exivity needs a user with the sysadmin role on the system ORG. Please follow this procedure:

• Click the Administration tab and click Users in the left panel.

• Click New, fill the required details.

• Note the username and password.

• Finally, click OK.

Create Exivity vCloud user (vCloud >= 9.1)

For environments with vCloud version 9.1 or higher, you can create a user with more fine grained permissions. Exivity needs a user with a reader role on the system ORG. Please follow this procedure:

First you will need to create a new custom role for the Exivity user:

• From the main menu, select Administration.

• In the left panel, under Access Control, click Roles

• Click New.

• Enter a name and, optionally, a description for the new role.

• Select the rights that you want to associate with the role. Exivity needs all the View Rights in the different tabs and also Perform Administrator Queries under the General tab.

• Click Save.

Once you have the Role created, you need to setup a new user and assign the Role previously created:

• On the VMware Cloud Service toolbar, click the VMware Cloud Services icon and select Identity & Access Management.

• Click Add Users.

• On the Active Users tab, fill the details of the user you want to add to the system organization.

• In the Role in organization text box, assign the role previously created.

Configure an Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For vCloud:

• Pick vCloud_Extractor_AdminVM from the list

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, next go to theVariables tab:

Fill in all required variables with the values that you gathered in the previous step. You have the option to encrypt them. Click on Update.

Once you've filled in all details, go to the Run tab to execute the Extractor clicking on Run Now:

Configure a Transformer

Once you have successfully run your vCloud Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day (today) as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions.

Select your vCloud dataset, and your preferred Reporting Columns to break down the report (we recommend only Org_name and VDC for the default report). When you are ready, click on Create.

Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Pre-requisites

This tutorial assumes that you have a Billing Account for your Google Cloud Projects in place. In the event this is not the case please follow the steps in this tutorial before proceeding.

Setting up BigQuery

Exivity will leverage on the GCP BigQuery service. This will enable the export of detailed Google Cloud billing data to a BigQuery dataset, giving Exivity the capability to query the billing data in a more granular manner than with the standard file export (which is going to be deprecated by Google).

First and foremost, Exivity will create a dataset containing the billing table which will be periodically queried. Please follow these instructions:

1. Go to the BigQuery service page.

2. Select the project that will contain your dataset in the project drop down.

3. Click CREATE DATASET.

   1. Enter a Dataset ID

   2. Select Data location (region)

   3. Set Default table expiration to Never

   4. Set Encryption to Google-managed key

4. Finally, click Create dataset

5. Note the Project, BigQueryProject and Table, these parameters will be used by Exivity on a later stage.

Once you have the dataset created in BigQuery you need to enable Cloud Billing export to BigQuery:

1. Sign in to the Cloud Billing Console, select Organization and main Billing Account.

2. Go to the Billling Export tab.

3. Click Edit Settings to enable the export.

   1. Select the project where you previously created the BigQuery dataset.

   2. Select the specific dataset from the Billing export dataset list.

   3. Click Save.

Creating Exivity Service Account

Exivity requires a GCP Service Account with the BigQuery User role asscociated in order to retrieve the billing data. Please follow this manual to create a Service Account, this manual to associate a Private Key with the service account, and finally this manual to associate the role with the service account.

Configuring the Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Google Cloud, please click Google_Cloud from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to Variables tab and fill the parameters:

• Hostname: Default endpoint for Google BigQuery with version

• Private: Provide the RSA private key PEM format

• Email: Mail address / service account associated with the private key, obtained in the Creating Exivity Service Account section.

• Project: Main GCP Project name.

• BigQuery project: GCP BigQuery Billing Project, obtained in the Setting up BigQuery section.

• BigQuery table: GCP BigQuery Billing Table, obtained in the Setting up BigQuery section.

Finally, click on Update.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to start collecting consumption data.

• to_date is the end date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure a Transformer

Once you have successfully run your Google Cloud Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button.

Select the Google Cloud Transformer that you just created, go to the Run Tab and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

When you click the Run Now button, you should get a successful result.

Create your Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Pre-requisites

This tutorial assumes that you have CUR (Cost and Usage Report) set up in your AWS environment. In the event that this is not the case please follow the steps in Turning on the AWS Cost and Usage Report before proceeding.

Please note that we also need the following requisites:

• S3 bucket where the CUR reports are placed

• The AWS region of the S3 bucket

• The path where the reports are placed inside the S3 bucket

To get this requirements you can go to the Amazon S3 service - Buckets and note the name and the region of our CUR S3 bucket

Once inside the CUR bucket, you must find the path where the monthly reports reside, please navigate the bucket until you find a folder like this one and note the path:

Obtaining the API Credentials

We have to create an IAM User that has read access to the CUR bucket. Please follow these steps:

• Go to IAM - Policies and click on Create policy.

• In the next screen: Service select S3, Actions select Read, Resources select Specific and in bucket click on Add ARN.

• In the next screen, type the name of your bucket followed by /* in Bucket name (example: billing-bucket/*) and click on Add.

• Click on Review Policy.

• Give a Name and a Description to the policy and finally click on Create policy.

Now, we will attach the policy to a user:

• Go to IAM – Users – Add user.

• Add an User name and select Access type: Programmatic access.

• Click on Attach existing Policies directly and filter by the name of the policy you just created.

• Select the policy and click on Next:Tags.

• Add a Key-Value pair for the Tags and click on Next:Review.

• Click on Create User.

• In the next screen click on Download .csv we will use this credentials to populate our Exivity Extractor in the next section.

• Finally click on Close.

Configure Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For AWS, please click AWS_CUR_Extractor from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to first tab: Variables

• Bucket: Type the name of your S3 CUR bucket.

• AWS region: The name of your S3 CUR bucket region code can be found in the Regions, Availability Zones, and Local Zones Documentation (Region column).

• Access key: You can find it in the csv that you have downloaded in the previous section.

• Secret key: You can find it in the csv that you have downloaded in the previous section.

• Cur report: Type the name of your CUR report.

• Bucket Directory: Type the path that you got in the previous section (if there is more than one folder separate them with /).

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires one parameters in yyyMMdd format:

• from_date is the date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your AWS CUR Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button. Select the AWS CUR Transformer and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

Create Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

When deploying the Azure Stack Extraction template for Exivity, some configuration is required within your Azure Stack environment and a lookup file needs to be created. The following process must be completed in order to report on Azure Stack consumption:

1. Create an Exivity Enterprise Application in your Azure AD for authentication

2. Configure a rate card lookup file

3. Configure an Extractor

4. Configure a Transformer

5. Create your Report

Creating an Enterprise Application

In order for Exivity to authenticate with Azure Stack, you will need to create an application in the Azure AD where you have registered your Azure Stack management node:

When you create this application in your Azure AD make sure it has (at least) the Reader Role in your Default Provider Subscription:

Create a rate card

As Microsoft does not provide rate card information via the Azure Stack Consumption API you will only obtain usage metrics from Azure Stack for all of the Meter IDs that are mentioned here by Microsoft.

Exivity provides a template rate card that you can use for creating your own rates. Please bear in mind that these rates are fictional, thus you should update it with your preferred values. However to get started, you can use the file linked above by placing it as a csv file in your Exivity home folder at the following location:

%EXIVITY_HOME_PATH%\system\extracted\AzureStack\rates\azure_stack_example_rates.csv

Configure Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Azure Stack:

• Pick Azure_Stack_Extractor_(App+Secret) from the list

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, go to first tab: Variables

Fill in all required variables marked within the red box in above screenshot. If you don't know some of the required GUIDs, most of these can be obtained by browsing to the Azure Stack management node URL:

• https://adminmanagement.**<your.domain.com>**/metadata/endpoints?api-version=2015-01-01

Another way to obtain some of this information is using the Diagnostics button in your management portal:

When you click the Show Diagnostics link, it should download a JSON file containing most of the parameters you'll need, such as Provider GUID, Audience GUID etc.

Once you've filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyyMMdd format:

• from_date is the date for which you wish to collect consumption data

• to_date should be the date immediately following from_date

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your Azure Stack Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Introduction

When deploying the Azure EA Extraction template for Exivity, some configuration is required within your Azure EA environment. The following process must be completed in order to report on Azure EA consumption:

1. Create an Access Key and Secret in your Azure EA portal

2. Configure the Azure EA Extractor

3. Configure your Azure EA Transformer

4. Create a Report definition

Creating an Access Key & Secret

In order for Exivity to authenticate with Azure EA, you will need to create an access key and secret in the Azure EA Portal. Also you will need to find your enrollment number. To do this, login to your Azure EA Portal on https://ea.azure.com and navigate to the Reports menu:

Just under the Windows logo in the left menu section you will find your enrollment number which you will need to provide later when configuring the Extractor. To create the Access Key & Secret, click on Download Usage and then on API Access Key. This brings you to the menu where you can manage your Access Keys and corresponding secret which you will need in order to configure the data Extractor.

Configure Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Azure EA:

• Pick Azure EA from the list

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, go to first tab: Variables

Fill in all variables in above screenshot, and feel free to encrypt any sensitive data using the lock symbol on the right.

Once you've filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyyMMdd format:

• from_date is the date for which you wish to collect consumption data

• to_date should be the date immediately following from_date

These should be specified as shown in the screenshot above, separated with a space.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your Azure EA Blob Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Exivity can be installed in your on-premises data center using the provided installer. You can automatically deploy it using the silent installation command-line options or execute it as an interactive installer.

To install Exivity, you'll need the following:

1. A system that complies with the Exivity minimal system requirements

2. The Exivity software installation executable

3. A valid Exivity license key

If you need help meeting one or more of the above requirements, please get in contact with our support department.

Interactive installation

To install Exivity interactively, execute the provided setup executable. Then follow the instructions depending on your intended architecture:

• Single-node deployment

• Multi-node deployment

Silent Installation

To execute a silent installation the following command line parameters are supported:

Example to install using all defaults:

<setup>.exe /S

Example to install using the custom path and Exivity admin user:

<setup>.exe /S /EXIVITY_PROGRAM_PATH="C:\Program Files\Exivity\program" /EXIVITY_HOME_PATH=D:\Exivity\home /ADMIN_USER=master /ADMIN_PASSWORD=P@ssword

Updating minor versions

Updating your installation of Exivity to a more recent minor and/or bugfix release (i.e. 3.x.x --> 3.x.x) is straightforward and may be done by installing the new version over the top of the old.

Manually update

Execute the setup executable. It will detect the installed version of Exivity, and will automatically upgrade when you click Next

Silent update

When executing <setup>.exe /S, your existing installation will be automatically upgraded.

Upgrading major versions

When upgrading to a more recent major version (i.e. 2.10.2 --> 3.x.x) consult first the release notes to verify any breaking changes that may apply to your installation.

Installing a valid SSL certificate

Exivity comes as standard with an untrusted self-signed SSL certificate. It is therefore highly recommended to replace the default certificate with an official one, signed by your Certificate Authority. To install a signed certificate, follow this procedure:

• Download the 32 bit version of openssl.exe from https://slproweb.com/products/Win32OpenSSL.html, and install this tool on the Exivity server

• Use the openssl.exe executable to generate a valid key file on the Exivity server by executing the following command:

openssl.exe genrsa -out exivity.key 4096

• Run the following command to create a certificate signing request file:

openssl.exe req -new -key exivity.key -out exivity.csr -addext "subjectAltName = DNS:example.com"

• You will be asked to enter general information like company name, city, etc. It is important to include the FQDN of the Exivity server when asked for Common Name (e.g. server FQDN or YOUR name) []:

• The generated CSR file should be sent to your Certificate Authority. After processing by your CA, you should receive back a .crt file. Rename this file to webcertificate.crt and rename your exivity.key to webcertificate.key and copy it to the directory %EXIVITY_PROGRAM_PATH%\server\nginx\conf. This should overwrite the existing .key and .crt files.

• Restart the Exivity Web Service Windows service to activate your signed certificate.

Configuring a separate web server portal

In some environments it may be desirable to separate the webserver from the backend components. This can be achieved by installing two separate Exivity instances. One instance could be placed in a DMZ and the second instance would then typically be deployed within a local network as shown in the following diagram:

To achieve this, first install Exivity on the backend node using the standard procedure described here. Afterwards install the Exivity software on the system that should become the User Portal and only install the Web Component. When asked, specify the API Backend hostname and port to finalize the installation.

Make sure to replace HOSTNAME_BACKEND_PORTAL with the actual hostname or IP address of the system that serves as your Exivity Backend Portal.

After completing your installation, you should now be able to access your Exivity User Portal.

Using an Internet proxy when extracting data

In cases where Exivity instance requires internet connectivity (i.e. to obtain Azure or AWS consumption data), and your network infrastructure requires use of a proxy server, it is necessary to configure a system environment variable.

Right click on This PC in an Explorer Window and click on Properties:

Then go to Advanced System Settings, then click the Environment Variables button:

Now add a new System Variable with the name ALL_PROXY and fill in the address of your proxy server as the value for this variable:

In case you do not want to use the proxy for certain address or domains, it is also possible to add an additional variable NO_PROXY:

After confirming the change, make sure to restart both the Exivity Windows Services.

Increase limit of filesize upload in the API

By default, Exivity API has a limit of 2048kb for filesize uploading, should you require to increase it, please modify the file located in %EXIVITY_PROGRAM_PATH%\server\php\php.ini

Enable PDF report exports

In order to generate PDF documents through the Exivity API, the Chrome browser needs to be installed and the directory where chrome.exe can be found should be made available in the systems Path environmental variable. Read how to add a directory to the Path environmental variable.

Depending on your installation method chrome.exe should be installed in one of those directories:

• C:\Program Files\Google\Chrome\Application

• C:\Users\{username}\AppData\Local\Google\Chrome\Application

Restart the Exivity API service after making these changes.

Introduction

When deploying the Azure CSP for Exivity, some configuration is required within your Microsoft Cloud Solution Provider Potal. The following process must be completed in order to report on Azure CSP consumption:

1. Create a Partner Center Web Application

2. Configure Extractors for Azure CSP Usage, Billing & Ratecard

3. Configure Transformers

4. Create your Report

5. Create your Workflows

It is necessary to create independent Extractors/Transformers for the Usage and Billing, the Usage Extractors will be retrieving the data on a daily basis giving an estimation of your daily costs. The Billing Extractor will consolidate the rates based on the blended costs per service for the billing period.

Create a Partner Center Web Application

Perform the following to create the Azure AD configured to access the Partner Center API

• Browse to Partner Center, , and login using credentials that have admin agent and global admin privileges

• Click on the following Dashboard –> Account Settings –> App Management

• Click Add key to create a new Application Key for you App ID that can be used with Exivity.

• Go to the Billing section.

• Open the last month's invoice in pdf format.

• Take note of your billing period

Configure Extractors for Azure CSP Usage, Billing & Ratecard

Go into the Exivity GUI and browse to Data Sources -> Extractors. Then click on Create Extractor and you should get a list of templates. Unfold Azure CSP and pick the usage template:

After selecting the template, click the green Create button on the bottom right. Now make sure to give the new Extractor a name in field at the top:

Now click again on the green Create button at the bottom right. Then click on the Variables menu item:

Now make sure to fill in your Client ID, Secret and your onmicrosoft.com domain. When required, you can encrypt security sensitive field using the lock button on the right of each field. Once you are filling in these details, click the Update button.

Now test the Extractor by going into the Run tab and providing a from and to date like in this example:

Now click the Run Now button and confirm it works as expected:

Create a second extractor using the template Azure_CSP_Invoice_Extractor and give it a name.

This extractor uses the same variables in the Variables menu item as the previous extractor. You can now test the extractor by going to the Run tab, this script uses 3 arguments:

• Positive offset: Starting on 0, it will retrieve the most recent invoice, a 1 will retrieve the previous report, etc.

• Year of the report: Year of the report you want to retrieve.

• Starting day of the billing period: If your billing period goes from 22nd to 21st, the input will be 22.

After filling the arguments you can test the extractor by clicking in Run Now.

Finally, follow the same steps for the Azure Rate Card Extractor, this extractor does not need any arguments.

Configure Transformers

Once you have successfully run your Azure CSP Usage, Billing & Rate Card Extractors, you can create the Transformers templates via Data Sources -> Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. You will need to create two separate transformers using these two templates:

The Azure_CSP_Daily-Usage Transformer will transform the daily usage data and the Azure_CSP_End-of-Month Transformer will consolidate the usage with the final blended rates.

Make any changes that you feel necessary and then select the Run tab to execute it for a single day as a test.Make sure that when running the Transformer you select custom range in the drop-down menu labelled Run for and select the same day as for which you have extracted consumption data in the previous step.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Create your Workflows

You may want to automate the CSP ETL process, you can achieve it by leveraging on the Exivity's Workflow capabilities. You will create two Workflows, one will run on a daily basis calculating the usage of your CSP subscriptions and the other one will run on a monthly basis to consolidate the service rates.

Start by browsing to Administration -> Workflows in the Exivity GUI and click on the +Create button.

Fill the Name and Description fields, in the SCHEDULES section configure the workflow to run on a daily basis at a convenient time.

In the STEPS section you can create as many steps as needed by adding them with the + button. For the first daily workflow, a minimum of 4 steps are required, two steps for the Usage and Ratecard extractors, one for the Transformer and one for the Report. Make sure to input the right FROM and TO date offsets. Click on Update to finish the creation of your first Workflow.

Create a second Workflow, fill the Name, and Description fields and configure the SCHEDULES section to run on a monthly basis, preferably 2 to 3 days after your month's billing period has finished. For the monthly workflow, a minimum of 3 steps are required, one step for the Billing extractor, one for the Transformer and one for the Report. Make sure to input the right FROM and TO date offsets (to cover the entire billing period) and arguments. Click on Update to finish the creating of the second Workflow.

The graphical user interface of Exivity is a pure client-side application, which means it runs inside your web browser. It communicates with the Exivity to obtain data records, report data and general configuration. This means all functionality available in the GUI can also be accessed programmatically.

The interface allows you to do the following:

• Develop extractors

• Create transformer (ETL) tasks

• Configure report definitions

• Run graphical usage & costs reports

• Run textual usage & costs reports

• Manage users & roles

• General configuration

• White labeling

More features are added on a regular basis.

Install / upgrade

A copy of the Exivity installer can be obtained . Installing or upgrading to the latest release is a straightforward process, refer to the for more information.

Release feed

You may register with the following to obtain information about new releases.

Changelog

v3.5.7

December 01, 2021

Bug fixes

• Fixed an issue when installing RabbitMQ locally When using the interactive installer and installing RabbitMQ locally, the installer would store invalid values for the RabbitMQ configuration. This has now been resolved.

• Fixed an issue when system returns from hibernation If a Windows system that is running the "Exivity Backend Service" would return from hibernation mode, the service (merlin.exe) would not recover and remain in a broken state. This has been resolved.

• Fixed an issue with the Transformer editor Resolved a corner case whereby the editor could break when using Transformer editor code snippets.

v3.5.6

November 22, 2021

Bug fixes

• Adjustment type Fixed an issue where the type of adjustment was always shown as premium.

v3.5.5

November 19, 2021

New features

• The name of the EXIVITY_AGGR_COUNT column created by the 'aggregate' statement is now configurable

  The 'aggregate' statement now supports an optional parameter (called 'counter_column') which allows you to specify the name of the column into which the aggregation counters will be written. Please refer to the 'aggregate' documentation for more details.

• Transformer scripts will now log a warning if no RDF files were updated or created

  When running a Transformer script that does not create or update an RDF file using the 'finish' statement, a warning will be written to the log to that effect.

• The 'terminate' statement in a Transformer will now accept 'with error' as optional parameters

  The statement 'terminate with error' is now supported by Transformer scripts and will cause the processing of the current day to abruptly end. A message will be written to the logfile to the effect that the error was explicitly requested and the task will terminate. If the 'continue' option is enabled, processing will resume on the next day in the range (if a range was specified) else the workflow will exit.

Bug fixes

• Fixed an issue when executing quoted scripts from a workflow

  When a script (i.e. Powershell) was quoted in a workflow step, the workflow would not execute the script. This has now been resolved.

• Fixed an installer issue

  When installing RabbitMQ into a program path that contains spaces, RabbitMQ could fail to install the service. This has now been resolved.

• Increased the number of API handlers

  This release now runs 18 concurrent API processes running on a node with the API service installed to allow a higher number of parallel requests to the REST API.

• Fixed an issue with report level labels

  Fixed an issue where report labels weren't updated.

• Fixed an issue with the Adjustments

  Resolved an issue where details in some cases weren't reflected in the form of the Adjustments screen.

• Fixed issue with archiving old log files

  In some cases, collecting old log files resulted in archiving some files that were already archived before.

• Fixed issue with updating an extractor

  When an extractor was updated with NULL arguments, the request was incorrectly considered invalid.

• Fixed an issue when creating budgets

  Resolved an issue where there was unexpected behavior when creating a multi-level budget in the GUI

• DEBUG level logs no longer contain the value of parameters provided to subroutines

  The values of parameters passed to subroutines are no longer included in DEBUG level logfiles as such values could potentially contain sensitive data

• Fixed an issue when viewing proximity log

  Solved an issue where certain log types caused the log screen to fail.

• Fixes issue with some users not being able to change password

  Some users with view_report permissions only were not able to change their password. This issue has now been fixed.

• Log message could contain metadata

  Logs entries can contain 'context' to help debugging. This is usually one JSON string. Some log entries could contain multiple contents, which was unexpected. Now if this occurs, these contexts are merged into one JSON string.

• Fixed an issue when clicking arrow list

  Solved issue where a list could disappear when clicking the arrow in a dropdown list in the GUI

• Fixed an issue with the report level filter

  Resolved an issue where a default report level wasn't selected when selecting a new report in the GUI

• Clarified the log entry written if an extractor script has an unmatched double quote on the last line

  In cases where the last line of an extractor script has an unmatched double-quote, and that last line is also not terminated with a carriage return, then an internal error would be generated. The result is correct (the script will not execute) but the error has now been changed to clearly indicate the exact nature of the problem.

• Fixed an issue related to sorting of datasets

  Resolved an issue with the date sort of datasets in the GUI

v3.5.4

September 15, 2021

New features

• Added more information to audit logs when deleting services The audit log now contains the service key when a service is deleted. Previously it recorded the event,\ but did not include specific information about the service itself.

• Added Filtering on workflow-step type The feature allows users to filter workflow steps by type.

• The installer has now valid code signing applied

  As of this version, the Exivity installer will be signed using the Exivity code signing certificate in order to increase the security of the distributed installer package.

Bug fixes

• Increased the number of API handlers This release now runs 18 concurrent API processes running on a node with the API service installed to allow a higher number of parallel requests to the REST API.

• Fixed an issue with overwriting a Lookup table Solved an issue where saving a lookup could lead to unexpected behavior

• Fixed an issue with user notification subscriptions Solved an issue where it was not possible to save a notification as a user

• Fixed an issue when exporting the Consolidated Summary report Solved an issue where the consolidated export was only exporting the first account

• Fixed a security issue related to User Access Control Solved an issue where the cache was not properly cleared and as such, the user could view invalidated data.

• Fixed an issue with sorting service names in the summary report Services in the summary were sorted on ServiceName instead of Service Description. This has now been resolved.

• Fixed an issue where a certain type of invalid XML parslet would cause an extractor crash In rare cases, using an XML parslet to extract an attribute value could cause a crash. This has now been fixed.

• Fixed missing budget filter "service" bug When a user selected the "service" filter in a budget, this data was not stored correctly. This fixes this issue.

• Fixed an issue with auto-completion There was an issue where auto-completion of example blocks in the Extractor/Transformer editor would not support "tabbing-through" function parameters. This has now been resolved. Additionally, an issue was fixed where the "rename" function would autocomplete wrongly to "convert".

• Fixes issue with links generated from singular form of the resource type Links for relationships inside responses were generated using the singular form of the resource.

• Fixes issue with incorrect links generated inside responses When a response contained a "link" attribute, it was incorrectly generated.

• Better handling of hash and whitespace characters in quoted strings within extractor scripts Previously, a hash symbol at the start of a word in a quoted string literal would be treated as the start of a comment, causing the string to be truncated and an "Unmatched quote" error generated. Additionally, quoted strings containing multiple consecutive whitespace characters would be modified during execution such that they were replaced with a single space. These issues have both been fixed.

v3.5.3

August 27, 2021

Bug fixes

• Fixed an issue with the ordering of headings in the Transformer Preview There was an issue where the Transformer Preview would sort the headings in reversed order. This has now been resolved.

• Fixed an issue when creating a Transformer from a template When creating a transformer from a template, after creation the GUI would show a "leave the page" popup. This has been resolved.

• Fixed an issue when creating an Extractor from a template When creating an Extractor from a template, after creation the screen would not switch to the Variables tab. This has now been resolved.

• Resolved issue where the user couldn't set alt_interval The user couldn't save a subscription that used alternative intervals because of invalid values. This is now solved.

v3.5.1

August 18, 2021

Bug fixes

• Fixed an installer error for the workflow migration script PHP environment errors could show up in the installer log during the execution of the workflow migration script. This has been resolved

• Added cleanup steps to uninstaller to remove PSQL and RabbitMQ traces

• Hidden scrollbar preview Resolved issue where the scrollbar wasn't visible in a Transcript Preview.

• Fixed an issue with environments and global variables in Transformer scripts Attempting to use different environments, and the global variables within them, in a transformer script may have caused an error and the script to fail. This has now been fixed.

• Workflow status Resolved an issue where the GUI could crash because of an invalid status state

v3.5.0

July 26, 2021

New features

• Implemented account lockout strategy User accounts will now get locked out for 15 minutes after 5 consecutive failed login attempts.

• Account names are now set by the latest date seen Previously, when preparing reports account names were set based on the last data to be processed, even if that data was older than that used to originally set the account name. Now, when accounts are being synchronised during the report preparation process, names will only be updated if the data being processed is newer than that which was used to last set or update the name.

• Increased calculation precision in Transformer scripts The precision of calculations in Transformer scripts has been increased to 14 decimal places.

• Added new log files to the log rotation policy Log files generated by the Notification component and by the new Scheduler component are included in the log rotation policy.

• Added the option to run a workflow for a date range When running a workflow manually, you can now choose to run it for a date range, not only for a single date. When running against a date range the steps are executed individually for each day in that range.

• Redesign of the Data Pipelines > Workflows screen Complete overhaul of the Workflows page, improving the configuration user experience for individual steps and making it easier to review associated step logs.

• Added a cascading DELETE parameter to the API API now supports a new cascade: boolean query parameter. This parameter, if true, will allow the severing of non-nullable relationships and will remove the related record afterwards. This new parameter specifically applies to the workflowstep and workflowschedule endpoints.

• Data aware report filters Report filters only show entities available in the selected reporting date range

• Highly available scheduler The scheduler has been rewritten to support the new message bus architecture and has the ability to run on multiple nodes at the same time.

• Creating a new report now requires the "all accounts" permission Only users that have access to all accounts will be able to create new reports. This avoids situations where a user could previously create a new report that was not visible after creation.

• API support for changing user login method It is now possible to change an existing users authentication method to local, LDAP or SAML2. Currently this is only supported by invoking the the REST API.

• Changed behaviour of logout when Single Sign-On is configured Exivity now allows the logout endpoint (SLO endpoint) to remain unconfigured, in which case the user will be logged out only from the application, not from the entire Single Sign-On environment.

• Budget notification

• Summary report notification Users can now choose to receive a pre-defined report in either CSV or PDF format through a notification

• File available notification

• Admin managed notifications Admin users and users with the manage_users permissions now have the ability to add notification channels and subscriptions to other users. This allows admin users to grant users subscriptions to notification types they would otherwise not have.

• Minimum commitment and Adjustments included in cost summary subtotals Details of minimum commitments and adjustments are now included in the cost summary subtotals for service categories.

• Webhook notification channel Allow webhooks as notification channel, forwarding the original event as JSON payload to a custom webhook URL

• Implement internal communications between components using RabbitMQ message broker This allows for improved multi-node setups

Bug fixes

• Fixed an issue with manually created accounts When creating accounts manually in the GUI, on occasion an account could be created which did not fit correctly into the account hierarchy. This has now been fixed.

• Fixed and issue with updating account names When preparing a report, it could be that some account names were not updated when they should have been. This has now been fixed.

• Fixed saving service resources including 'budgetitems' relationship When saving a service resource including the budgetitems relationship, an error was returned. This has been fixed and the relationship between service and budgetitems works both ways.

• Fixed an issue with some combinations of HTTP methods in extractors In extractor scripts, making an HTTP DELETE request could cause subsequent HTTP calls to continue to use the DELETE method even if another method was specified. This has now been fixed.

• Added the 'notificationsubscription' relationship for a user entity When requesting a user entity, the notificationsubscription relationship can now be included in the response.

• Audit screen freeze Fixed an issue where the audit screen could become unresponsive

• Fix Publish Report notification The clear option for the account field now works.

• Fix state update adjustments Fixed an issue where a field wouldn't be updated when commuting to the next resource.

• Subscription inherited rates Fixed some incorrect behavior with rate inheritance on the subscription management screen

• Removed the limit of CSV files that could be open at once in extractor scripts Previously, a maximum of 16 CSV files could be open for writing in extractor scripts. This limit has now been removed and any number of CSV files can be open simultaneously

• Fix decimal inputs Fixed an issue where an input would not accept a decimal value with leading zero.

• Lookup file Fixed an issue where switching between lookup files wouldnt cause the screen to be updated

• Show manually created service Solved an issue where manually created services where not visible before a browser refresh

• Disclaimer modal Fixed an issue where the disclaimer had to be dismissed twice

• Fixed an issue with compound MSSQL statements in extractors where the first statement is not a SELECT Usually, SQL statements executed in an extractor script are based around some form of SELECT. However in some cases more complex compound statements are required. Previously, a statement using a temporary table in MS SQL server, such as ... CREATE TABLE #jos(Name char(20)) INSERT INTO #jos (teller) VALUES ('2') SELECT * FROM #jos; ... would not return any results, as the INSERT statement returns an empty result set. Extractors now ignore empty result sets in such statements and return the first result set that is not empty (assuming there is any data returned by the query at all).

• Fixed an issue with transformer scripts when importing data with non-escaped quotes Importing data with non-escaped quotes could sometimes cause Transformer scripts to fail. This has now been fixed.

• Fixed types for returned attributes When accessing various endpoints, id attributes were returned as integers instead of strings. This was fixed to comply with JSON:API specifications.

• Fixed incorrect display of date fields in slack notifications When a slack notification was sent, the date of the notification was wrongly set to sometime in the future

• Fixed incorrect display of extra HTML tags in emails When emails are sent, some HTML tags were shown as text in the email body

• Fixed an issue with encoding of double quotes when exporting account data as CSV

• Fixed long delay when displaying a large number of budgets when including reports

• Fixed an issue with displaying workflow start date in dashboard Previously the start date of a workflow was displayed incorrectly in the Workflow dashboard widget

• Fixed an issue with creating adjustments There was a problem where the GUI did not allow the creating of adjustments that had a non-integar amount. This has now been fixed.

• Fixed a divide-by-zero error on reports In corner cases where the final chargeable quantity for all line-items on a report is 0, a divide-by-zero error would occur. This has now been fixed.

• Fixed the description of the 'Budget' permission This permission only provides access to the Budget report. In the previous version, the description of this group permission contained false information.

• Fixed an issue when configuring a Budget for a leaf node If a leaf node in the budget structure has the distribution set to even or account, the Budget engine generated a false warning that an amount for the node is left undistributed. This has now been resolved.

v3.4.3

March 22, 2021

New features

v3.4.2

February 24, 2021

Bug fixes

• Fixed an issue with SAML user account access provisioning Previously, user account access provisioning would grant access to a matching account, including all its child accounts. With this release, the SAML user account access provisioning only grants a user access to the first matching account (i.e. with the lowest depth) in the account hierarchy.

• Fixed an issue where reports could become invalid for certain user account permission configurations

• Revolved a race condition related to preparing a report On rare occasions the actions of multiple users administering accounts and reports simultaneously could result in duplicate charges in reports. This has now been fixed.

• Fixed a very rare issue with minimum commit Resolved an issue whereby it was possible on extremely rare occasions for the minimum commit calculations applied to one service to be incorrectly applied also to other services in a report.

v3.4.1

December 08, 2020

Bug fixes

• Fixed an issue with PDF export of the summary report In certain environments, the PDF export feature was broken. This issue has now been resolved.

• Default SAML user group re-applied when a user logs in An issue has been resolved which caused the default user group to be re-applied when existing users were logging in through a SAML Identity Provider.

v3.4.0

November 17, 2020

New features

• Executing a Reports now requires a valid license

Bug fixes

• Fixed a minor issue with the calendar widget

• Fixed a minor issue with the SAML SLS endpoint

• Fixed an issue with custom rates and subscriptions

  Sometimes a subscription would not reflect changes in custom rates for a service. This has now been fixed.

v3.3.0

November 02, 2020

New features

• Improved user provisioning (beta) Added options to provision user attributes from a SAML Identity Provider/AD response. It's possible to provision the users display name, username and email address.

• Added claims based account access provisioning (beta) Added options to provision users permission levels using attributes from a SAML Identity Provider/AD. It's possible to provision both the usergroup and user account access (by matching either the account key or metadata value to a SAML/AD response attribute).

Bug fixes

• Change to log filenames for extractors and transformers When scheduling extractors and transformers with workflows, the selected environment is now part of the log filename to distinguish their log files when they are running at the same time.

• Fixed opening curly brace detection When opening curly brace wasn't preceded by space, sometimes it wasn't properly processed. It has been fixed

• Improved error message on lookups screen Fixed an issue where a user might get a confusing error message when saving a lookup file.

• Fixes initializing directories in home directory There was a bug when the home directories were initially created and two directories were wrongly created in the /home directory.

• Improved support for UTF-8 in usernames

v3.2.7

October 09, 2020

New features

• Added support for the Safari web browser

  Exivity now supports the Safari web browser

• Added ability to set custom escape character in Transformer Previewer Added a dropdown to the transformer previewer where the client can select which escape character to use during Transformer preview mode

Bug fixes

• Fixed an issue with the Dataset Manager Resolved regression issues regarding the display of RDF dates

• Fixed an issue with report filters There was a bug if a user accidentally submits a string instead of a number on filtering 'parent_account_id' with report. This is now being fixed

• Fixed an issue with displaying workflow start times in the workflow list The time column of the workflow list omitted the hour at which the workflow would run for daily and monthly workflows

• Resolve global variables in "uri" statement In some statements (i.e. uri) global variables weren't properly resolved. This has now been fixed

• Fixed an issue with COGS charges for services with an average monthly charge model COGS charges for monthly services that used the average charge model could be slightly lower than they should be, as the COGS rate for the first day seen was not factored into the rate averaging. This has now been fixed

• Fixed an issue when managing rates An aggregation_level error could occur when updating rates. This has now been resolved.

v3.2.5

September 30, 2020

Bug fixes

• Improved error message when creating invalid revision When creating a duplicate rate revision for the same effective date, an appropriate error message is now shown

• Improved error message for LDAP Added a more meaningful message when unable to connect to an LDAP server

• Service Category names must be unique It was possible to have duplicate service category names. This could lead to confusion and has now been resolved. The Service Category name must now be unique

• Resolved a timezone offset issue for Workflows Resolved an issue where timezones were incorrectly applied when creating workflows

• Resolved an issue with the Reports menu Fixed an issue where the report screen would crash after selecting multiple reports

v3.2.4

September 9, 2020

Bug fixes

• Increased the boundaries of some USE arithmetical operations

  The operators +=, -=, *= and /= were limited to a 32-bit range and this could cause an overflow in certain real-world applications. The range has now been increased to match that of the other arithmentical operations which is based on the range of a 64-bit floating point value: 1.7E +/- 308 (up to 15 digits).

• Fixed an issue where the CSV export of the instance report could fail The CSV export of the instance report did not work when the usage column was included and some of the instances did not have any usage data

• One-off subscriptions should ignore the charge_day field Only recurring subscriptions require the "charge day" field, but it was required for all subscriptions. This has now been fixed

• Fixed a decimal precision issue with the timeline graph The Y-axis of the timeline sometimes displayed long floats. To resolve this the precision has been fixed to two decimals

• Fix for "space" as thousand separator It is possible to select different symbols as a thousand separator for large numbers shown to users. One of the options, space (" "), was not saving correctly. This has now been resolved

• Fix error handling transformer Fixed an issue where it was possible that a failed transformer error was not processed properly which prevented the transformer from giving the user feedback

• Added index to speed-up report preparation During report preparation, Edify executes several queries to the adj_lookup table. An index has been adding to this table to improve performance

v3.2.1

September 1, 2020

• Quantity and Charge Adjustment are now handled in strict order As of this release, quantity adjustments are applied first and before any charge related adjustments. This in turn enables the ordering of individual adjustment policies.

• Implemented translations settings (Beta) Implemented an automatic translation feature for Dutch, German, and French. This can be configured on a system or user level. Currently, this is released as a beta feature.

• Delete old Nginx log files Old Nginx log files will now be deleted by the garbage collector. This will help reduce disk space.

• Improve user error page Updated the error page to improve user experience. Customer logo will now be shown here, if it has been set.

• Adjustment API endpoint will now accept order In the past, there was no way to reorder adjustments. Users had to delete and recreate them to do this. This has now been solved. The order field now takes in an order number. If this is not used, the current functionality will still work, using the creating date instead.

• Reports can now include adjustment name The adjustment name can now be added to a report. This is useful for report users that do not have access to view the whole adjustment.

• Support in UI for changing the order in which Adjustments are applied Since it is possible to create multiple Adjustments for a single account, they may affect each other. It is therefore desirable to have the ability to control the order in which they are applied. As of this version, it is possible to change the order in which Adjustments are applied.

Bug fixes

• Fixed an issue when partial preparing manual services When partial prepare kicks in, it previously skipped manual services. This has now been resolve.

• Fixed SAML ACS invalid schema error Some users were experiencing an Invalid Schema bug when accessing the /v1/auth/saml/acs endpoint. This has now been fixed.

• Fixes "Nginx Log File" not found error The location of some log files could not be found. This has been resolved.

• Improved invalid token handling Changed how this error is logged. It will now be treated as a notice instead of a warning.

• Fixed an issue with running transformers

  In release 3.2.0 there was a problem running transformers from the GUI. This has now been fixed.

v3.1.5

July 21, 2020

Bug fixes

• Premature validation warning

  Resolved an issue where a user could get a premature validation error for certain input fields.

• Fixed an issue with the budget report

  With certain budget configurations, the budget report would display a message instead of the report even if the budget configuration was valid. This has now been resolved.

• Improved user message when execution time is exceeded

  Improved the message when the execution time of an Extractor, Transformer or Report is being exceeded

• Fixed an issue with budget leaf account distribution When configuring a budget for a budget leaf, the distribution setting will be forced to be of the type "shared"

• Fixed an issue when deleting services Previously it wasn't possible to delete a service if it still had any associated rates. This behavior has now been corrected so that services including its relationships will be deleted

• Fixed an issue with the Garbage collector exits code The Garbage collector would previously always return an exit code of 259 when invoked manually. This has been resolved

• Fixed an issue where subscriptions could show up on more than one report In some cases, a subscription (created for an account associated with a specific report) could be shown when a different report was run. This has now been fixed

• Fixed graph image export When exporting a graph chart (i.e. Pie, Bar, Line) in any of the reports (Accounts, Services Instances) the file download would not start. This has now been resolved

• Restored indicators for account access list Pencil-like indicators will inform the user where nested accounts have been selected. This functionality was unintentionally removed from the user management screen but has been placed back

• Horizon performance improvements Implemented two seperate changes to improve Horizon budget execution performance. (1) budget gets validated only once after a budget configuration change and (2) added a new database index to improve database query execution

• Always show decimal values Resolved an issue where some values would be displayed using their scientific notation

• Resolved issue when deleting services Fixed an issue where the user would get an error when making multiple delete requests for services

v3.1.2

July 3, 2020

New features

• Ability to skip database backup when updating When updating the software, the user can now skip the PostgreSQL database backup to improve update speed

Bug fixes

• Fixed an issue with the webproxy.conf proxy_pass URL

  When updating the software, the proxy_pass URL for the webproxy.conf NGINX configuration was always reverted back to https://127.0.0.1:8002. This has now been resolved

v3.1.1

June 28, 2020

New features

• Improved auditing when managing accounts manually Added a number of new audit points when performing manual account management

• Auditing token creation correctly logs client IP address when API is behind a proxy server

• Updating user profile information now requires providing the current password

• Implemented current password validation A user is now required to enter his/her current password before any changes to its profile are applied

• Added logging audit entries for metadata, service subscriptions, and budgets

Bug fixes

• Fixed an issue where some users could select budgets even if it would not contain any details

• Fixed an issue which sometimes caused the workflows screen to load for a long time A resource-intensive operation that ran whenever the workflow page is visited got removed

• getCUPRs function ported to PSQL getCUPRs function was changed from using SQLite global DB to PSQL global DB

• Mitigated a potential security issue (internal reference EXVT-3773)

• Fixed an issue were sometimes log files would appear malformed in the Transformer run tab

• Reverted the way a user deletes an RDF

• Fixed form behavior in rates screen

• Fixed "other" option behavior at services

• Fix report filter When switching reports, some of the filters were not reset and appeared unused while they actually held a value not related to the currently selected report. This made the reports appear empty. This has been resolved.

• When the API runs out of memory (possible for very large reports) it will respond with a descriptive message

• Incorrect display of minimum_commit_delta_charge and minimum_commit_delta_quantity When a minimum commit quantity was set for a service rate revision, the resulting charges could show up incorrectly by having the quantity and minimum commit delta swapped on the reports

v3.0.5

May 26, 2020

Bug fixes

• Fixed issue at /lookups where in specific cases an error was undefined

• Fixed issue where user was shown an incorrect validation warning When providing a number with more then one decimal, the user interface would show an error message. This has been resolved.

• Added cache check in partial preparation There was an issue that partial preparation code didn't check for cache table presence before analysis, which caused execution errors, and it has been fixed.

• Fixed escape option behaviour in 'import' There was an error that '\"' sequence was always treated as escaped quoted even when 'escape' option switched off (default), it has been fixed.

• Rate revision date never changes Transcript was trying to change rate revision date when it detects an attempt to create new revision with the same attributes but earlier date, which conflicted with database constraint, so it was removed in order to avoid execution errors.

• Fixed issue where line breaks were not shown When providing an address for the summary report with multiple new lines, only the first would be shown. This has been resolved.

• Fixed undefined tool tip at workflows The was a column which showed a tool tip with undefined as value. This has been resolved.

• Select a single date for transformer run After introduction of a new calendar, the single date selection was missing. This has now been resolved.

• Services and categories at subscriptions are now alphabetically sorted

• Fixed an issue where dataset columns were missing After repreparing a report, the columns associated with a dataset were not visible. This has been resolved.

• Fixed visibility newly created dataset When running a transcript, newly created datasets were only visible after refresh. This has been resolved.

• Show red underline when incorrect value for textarea input

• Show correct validation values when saving environment

  When saving an environment, we will now show the correct error messages for missing values.

v3.0.4

April 30, 2020

New features

• New feature: Workflows widget When Beta features are enabled, a new Workflow traffic-light widget will be shown on the main Dashboard page.

• Added more information to the summary report The summary report now contains the following additional columns: service_key, account keys (between 1 and 5 inclusive depending on the report), start_date, end_date. These columns are also included in CSV exports.

• Improved quoting in exported CSVs To avoid potential complications with Excel, any cell values in CSVs exported from Exivity that are not numbers and begin with any of the characters =,+,- or @ are now preceded with a single quote.

• A new setting has been added to allow users to stay logged in. If disabled (default), users need to log in each time they open Exivity in their browser.

• Administrators can now set the token lifetime. When the token lifetime expires, users need to log in again.

• Added ability for users to log out of all devices.

• Mitigated a potential security issue (internal reference EXVT-3457)

• Mitigated a potential security issue (internal reference: EXVT-3455)

• Mitigated a potential security issue (internal reference EXVT-3270)

• Implemented a new charge model for monthly services The charge for a monthly services may now be based on the usage of a specific day in the month.

• Improvements in report performance Reports are now pre-loaded in the database which speeds up report loading times in the GUI.

• Added support for LDAP authentication LDAP authentication was available as beta feature already, and is now generally available. A guide will be added to our documentation soon. Configuration options are available in the Settings screen (Single sign-on tab).

Bug fixes

• Fixed an issue with manage_metadata_definitions Granting the "Manage Metadata" permission is now possible when creating a security group

• Fixed an edge case for incorrect net value on accounts table Whenever cogs or charge had 0 as value the net value wouldn't be displayed correctly on the details table of the accounts report.

• Fixed an issue when creating a manual service Manual per unit cogs type services will not require cogs column

• Fixed invalid format issue metadata lists Previously empty lines we're persisted as list option. Options are now trimmed and validated before persisting.

• The API now requires additional attributes for certain service types

• Fixed a small issue with the metadata selector in the report management screen

• Removed toolbar from PDF export The Summary report PDF export included the toolbar on the top of every page of the PDF document. This has now been removed.

• Fixed an Extractor XML parsing issue USE could previously fail when iterating over an empty XML node inside a foreach loop. This has now been resolved.

• Fixed modified display issue in Accounts Overview When switching from Account Name to Account Key in the Accounts Overview screen, the modified state / pencil icon disappeared previously. This has now been resolved.

• Fixed an issue when scrolling in the Accounts Overview When having a large amount of Accounts in a report definition, the Accounts Overview screen could generate cosmetic glitches when scrolling

• Fixed a Transformer issue with skip_corrupted_records When import option skip_corrupted_records was set, import could fail if last column in the record is missing closing quote. This has been resolved.

• Fixed a Transformer issue with aggregate When using the aggregate max function, Transcript could pick the wrong value.

• Fixed a false warning about invalid COGS type in the logfile When preparing reports, in some cases a lot of warnings would appear in the logfile that state: Invalid cogs_type (0) in database for service ID nnn where "nnn" is a service ID. This was a false warning and could be ignored but looked concerning and could lead to larger logfiles. This issue has now been fixed.

• Removed "remainder" option from lowest level accounts when setting budgets When a budget is set for an account at the deepest report level, the "remainder" option is no longer shown, as there are no sibling accounts to share the budget with.

• Fixed an issue where (harmless) SQL errors could be logged when deleting services When deleting services, SQL errors could be present in the logfile for days where reports that reference those services had not been prepared. This has now been fixed.

PostgreSQL

The biggest single change is the use of a new database engine powering all application state, audit logs and processed report data. Upgrading to this new database engine is transparent and the installer will take care of installing the database server as part of the regular installation process. After or during upgrade, it is possible to leverage an .

Windows Services

In version 2.x.x there were only two Exivity Services installed:

With Exivity version 3.0.0 up to version 3.4.3, assuming all components are installed on a single host system, there was a total of 4 different services:

As of version 3.5.0, the following Exivity services are to be considered when all components are deployed on a single host, there will be 7 different services:

In case you were using a service account in Exivity version 2.x.x for the Exivity Scheduling Service and the Exivity Web Service, you will have to reconfigure this service account for both services, as well as the Exivity API Service. In most cases, the Exivity Database Service may continue to run un the Local System account.

Default TCP ports

In v2.x.x, the default port for the Exivity GUI was 8001 and 8002 for the Proximity API. Both services were already available through port 443 (the default port used for https traffic, which means clients don't have to explicitly specify the port) and in v3.x.x this will be used by default:

Default Security Settings

As of Exivity version 3 more strict security settings are applied by default. These can be found under Administration > Settings. One important item which should be considered when upgrading a multi node environment, is the use of CORS. It is required to list all possible front end UI nodes in the CORS origins field:

Transformer changes

@FILE_EXISTS and @FILE_EMPTY

Previously, it would only check for files in the system and exported folders within the Exivity home directory, and if a specified path + filename did not start with system/ or exported/ then these would be prepended automatically before the check as done.

This behaviour has been changed in the following ways in v3.x.x:

• Any path + filename within the Exivity home directory can now be checked

• path + filenames are now accepted and treated as being relative to the Exivity home directory

• The folders system/ and exported/ are no longer automatically prepended

Consider a file somefile.csv in the %EXIVITY_HOME_PATH%/exported folder. Previously with version 2.x.x a user could check for the existence of this file using the following example:

In version 3 it is required to include the entire path relative to the %EXIVITY_HOME_PATH% :

Extractor changes

HTTP server certificate validation

The default behaviour of the HTTP subsystem in USE was changed to fully validate server SSL certificates, which may cause some USE scripts to fail. This typically applies to data Extractors, which are connecting to on premise data sources that use self signed SSL certificated. In version 2.x.x, the default behavior was to ignore these certificate errors. You can identify these errors in your Extractor logs:

Extractor & Transformer Schedules

GUI changes

There are some minor changes in the GUI which are not backwards-compatible:

• We removed the Excel export options from the reports. They were using the CSV format under the hood (i.e. they never actually produced valid Excel worksheets). In the future, we plan to implement proper Excel export formats for the reports, including a full summary report Excel export.

Changes to data processing for reports

Due to changes to the processing of reports, when making changes to either services, rates or adjustments, associated reports should be prepared. In v2.x.x this was already required when making changes to services or rates. Since v3.x.x this will be also required when making changes to adjustments. We've planned further improvements making this more transparent (i.e. handling preparation of reports automatically in the background).

API changes

Normalised date/time data in responses

Some endpoints were returning dates and timestamps in different formats. This has been normalised in such a way that all responses use the same serialisation for dates and timestamps:

• A date is always represented as ISO-8601 string: "yyyy-mm-dd", e.g. "2020-01-29"

• A date/time is always represented as ISO-8601 string in UTC: "yyyy-mm-ddThh:mm:ssZ", e.g. "2020-01-29T11:26:52Z". Note that the Z suffix denotes the UTC time standard.

This affects the responses (attributes) for the following group of API endpoints:

• /v1/audit

  • created_at

• /v1/budgetrevisions

  • effective_from

• /v1/dsets

  • earliest_rdf

  • latest_rdf

  • rdf_detail.created

  • rdf_detail.updated

• /v1/extractors

  • last_modified

• /v1/log

  • created

  • lines.date

• /v1/reports

  • data_status.first_date

  • data_status.last_date

  • data_status.status.date

• /v1/services

  • created_at

  • updated_at

• /v1/transformers

  • last_modified

• /v1/workflows

  • created_at

  • updated_at

• /v1/workflowschedules

  • start_time

  • next_run

Changed functionality

/v1/reports/{id}/run endpoint

• The JSON format in the response is simplified. See examples below.

This is an example response from v2.x.x:

This is an example response from v3.x.x:

Remove deprecated functionality

/v1/usergroups endpoint

Removed deprecated permission aliases.

• upload_files (use manage_files instead)

• manage_configuration (use manage_settings instead)

• manage_system (use manage_settings instead)

/v1/reports/{id}/run endpoint

Functionality deprecated in v2.x.x has been removed. If you were relying on any of the following functionality, please use the suggested replacement instead:

• The pdf/invoice option for the format query parameter has been removed. Please use pdf/summary instead.

• The invoice_options query parameter has been removed in favour of summary_options.

/v1/configuration endpoint

• Configuration keys prefixed with INVOICE_ are replaced by respective keys prefixed with SUMMARY_.

/v1/workflowsteplogs endpoint

Also relevant to other endpoints including workflowsteplogs .

• Removed the last_log attribute. Include the last log by specifying the query parameters include=steplogs&related[steplogs][limit]=1&related[steplogs][sort]=-start_ts.

• Removed the timestamp attribute. Use start_timestamp instead.

• Removed the message attribute. Use a combination of status and output instead.

/v1/file endpoint

Only applicable to POST requests to this endpoint.

• The filename in the response from this endpoint will no longer include the import/ prefix to better align for other requests in this endpoint. See example below.

/v1/usergroups endpoint

The following permissions has been removed in favour of their new counterparts:

• UPLOAD_FILES has become MANAGE_FILES

• MANAGE_CONFIGURATION has become MANAGE_SETTINGS

• MANAGE_SYSTEM has become MANAGE_SETTINGS

An introduction to Services

In Exivity Services can be anything that corresponds to a SKU or sellable item from your Service Catalogue. It should relate to a consumption record (or multiple records) from your extracted data sources.

For example: with most public cloud providers, the provider defines the chargeable items that are shown on the end of month invoice. However, when working through a Managed Services Provider, a Cloud Services Provider or a System Integrator, additional services can be sold on top of those. Potentially, you may want to apply an uplift to the rate or charge a fixed amount of money every month for a certain service. Different scenario's are possible here, it all depends on your business logic.

Terminology

A service is a named item with associated rates and/or costs used to calculate a charge that appears on a report, where rates represent revenue and costs represent overheads.

When discussing services and their related charges a number of terms are required. Exivity uses the following terminology in this regard:

Creating service definitions

If the task does not complete successfully then the service definitions cached in memory are discarded, the expectation being that the task will be re-run after the error condition that caused it to fail has been rectified and the services will be written to the global database at that time.

Types of charges

There are different types of charge that can be associated with a service. Collectively these influence the total charge(s) shown on the report and Exivity supports the following charge types as described in the Terminology table above:

• unit rate

• COGS rate

At least one of these charge types must be associated with a service definition.

Once the resulting charge has been calculated based on the charge types, it may be further modified through the application of adjustments, proration and minimum commit (all of which are detailed later in this article).

Charge intervals

In order to calculate the charge(s) associated with usage of a service Exivity needs to know the period of time for which each payment is valid. For example a Virtual Machine may have a daily cost associated with it, in which case using it multiple times in a single day counts as a single unit of consumption whereas Network Bandwidth may be chargeable per Gigabyte and each gigabyte transferred is charged as it occurs.

The charge interval (also termed simply interval) for a service can be one of the following:

• individually - the charge for a service is applied every time a unit of the service is consumed, with no regard for a charging interval

• daily - the charge is applied once per day

• monthly - the charge is applied once per calendar month

Charge models

Monthly services may be charged in different ways:

Peak

For each day of the month, a 'candidate' charge is calculated using Quantity * Unit Rate . The monthly charge will reflect the day of the month which resulted in the highest charge.

In the event that multiple days share the same highest charge then that charge will be associated with the first of those days seen, unless a subsequent day in that set has a higher quantity, in which case the charge will be associated with that subsequent day.

Average

The average unit rate for those days where usage was seen in the month is calculated, and multiplied by the average quantity for each day in the month. When calculating the average quantity, any days for which there was no consumption are factored in as having a quantity of 0.

Specific Day

The charge is based on the quantity consumed on a specific day of the month.

Last Day

The charge is based on the quantity consumed on the last day of the month.

Minimum commit

The minimum commit is the minimum number of units of consumption that are charged every interval, or (in the case of services with an interval of individually) every time the service is used. If fewer units than the minimum commit are actually consumed then the service will be charged as if the minimum commit number of units had been used.

Proration

After the charge for usage of a monthly service has been determined, it may be prorated by modifying that charge based on the frequency of the usage.

This process will reduce the charge based on the number of days within the month that the service was used. For example if consumption of a service with a monthly charge interval was only seen for 15 days within a 30 day calendar month then the final charge will be 1/2 of the monthly charge.

Service definitions

A service definition comprises two categories of information:

1. The service - Metadata describing fixed attributes of the service such as its name, description, group, interval, proration and charge type(s)

2. The rate revision - Information detailing the charge type(s) associated with the service (the rate and COGS values) and additional information detailing the date(s) for which those values should be applied

The following tables summarize the data members that comprise each of these categories:

Service attributes

Rate revision attributes

The rate_col and cogs_col are used when the specific value to use is derived at report-time from the usage data, as opposed to explicitly being included in the rate revision itself.

A service may have any number of associated rate revisions so long as they have different effective_date or minimum commit values. This means that a service can have different charges applied depending on the date that the report is to be generated for, or depending on the specific values in the columns used by a report.

A service may use either or both of rate and cogs.

Stacking of Quantity and Charge Adjustments Policies can cause negative charges

Affected versions

All versions of Exivity

Description

When creating two Absolute Discount Adjustment Policies, where both Policies apply to the same Account, Service and date Period, there is a chance of generating negative charges for the services where both Adjustment Policies apply to as can be seen in the following screenshot:

Workaround

The current workaround is to avoid stacking multiple Absolute Adjustments for each unique Account + Service combination. Instead, try to combine them within a single Adjustment Policy

Status

This issue is currently being resolved (internal reference: EXVT-<TBD>) and will be included with the release after version 3.5.7.

Installing or Upgrading to version 3.5.6 populates invalid RabbitMQ settings

Affected versions

Exivity versions 3.5.0 till 3.5.6

Description

When installing or upgrading to version 3.5.0 - 3.5.6, when installing RabbitMQ on the local system, the initial configuration values are incorrectly populated:

Workaround

To work around this issue, you may uncheck the checkbox Install Local RabbitMQ Engine and then check the same checkbox again. The values will now be correctly populated.

In case you have already executed the upgrade progress and your system is broken, you will need to change the config.json to align the mq settings:

Status

This issue is currently being resolved (internal reference: EXVT-5042) and will be included with the 3.5.7 release.

Discount Adjustments are created as Premium

Affected versions

Exivity versions 3.5.0 till 3.5.5

Description

When a new Adjusmtent Discount policy is created, it will be created as a Premium policy instead. To confirm this behavior, when refreshing the screen they will show up as Premium:

Workaround

In order to create a discount policy in version 3.5.5, use the API interface to create one.

Status

This issue is resolved with the 3.5.6 release.

Installer upgrade generates an error: Error opening file for writing

Affected versions

Exivity versions 3.5.0 and higher

Description

When running the installer, an error could popup where the installer is unable to update certain files such as the erlsrv.exe executable:

Resolution

This issue occurs when Windows or a 3rd party application locks this file. This can happen with certain monitoring and anti-virus software. In the example given above, the solution would be temproraly stopping the Windows Event Log service, as shown in the screenshot below:

After stopping the service, click the Retry button in the installer.

Relative quantity discounts can cause negative charges with prorated monthly services

Affected versions

All version of Exivity

Description

When configuring a prorated monthly service, and then applying a relative quantity discount (adjustment), in certain cases the total charge can be less than 0 resulting in a credit. Although this might be an unusual configuration and might not always happen, it is important to be aware of this behavior.

Status

This issue is pending. (internal reference: EXVT-1337)

Exivity Backend Service does not start / access to merlin.exe has been denied

Affected versions

Exivity version 3.5.0 and higher

Description

Some Anti-Virus software vendors (specifically: McAfee and APEX) incorrectly flag the Exivity binary merlin.exe as a backdoor/trojan horse. As a result, the Exivity Backend Service might not be able to start as shown in the example below:

The Windows event viewer may display the following error (depending on your A/V vendor):

Resolution

Response by certain A/V vendors:

[merlin.exe] is detected by [a coupe of] AV's as a BackDoor as it's functionality is very similar and malware authors can use this to their own advantage to compromise systems and gather sensitive system information.

We also understand that [merlin.exe] can be used for legitimate purposes and therefore we will change the classification of the sample submitted to PUP. This way will be able to exclude detection through our scanner configuration but at the same time we are still protecting our customers who may be not be aware of this software running on their systems.

Hence if you want to allow this application in your environment kindly exclude it by detection name.

Announcement regarding CVE-2021-44228

Exivity does not ship any Java-powered software components. As such the Exivity software solution is not affected by . For more information regarding this matter, please file a request at the Exivity or send an e-mail to .