Exivity is a metering and billing software solution for public and private cloud environments that allows you to report on cloud consumption from any IT resource. Exivity enables you to apply your MSP/CSP business rules and makes any type of Pay-as-you-Go model work. It also facilitates internal charge-back and show-back requirements for Enterprise IT.

These things are done by extracting IT consumption data from various endpoints and then mapping this data to meaningful customer-specific information such as services, customer IDs, names, and contracts.

There are four main steps involved in a successful deployment:

1. Extract

2. Transform

3. Report

4. Integrate (optional)

Extract

The Extract step defines your data sources such as:

• APIs that return usage data, service catalogue, rate card, customer/subscriber lists, and similarly available records from public or private clouds

• APIs or ODBC queries that return contracts, customer names, IDs, and other contextual lookup data from CMDB / CRM systems

• Flat files on disk in CSV, JSON, or XML format

Exivity provides a rich scripting interface via its Unified Scriptable Extractor (USE) component which facilitates integration with almost any data source. For most of the big cloud platforms, we provide template extractor scripts as part of the product. Additionally, you can also write your own USE scripts from scratch in order to integrate with custom data sources.

Transform

The Transform step provides a powerful for processing extracted data. Using it you can merge consumption metrics, contract details, customer information, custom metadata, service definitions or any other imported information to produce an enriched and/or normalised result.

This is done using the Transcript component, which executes user-definable scripts (termed tasks) in order to produce a meaningful set of data suitable for reporting against. Often this data will feed a consolidated bill of IT based on the various different consumed services.

Transcript also allows you to define and populate services and rates, either of which may be passed through from cloud data, defined as custom offerings, or a mixture of the two.

Report

Exivity provides a modern responsive User Interface, that allows you to 'slice and dice' the processed data in any way you choose. Multiple Report Definitions can be created with ease which allows you to graphically and textually display both cost and usage statistics.

Integrate

We think that Exivity should be part of your automation landscape, where it can provide (for example) line items that can be digested by your ERP and/or invoicing system. Therefore we consider Integrate as the logical final step for any deployment where it is useful.

To this end, we offer an open and fully-featured REST API. Our GUI uses our own API for all back-end processes meaning that all textual data shown in the Exivity GUI is also obtainable via our API.

Detailed information about the REST API can be found at

Announcement regarding CVE-2021-44228

Exivity does not ship any Java-powered software components. As such the Exivity software solution is not affected by CVE-2021-44228. For more information regarding this matter, please file a request at the Exivity support portal or send an e-mail to [email protected].

Exivity provides a catalogue of USE extraction scripts that can be used to integrate with almost any cloud provider, hypervisor or legacy IT end point. We've published some of our templates on GitHub for your convenience.

This repository contains Extractors for VMware, Azure, Amazon and others. However, if you are currently missing an integration template and are unwilling or unable to create your own, feel free to drop us an e-mail at [email protected].

The gosub function in a data Extractor allows invoking a subroutine in a data extractor script. Subroutines are useful to avoid having to duplicate code snippets.

Our customers and Solution Architects have created several useful subroutines over the last few years. We are keeping a small library of the most useful ones on our public docs. You can find them listed on this page.

The documentation in this section assumes knowledge of the USE script basics.

This Subroutine checks if a date is in the YYYYMMDD format, if not then it raises and error.

Syntax

gosub check_dateformat ("YYYYMMDD")

Code Snippet

subroutine check_dateformat {
	match date "^(([0-9]{4}(0[1-9]|1[0-2])(0[1-9]|1[0-9]|2[0-9]|3[0-1])))" ${SUBARG_1}
	if (${date.STATUS} != MATCH) {
		print Argument error: ${SUBARG_1} is not in YYYYMMDD format
		terminate with error
	}
}

The Instances report will provide an overview of all chargeable resources. It is recommended to apply filters to this report. Therefore the best approach is to drill down into Instances from the Accounts or Services report. This allows a user to easily drill down into all resource instances for a particular Account, Service, or both:

After configuring a budget, it is possible to report on current budget spendings. To do so, first browse to the Reports > Budget screen and select a Budget from the drop-down list:

Also, make sure to select a date range that matches with one or more of the configured budget revisions. Once selected, the budget report will start loading. This may take a few seconds, and a report similar to this will be shown on your screen:

Exivity uses two main disk directories when it's installed. One to store the user data and one to store the software files. These directories are called the home and program directories, respectively.

On the system where Exivity is installed, the following environmental variables contain a absolute path reference to these directories: EXIVITY_HOME_DIRECTORY and EXIVITY_PROGRAM_DIRECTORY.

Home directory

The feature in Exivity can be leveraged to store relevant information about contracts (like Contract ID, Customer ID, contact details, etc.) together with in report summaries. First, we need to create a Metadata definition, then associate it to one or more . These are the steps to achieve this:

1. Navigate to the Data Pipelines > Metadata menu and click Create Metadata.

2. Choose a meaningful name for your Metadata definition.

The Services report provides the ability to report on your metered-based IT consumption costs from a Services perspective.

Once logged into the system, navigate to the Reports > Services menu. From here you will get a report grouped by the services consumed. This report can be refined using several filters.

Filters and Reporting Depth

Once you have selected your date range and report you can start viewing your data. By default, it will show all consumed services for this report for the accounts you have permission to access. If you want to limit your view you can change the reporting '

The Summary report provides a detailed breakdown of costs in an invoice-like format. The defined in the system determine what you'll see here.

Once logged into the system, go to Reports > Summary. From here you're able to generate different kinds of detailed costs reports, which can be used for billing, chargeback and showback.

Filters and Reporting Depth

Once you have selected your date range and chosen which report you to activate will be presented with the summary cost report as shown above. By default it will show all consumed services for an account. As accounts are hierarchical, this view will include consumed services for all children of the selected account. Therefore it is important that you select an appropriate

This Subroutines checks if the FROM and TO date are in order. In the case that there is only 1 day entered, it will automatically fill in the second day in a "1 Day Mode".

Syntax

Code Snippet

This Subroutine extracts the day, month and year from a given date in YYYYMMDD format.

Syntax

Code Snippet

This subroutine allows validating an HTTP response from a .

Syntax

Code snippet

The clear statement is used to delete all HTTP headers previously configured using the statement.

Syntax

clear http_headers

Syntax

decimal_to_ipv4variable_name

decimal_to_ipv4source_variable_nameas

The discard statement is used to delete a named .

Syntax

discard{buffer_name}

The exit_loop statement will terminate the current loop.

The environment statement specifies the name of the environment to use for resolving .

Syntax

environment name

Exivity has a powerful Workflow engine and Notification engine. These two combined allow users to receive their billing reports monthly by email or other notification channels.

These are the steps to configure receiving your monthly billing report by email:

1. Navigate to the Notifications menu. Click on your username at the top right corner of the screen, then select My Notifications.

2. Fill in the details (Name, Title and Description) and make sure the Trigger is set to Workflow Ended.

3. Select the .

4. Apply the trigger of the Workflow Status to Successful workflows:

5. In the Filenames section, it is possible to use to select which files to export.

For example, if you configured the publishing of a monthly report (on the 15th day of every month) for your Departments account, the file name will be similar to report-Departments-account-range-20220115-20220315.You can select your file with a regular expression whose name starts with the string "Report-Departments" and ends with "15":

/report-Departments(.*?)15/

In this example, the (.*?) in the regular expression has the role of matching any character 0 or more times, meaning that any string can be between "report-Departments" and "15".

You may also be interested in selecting only the files that start with a certain String. For example to select the files starting with "report-Departments", you may use the following regular expression:

/report-Departments*/

Or you may choose to select the files that end with a certain string, for example, ending with "15":

/&15/

6. In this scenario, you want to send the file along with the notification only if the file was created/modified after the start time of that workflow. To achieve this, tick the Enabled box next to Since workflow start time.

7. It is possible to compress the files by ticking the Enabled box next to Compress Attachments.

8. Select the . You may want to send the report to your email.

9. Finally, click the Create button.

Exivity enables customers to generate summaries of the cloud spend on a schedule. You may choose to publish a report of your resource usage and costs quarterly, monthly or daily. This article guides you through the steps of publishing a monthly report: In order to trigger the publishing automatically, you must use the Workflow engine.

1. Navigate to the Data Pipelines > Workflow menu and select Workflows.

2. Click Create Workflow.

3. Provide a meaningful name and description.

4. Click Add Schedule.

4.1. The Type selection should be monthly.

4.2. Furthermore, the interval field Run Every should be set to 1 Month.

4.3. Select the start date (Effective from), Start time and Time zone.

5. Click the Add Step button and set the type to Publish report.

6. Choose your report from the drop-down list.

7. The Timeout setting allows you to choose the interval of waiting, before trying to execute this step again, in the event of a failure.

8. Finally, click Create.

In order to have a fully functional solution, you need a valid license.

These are the steps to reset your license in case it has expired: 1. Navigate to the Administration > Settings menu.

2. Select the License tab and paste the new license serial key under the New license box.

3. Click the Update button. Afterwards, you should see the Valid status along with the expiry date.

The Accounts report provides the ability to drill down into your metered based IT consumption costs. The way you've created your Report, determines on what values you can zoom into.

Once you've logged into the system, navigate to the Reports > Accounts menu. Here are a few key parameters you can use to define how your report is generated:

Date Selection

The date selector is important to limit the scope of data you're focusing on.

First, select the date range you are interested in. This can be a single month, a 3-month time period, half a year, a full year, or a custom date range.

Report

Your Exivity solution can have more than one report definition created. If this is the case, you will need to select the appropriate report containing the data you wish to examine. An end user will only see reports listed that they have permissions to, and the first of those is automatically selected.

Drill Down, Services & Report Depth

After selecting a date range and report you can start drilling down into your data in a number of ways:

1. Moving your mouse over one of the accounts will reveal a toolbar as shown above. For each account, you have the option to click on the Drilldown control in that toolbar. This will do the following:

   1. Descends one level deeper into the report

   2. Updates your view of the data to reflect that deeper level

Adjustments allow a user to create account-specific rate adjustment policies. An Adjustment policy supports applying a discount or a premium using one of these modifiers:

1. a certain amount of money (i.e. $ 100)

2. a certain quantity (i.e. 100 GB/hours)

3. a percentage (i.e. 10%)

This Adjustment can then be applied to a single service, multiple different services, or one or more service categories.

Create an Adjustment Policy

To create a new Adjustment policy for an account, follow these steps:

1. From the menu on the left, select Services > Adjustments

2. Then select the Account from the list of accounts for which you want to create an adjustment policy

3. After selecting the account, click Add Policy, and provide a meaningful name for your policy in the right screen where it says Adjustment name

Introduction

Extraction is the process by which USE (Unified Scriptable Extractor) retrieves data from external locations. The following types of data sources are supported:

USE script

A USE script is required for USE to operate. Further information can be found via the links below:

An introductory overview of the scripting language:

A reference guide for the USE scripting language:

How to parse XML and JSON data

Template scripts that can be used as starting points for common data sources:

The basename statement is used to extract the filename portion of a path + filename string.

Syntax

basenamevarName

basenamestringasvarName

Details

Given a string describing the full path of a file, such as /extracted/test/mydata.csv the basename statement is used to identify the filename (including the file extension, if any) portion of that string only. If there are no path delimiters in the string then the original string is returned.

The basename statement supports both UNIX-style (forward slash) and Windows-style (backslash) delimiters.

As a convenience in cases where the full path needs to be retained, the result of the operation can be placed into a separate variable by using the form basename string as varName where string is the value containing the full path + filename and varName is the name of the variable to set as the result.

Examples

Example 1

The following script ...

... will produce the following output:

Example 2

The following script ...

... will produce the following output:

The encode statement is used to base16 or base64 encode the contents of a variable or a named buffer.

Syntax

encode base16|base64varName|{buffer_name}

Details

The encode statement will encode the contents of an existing variable or named buffer, replacing those contents with the encoded version.

The result of encoding the contents will increase their length. With base16 encoding the new length will be double the original, but the exact size increase will depend on the contents being encoded.

When encoding a variable, if the size of the result after encoding exceeds the maximum allowable length for a variable value (8095 characters) then the USE script will fail and an error will be returned.

Example

The following script ...

... produces the following output:

The escape statement is used to escape quotes in a variable value or the contents of a named buffer.

Syntax

escape quotes invarName|{bufferName}[usingescape_char]

Details

If a variable value or named buffer contains quotes then it may be desirable to escape them, either for display purposes (to prevent USE from removing them before rendering the data as output) or in order to satisfy the requirements of an external API.

The escape statement will precede all occurrences of the character " with a specified escape character (backslash by default) as shown in the example below. This operation is not just temporary - it will update the actual contents of the variable or named buffer.

Example

Given an input file called 'escapeme.txt' containing the following data:

The following script:

... will produce the following output:

The gosub keyword is used to run a named subroutine.

Syntax

gosub subroutineName([argument1, ... argumentN])

Details

The subroutineName provided to the gosub statement must be that of a subroutine defined elsewhere in the script using the statement.

If any argument contains white-space or a comma then it must be quoted:

gosub getfile("directory with spaces/filename.txt")

The opening bracket after subroutineName may or may not be preceded with a space:

gosub getfile ("filename.txt")

To call a subroutine with no parameters, use empty brackets:

gosub dosomething()

Example

Please refer to the example in the documentation for the statement.

The unzip statement is used to unzip the data in a named buffer.

Syntax

unzip{buffer_name}

Details

The unzip statement will extract a single file from a zip archive stored in a named buffer. In order for this to succeed, the buffer must have been previously populated using the statement, and the data within the buffer must be a valid ZIP file.

A warning will be logged, the buffer left intact and the script will continue to execute if any of the following conditions arise:

• The buffer is empty or does not contain a valid ZIP archive

• The ZIP archive is damaged or otherwise corrupted

• More than 1 file is present within the archive

After the unzip statement completes, the buffer will contain the unzipped data (the original ZIP archive is discarded during this process).

The filename of the unpacked file is also discarded, as the resulting data is stored in the buffer and can subsequently be saved using an explicit filename as shown in the example below.

Example

The print statement is used to display text to standard output while a USE script is executing.

Syntax

print [-n]word|{buffer_name} [... word|{buffer_name]

Details

The print statement enables user-defined output to be generated during the execution of a USE script. When retrieving data from external sources it may take some time for a lengthy series of operations to complete, so one use of the print statement is to provide periodic status updates during this time.

The print statement will process as many arguments as it is given, but at least one argument is required. If the first argument is -n then no newline will be output after the last argument has been echoed to standard output, else a newline is output after the last argument.

Arguments that are normal words will be sent to standard output followed by a space. Arguments referencing a named buffer will result in the contents of the buffer being displayed.

Note that print will stop output of data from a named buffer as soon as a NUL (ASCII value 0) character is encountered

Example

The graphical user interface of Exivity is a purely client-side application, which means it runs inside your web browser. It communicates with the Exivity to obtain data records, report data and general configuration. This means all functionality available in the GUI can also be accessed programmatically.

Throughout this documentation, the graphical user interface will be referred to as the Glass Interface.

The interface allows you to do the following:

• Develop

The 'Rates' screen allows you to configure manual rates for services that do not have a rate provided with their data source. Services must have at least have a default Global Rate and may also have optional customer-specific rates.

Rate types can be configured as either manual or automatic. In both cases the rate is applied per unit of consumption.

Automatic services obtain the rate (and/or interval value) from a column you specify, whereas manual services allow a rate to be explicitly specified. If proration is enabled, the charge on the cost reports is reduced based on the number of days in the month that the service was used.

By default each service has a global rate configured. This will be applicable to all accounts that consume this service unless a customer-specific rate has been configured for that account, in which case it will take precedence over the global rate.

The generate_jwt statement is used to generate an -compliant JWT (JSON Web Token) which can be used, for example, for .

Syntax

generate_jwt keykey component1 [... componentN]as

The json statement is used to format JSON in a .

Syntax

json format{buffername}

Syntax

ipv4_to_decimalvariable_name

ipv4_to_decimalsource_variable_nameas

The pause statement is used to suspend execution of a USE script for a specified time.

Syntax

pausedelaytime

The gunzip statement is used to inflate a GZIP file

The save statement is used to write the contents of a to disk.

Syntax

save{buffer_name}asfilename

While executing a USE script, various messages are written to a logfile. The loglevel option determines the amount of detail recorded in that logfile.

Syntax

loglevelloglevel

The terminate statement will exit the USE script immediately.

Syntax

terminate [with error]

The return statement is used to exit a subroutine at an arbitrary point and return to the calling location.

Syntax

return

Exivity can be easily deployed on any system running Windows Server 2016 or later.

Single Node System Architecture

In order to deploy a single node Exivity system, be aware of the following system architecture:

All components can be installed on a single node. However, even in a single node deployment, it might be desirable to run the PostgreSQL database and RabbitMQ system on a different system. Apart from installing PostgreSQL from the Exivity installer, the Exivity software is also compatible with any PostreSQL compatible database engine (PostgreSQL on Linux, Amazon RDS, CockroachDB, etc) in order to achieve High Availability or Load Balancing. The same goes for the RabbitMQ component.

Deploying Exivity on a Single Node

After starting the Exivity installer and providing a valid license, ensure to have all components selected and click Next:

Then make sure to provide a valid path for the Exivity program files:

And select a folder for the Exivity home files:

Provide a custom administrator username and password, or leave the default:

Then provide the details for the PostgreSQL database. The installer will configure a local PostgreSQL database by default:

In case you prefer to use a remote PostgreSQL database system, ensure to deselect 'Install Local PSQL Engine' and provide your PSQL server credentials:

When you are finished configuring your PostgreSQL database settings, click the Next button to configure RabbitMQ. To use a remote RabbitMQ instance, deselect the 'Install Local RabbitMQ Engine' and provide the appropriate hostname, username, password, vhost and TCP port. In case you require TLS/SSL towards your RabbitMQ instance, select that checkbox as well:

Once the installation is finished, ensure to check 'Start the Exivity Windows Services' to start the Exivity services after clicking Finish.

Executing a Single Node Silent Installation

Follow the below steps to execute an unintended setup using the silent installer flags supported by the setup program.

1 - installing Exivity on a single node with a custom PostgreSQL data path

The below example will silently install Exivity with mostly default settings, except having the Exivity program, home, and PGDATA in separate locations:

2 - installing Exivity on a single node using a remote PostgreSQL database and RabbitMQ

The below example will silently install Exivity while using a remote PostgreSQL database:

Introduction

Apart from installing Exivity in any on-premises environment, Exivity can also be deployed from the Azure Market Place (AMP). Deploying Exivity on AMP is straightforward, and can be finished within a few minutes via your Azure Portal.

Azure Marketplace Offering

Login to your Azure Portal at and then go to the Marketplace to search for the Exivity offer:

Once you've selected the Exivity offering, you should be presented with the following screen:

After clicking the Create button, you will be redirected to the VM deployment wizard

Deployment Wizard

1. Fill in a Windows user/pass and pick your deployment Resource Group:

1. Try to pick a recommended VM size type that has enough CPU's and Memory (see for general system requirements). Smaller machines are possible, but will influence performance:

1. You may select any additional options, but none are required for running Exivity successfully, so you may skip this page simply by clicking the OK button:

1. Review the summary and click Create to deploy your Exivity VM:

This may take a few minutes. You may review the status of the Virtual Machine in your VM list:

Write down the Public IP address once it is available. Optionally you may configure a custom DNS name to have an easy way to connect.

Connecting to your Exivity instance

You can log in to your Exivity instance with RDP, but after deployment, you should be able to connect to your instance using the public IP address or DNS name of your Exivity instance considering the following default URL:

• https://<Your_Public_IP>:8001

The default admin username is admin with password exivity.

By default no data is loaded into the system, so you'll have to create a new for obtaining consumption data and a to process that data. A is then created to be able to report on your consumption metrics and costs.

Next steps

A couple of getting started guides are provided , but feel free to drop us an or create a in our support portal. We will then assist you to get you started for your specific use case.

Introduction

Exivity can be deployed from the AWS Market Place allowing you to have a functional Exivity solution in a matter of minutes. This tutorial will get you up and running.

AWS Marketplace Offering

Login to your AWS portal and access the Exivity offering .

1. Click on Continue to Subscribe.

2. Read our Terms and Conditions, and when ready, click on Continue to Configuration.

3. Select the Region where you want to deploy Exivity and click on Continue to Launch.

Deployment Wizard

1. In the first screen, try to pick a recommended VM size type that has enough CPU's and Memory (see for general system requirements). When you are done with your selection click on Next: Configure Instance Details.

1. In this section, you can select your VPC Configuration, or leave the default values. When you are done with your configuration click on Next: Add Storage.

2. In the Storage section, two drives are recommended, you can influence the Volume Type and the Size (GiB) parameters. When ready, click on Next: Add Tags.

3. In the Tags section, include the tags that are meaningful to you, as a minimum, a Name tag is recommended. Click on Next: Configure Security Group.

1. In the Security Group section, you can leave the default recommended security group or add more rules if needed. Click on Review and Launch.

2. Review the details and click on Launch, select your preferred Key Pair to connect to the instance.

In a few minutes, your instance will be deployed, you can track the progress in your EC2 Dashboard:

Write down the Public IP address / Public DNS and the Instance ID once they are available.

Connecting to your Exivity instance

You can logon to your Exivity instance with RDP, but after deployment, you should be able to connect to your instance using the public IP address or DNS name of your Exivity instance considering the following default URL:

• https://<Your_Public_DNS>:8001

The default admin username is admin with password Instance ID.

By default no data is loaded into the system, so you'll have to create a new for obtaining consumption data and a to process that data. A is then created to be able to report on your consumption metrics and costs.

Next Steps

A couple of getting started guides are provided , but feel free to drop us an or create a in our support portal. We will then assist you to get started for your specific use case.

Introduction

When deploying the vCloud Extractor template for Exivity, some input and configurations are required from your vCloud environment. The following process must be completed in order to report on vCloud consumption:

1. Create Exivity vCloud user (vCloud < 9.1)

2. Create Exivity vCloud user (vCloud >= 9.1)

3. Configure an Extractor

4. Configure a Transformer

5. Create your Report

Create Exivity vCloud user (vCloud < 9.1)

For environments with a vCloud previous to 9.1 Exivity needs a user with the sysadmin role on the system ORG. Please follow this procedure:

• Click the Administration tab and click Users in the left panel.

• Click New, fill in the required details.

• Note the username and password.

• Finally, click OK

Create Exivity vCloud user (vCloud >= 9.1)

For environments with vCloud version 9.1 or higher, you can create a user with more fine-grained permissions. Exivity needs a user with a reader role on the system ORG. Please follow this procedure:

First you will need to create a new custom role for the Exivity user:

• From the main menu, select Administration.

• In the left panel, under Access Control, click Roles

• Click New.

• Click Save.

Once you have the Role created, you need to set up a new user and assign the Role previously created:

• On the VMware Cloud Service toolbar, click the VMware Cloud Services icon and select Identity & Access Management.

• Click Add Users.

• On the Active Users tab, fill in the details of the user you want to add to the system organization.

Configure an Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For vCloud:

• Pick vCloud_Extractor_AdminVM from the list

• Provide a name for the Extractor in the name field

• Click the Create button.

Once you've created the Extractor, next go to the Variables tab:

Fill in all required variables with the values that you gathered in the previous step. You have the option to encrypt them. Click on Update.

Once you've filled in all details, go to the Run tab to execute the Extractor clicking on Run Now:

Configure a Transformer

Once you have successfully run your vCloud Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day (today) as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions.

Select your vCloud dataset, and your preferred Reporting Columns to break down the report (we recommend only Org_name and VDC for the default report). When you are ready, click on Create.

Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Introduction

When deploying the vCenter Extractor template for Exivity, it is required to configure a user with appropriate permissions. Additionally, the following process must be completed in order to report on vCenter consumption:

1. Create Exivity vCenter user

2. Configure an Extractor

3. Configure a Transformer

4. Create your Report

Exivity supports Out of the Box integration with vCenter version 6.5 and higher. For versions before 6.5, please contact [email protected].

Create Exivity vCenter user

Exivity needs a user with a reader role in order to retrieve consumption data. Please follow this procedure:

• In your vCenter Configuration Manager go to Configuration > Local Users and Groups > Users.

• Create a new User, insert username and password.

• Take note of the username and password, they will be used later on to configure Exivity

Configure an Extractor

To create the Extractor, browse to Data Sources > Extractors in the Exivity GUI and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. To create a new Extractor based on the vCenter template follow these steps:

• Provide a name for the Extractor in the Name field above

• Pick vCenter 6.5 (VM Inventory REST API) template from the list

• Click the Create button.

Once you've created the Extractor, go to the Variables tab:

Fill in all required variables with the values that you gathered in the previous step. You have the option to encrypt a variable in case it contains sensitive information (i.e. password) by clicking the lock icon on the right of each variable field. When finished, click Update.

Once you've filled in all details, go to the Run tab to execute the Extractor clicking on Run Now:

If the variables are correct and your vCenter is reachable for Exivity, you should get a successful result.

Configure a Transformer

Once you have successfully run your vCenter Extractor, you can create a Transformer template via Data Sources > Transformers in the Exivity GUI. Browse to this location and click the Create Transformer button. Make any changes that you feel necessary and then select the run tab to execute it for a single day (today) as a test.

Create a Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Reports > Definitions:

Select your vCenter dataset, and your preferred Reporting Columns to break down the report (we recommend only cluster_name for the default report). When you are ready, click on Create.

Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date range from the date selector shown when preparing the report.

Once this is done you should be able to run any of the Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

It is possible to define a Budget on any level within your organization. This enables different audiences to monitor costs across different clouds. When combining this feature with Notifications, a business owner can now set budget thresholds to inform customers, departments, or project owners when they are reaching their configured budget.

Create a budget

In order to create a budget, navigate to Accounts > Budgets. Then click the Create button to create a new budget. In this menu, a couple of items are presented:

Global options

Global options apply to the entire budget, and apply to the following items:

• Interval: determine whether the budget is applied Monthly, Quarterly or Yearly

• Apply to: a configured budget is by default applied to the total Charge of the configured Interval. It is however also possible to create a budget that is applied to the Cost of Cogs (COGS) instead

Revisions

A budget configuration can potentially change year over year, and therefore it is possible to create different budget revisions. Each revision can have the following settings applied:

• Revision start date: the start Month, Quarter or Year for this budget revision

• Filter by: typically a budget is applied to a single or multiple Accounts. It is however possible to add additional filtering on the Service or Service Category. By applying Service based filtering, it is possible to limit the scope for a configured budget

Accounts

When a budget is created, it is possible to set a budget money amount for one or multiple accounts. In case each account for which a budget is set also has 1 or multiple levels of child accounts, it is possible to control how the budget 'trickles down' the organizational structure:

• Account selection: it is required to select an Account from any level in your Report Definition. It is then possible to set a budget value (i.e. $100000) in the grey box next to the Account.

  • The same applies to any Child Accounts for which you may set/overwrite a budget. You may add a Child Account to the list, by clicking the green button left of the account name

To create the Budget, click the Create button. It is now possible to view the spendings under budget via the

Changing a budget

An existing budget can be changed by navigating to Accounts > Budgets and then clicking the budget which you want to change.

Changing an existing budget revision

Once a budget has been saved, you will be unable to change the start date unless you edit the Budget Revision:

Once you have enabled edit mode for an existing Budget Revision, you will be able to change the start date:

After making these changes you will need to save these by clicking the blue checkbox on the right. You can also cancel your change using the blue x-sign, or delete the revision by clicking the red recycle bin.

Adding a new budget revision

Similar to changing a Budget Revision, it is possible to add a new revision that holds a different start date with the revised budget plan. To do this, you will need to click the green + sign:

The encrypt statement is used to conceal the value of a variable, such that it does not appear in plain text in a USE script.

Syntax

encrypt varname = value_to_be_encrypted

Details

Encrypting one or more variables

Any variable prefixed with the word encrypt will be encrypted by the pre-processor and the script file itself will be modified as follows:

• All text (including trailing white-space) from the word following the = character up to the end of the line is encrypted

• The encrypted value is base64 encoded

• The original variable value in the USE script is substituted with the result

This process is repeated for all variables preceded by the encrypt keyword.

Using encrypted variables

Once encrypted a variable can be used just as any other, the only requirement being that the encrypted keyword preceding its declaration is not removed or modified.

To change the value of an encrypted variable simply replace the declaration altogether and precede the new declaration with encrypt. Upon first execution, the USE script will be updated with an encrypted version of the variable as described above.

Example

Firstly, create the script as usual, with encrypt preceding any variables that are to be encrypted:

Secondly, run the script. Prior to execution, the script will be automatically modified as shown below:

The hash statement is used to generate a base-16 or base-64 encoded hash of data stored in a variable or named buffer.

Syntax

hash sha256 [HMAC [b16|b64]key] target|{target}asresult[b16|b64]

hash md5target|{target}asresult[b16|b64]

Details

The hash statement uses the contents of target as its input and places the final result into result. The SHA256 and MD5 hash algorithms are supported.

If target is surrounded with curly braces like {this} then it is taken to be the name of a memory buffer and the contents of the buffer will be used as input. Otherwise, it is treated as the name of the variable, the value of which will be hashed.

By default, the resulting hash is base-16 encoded and the result placed into the variable specified by the result argument.

If the optional HMACkey arguments are provided when the hash type is sha256 then the secret in key will be used to generate an HMAC-SHA-256 result. The optional b64 or b16 argument following the HMAC option indicates that the key is base-64 or base-16 encoded. By default, a clear-text key is assumed.

If the optional b64 argument is used (base64 may also be specified) after the result variable, then the result will be base-64 encoded.

The optional b16 argument (base16 may also be used) after the result variable is provided for completeness, but need not be specified as this is the default encoding to use.

Example

Running the script:

results in the following output:

The loop statement executes one or more statements multiple times.

Syntax

Details

The loop statement will loop indefinitely unless one of three exit conditions cause it to stop. These are as follows:

1. The number of loops specified by the count parameter are completed

2. At least as many milliseconds as are specified by the timelimit parameter elapse

3. An statement explicitly exits the loop

In all three cases when the loop exits, execution of the script will continue from the first statement after the closing } marking the end of the loop.

In the event that both count and timelimit parameters are specified, the loop will exit as soon as one or other of the limits have been reached, whichever comes first.

Both the count and timeout parameters are optional. If omitted then the default for both of them will be infinite.

The loop statement will automatically create and update a variable called loop_label.COUNT which can be referenced to determine how many times the loop has executed (as shown in the example below). This variable is not deleted when the loop exits which means that it is possible to know how many times any given loop executed, even after the loop has exited.

Example

The loop shown above will result in the following output:

Overview

The lowercase statement is used to set all letters in a variable or named buffer to lower case.

Syntax

lowercasevar_name

lowercase{buf_name}

Details

The single parameter to the lowercase statement determines whether or not the text to be normalised to lower case is located in a variable value or in a named buffer.

If the parameter starts and ends with { and } respectively then the text to be processed is in the named buffer identified by the contents of the curly braces.

If not, then the text to be processed is the value of the variable named by the parameter.

Examples

The following script:

... produces the following output:

Overview

The uppercase statement is used to set all letters in a variable or named buffer to upper case.

Syntax

uppercasevar_name

uppercase{buf_name}

Details

The single parameter to the uppercase statement determines whether or not the text to be normalised to upper case is located in a variable value or in a named buffer.

If the parameter starts and ends with { and } respectively then the text to be processed is in the named buffer identified by the contents of the curly braces.

If not, then the text to be processed is the value of the variable named by the parameter.

Examples

The following script:

... produces the following output:

Server

Exivity can be installed on any Microsoft Windows 2016 or higher server in your on-premises data center or in the cloud. Depending on the amount of data, Exivity recommends the following system configuration:

Introduction

When deploying the Azure Stack for Exivity, some configuration is required within your Azure Stack environment and a lookup file needs to be created. The following process must be completed in order to report on Azure Stack consumption:

1. Create an Exivity Enterprise Application in your Azure AD for authentication

Introduction

When deploying the Azure EA for Exivity, some configuration is required within your Azure EA environment. The following process must be completed in order to report on Azure EA consumption:

1. Create an Access Key and Secret in your Azure EA portal

Exivity offers the possibility to . Users can get notified when a workflow has finished successfully or when it has failed. In some scenarios, you may want to get informed by a Monitoring System about any Failed Workflow alerts. This is possible through webhooks.

What are Webhooks?

Webhooks are automated messages sent from one app to another when something happens.

In simple words

The services screen gives a user the ability to view and change the available services in the service catalogue of the Exivity deployment. When creating new services, it is required to use a Transformer with the statement.

Obtaining details of a Service

To view the details of a service that has already been created, click on one of the services listed in the Services > Overview screen:

It is possible to add a new service or change an existing one using the following parameters:

Subscriptions enable users to manage one-off and recurring transactions for charging non-metered services.

• One-off Subscriptions may be used for managing server setup fees or applying a one-off correction.

• Recurring Subscriptions are typically used to charge a specific service for a certain quantity every month or every year.

Subscriptions are always applied to a leaf account on the deepest level of a definition.

The Data pipelines menu allows an admin of the Exivity solution to manage USE 'Extractors'. USE has its own language reference, which is fully covered in a separate chapter of this documentation.

As described in the , you are free to use your editor of choice to create and modify USE Extractors. However, the GUI also comes with a built-in USE Extractor-editor.

Creating Extractors

To create a new USE Extractor

The match statement is to used search either a specified string or the contents of a named using a regular expression.

Syntax

matchlabel expression target

The if statement is used to conditionally execute one or more statements. In conjunction with an optional else statement it can cause one or other of two blocks of statements to be executed depending on whether an expression is true or false.

Syntax

if(expression)

Prerequisites

This tutorial assumes that you have CUR (Cost and Usage Report) set up in your AWS environment. If this is not the case please follow the steps in Turning on the AWS Cost and Usage Report before proceeding.

Please note that we also need the following requisites:

• S3 bucket where the CUR reports are placed

• The AWS region of the S3 bucket

• The path where the reports are placed inside the S3 bucket

To get these requirements you can go to the Amazon S3 service - Buckets and note the name and the region of our CUR S3 bucket.

Once inside the CUR bucket, you must find the path where the monthly reports reside, please navigate the bucket until you find a folder like this one and note the path:

Obtaining the API Credentials

We have to create an IAM User that has read access to the CUR bucket. Please follow these steps:

• Go to IAM - Policies and click on Create policy.

• In the next screen: Service select S3, Actions select Read, Resources select Specific and in bucket click on Add ARN.

• In the next screen, type the name of your bucket followed by /* in Bucket name (example: billing-bucket/*) and click on Add.

• Click on Review Policy.

• Give a Name and a Description to the policy and finally click on Create policy.

Now, we will attach the policy to a user:

• Go to IAM – Users – Add user.

• Add a User name and select Access type: Programmatic access.

• Click on Attach existing Policies directly and filter by the name of the policy you just created.

• Select the policy and click on Next:Tags.

• Add a Key-Value pair for the Tags and click on Next:Review.

Configure Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For AWS, please click AWS_CUR_Extractor from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to the first tab: Variables

• Bucket: Type the name of your S3 CUR bucket.

• AWS region: The name of your S3 CUR bucket region code can be found in the (Region column).

• Access key: You can find it in the csv that you have downloaded in the previous section.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires one parameter in yyyMMdd format:

• from_date is the date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure Transformer

Once you have successfully run your AWS CUR Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button. Select the AWS CUR Transformer and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

Create Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Select the column(s) by which you would like to break down the costs. Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date selector shown when preparing the report.

Once this is done you should be able to run any of the Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

Prerequisites

This tutorial assumes that you have a Billing Account for your Google Cloud Projects in place. In the event this is not the case please follow the steps in this tutorial before proceeding.

Setting up BigQuery

Exivity will leverage the GCP BigQuery service. This will enable the export of detailed Google Cloud billing data to a BigQuery dataset, giving Exivity the capability to query the billing data in a more granular manner than with the standard file export (which is going to be by Google).

First and foremost, Exivity will create a dataset containing the billing table which will be periodically queried. Please follow these instructions:

1. Go to the service page.

2. Select the project that will contain your dataset in the project drop down.

3. Click CREATE DATASET.

Once you have the dataset created in BigQuery you need to enable Cloud Billing export to BigQuery:

1. Sign in to the , select Organization and main Billing Account.

2. Go to the Billing Export tab.

3. Click Edit Settings to enable the export.

Creating Exivity Service Account

Exivity requires a GCP Service Account with the BigQuery User role associated to retrieve the billing data. Please follow to create a Service Account, to associate a Private Key with the service account, and finally to associate the role with the service account.

Configuring the Extractor

To create the Extractor in Exivity, browse to Data Sources > Extractors and click the Create Extractor button. This will try to connect to the Exivity Github account to obtain a list of available templates. For Google Cloud, please click Google_Cloud from the list. Provide a name for the Extractor in the name field, and click the Create button.

Once you have created the Extractor, go to Variables tab and fill the parameters:

• Hostname: Default endpoint for Google BigQuery with version

• Private: Provide the RSA private key PEM format

• Email: Mail address/service account associated with the private key, obtained in the Creating Exivity Service Account section.

• Project: Main GCP Project name.

• BigQuery project: GCP BigQuery Billing Project, obtained in the Setting up BigQuery section.

Finally, click on Update.

Once you have filled in all details, go to the Run tab to execute the Extractor for a single day:

The Extractor requires two parameters in yyyMMdd format:

• from_date is the date for which you wish to start collecting consumption data.

• to_date is the end date for which you wish to collect consumption data.

When you click the Run Now button, you should get a successful result.

Configure a Transformer

Once you have successfully run your Google Cloud Extractor, you should be able to create a Transformer template via Data Sources > Transformers and click the Create Transformer button.

Select the Google Cloud Transformer that you just created, go to the Run Tab and run it for a single day as a test. Make sure that it is the same day as for which you extracted consumption data in the previous step.

When you click the Run Now button, you should get a successful result.

Create your Report

Once you have run both your Extractor and Transformer successfully create a Report Definition via the menu option Report Definition via the menu Reports > Definitions:

Select the column(s) by which you would like to break down the costs (you can start with only project_name as a test). Once you have created the report, you should then click the Prepare Report button after first making sure you have selected a valid date selector shown when preparing the report.

Once this is done you should be able to run any of the Accounts, Instances, Services or Invoices report types located under the Report menu for the date range you prepared the report for.

The buffer command is used to create and/or populate one of these named buffers with data.

Syntax

buffername=protocol protocol_parameter(s)

Details

The first argument to the buffer statement is the name of the buffer to create. If a buffer with this name already exists then any data it contains will be overwritten.

There must be a whitespace on both sides of the 'equals' symbol following the buffer name.

The following protocols are supported:

file

bufferbuffername= filefilename

The file protocol imports a file directly into a buffer. This can be very useful when developing USE scripts, as the USE script for processing for a JSON file (for example) can be implemented without requiring access to a server.

If the specified buffer name already exists, then a warning will be logged and any data in it will be cleared before importing the file.

data

bufferbuffername= datastring

The data protocol populates the buffer with the literal text specified in string. This is useful when extracting embedded JSON. For example, the JSON snippet below contains embedded JSON in the instanceData field:

In this case, the instanceData field can be extracted using a parslet, placed into a new buffer and re-parsed to extract the values within it. Assuming the snippet is in a file called my_data.json this would be done as follows:

http

bufferbuffername= httpmethod url

Once the HTTP request has been executed, any data it returned will be contained in the named buffer, even if the data is binary in format (eg: images, audio files or anything else non-human readable).

If the HTTP request returned no data, one of the following will apply:

• If the buffer does not already exist then the buffer will not be created

• If the buffer already exists then it will be deleted altogether

For details of how to access the data in a named buffer, please refer to the article.

odbc

bufferbuffername= odbcdsn [username password] query

where:

• dsn is the ODBC Data Source Name (this should be configured at the OS level)

• username and password are the credentials required by the DSN

• query is an SQL query

Once the query has been executed, the resulting data is located in the named buffer. It can subsequently be saved as a CSV file to disk using:

save {buffername} as filename.csv

The resulting CSV uses a comma (,) as the separator and double quotes (") as the quoting character. Any fields in the data which contain a comma will be quoted.

odbc_direct

bufferbuffername= odbc_directquery

where query is an SQL query.

Executes SQL query against ODBC datasource that is described in odbc_connect parameter.

Once the query has been executed, the resulting data is located in the named buffer. It can subsequently be saved as a CSV file to disk using:

save {buffername} as filename.csv

The resulting CSV uses a comma (,) as the separator and double quotes (") as the quoting character. Any fields in the data which contain a comma will be quoted.

Examples

The following examples retrieve data from ODBC and HTTP sources:

The foreach statement defines a block of zero or more statements and associates this block with multiple values. The block of statements is executed repeatedly, once for each value.

Syntax

foreachparsletasloop_label{

}

Details

The foreach statement is used to iterate over the values in an array or object (identified by a ) within the data in a .

The loop will execute for as many elements as there are in the array, or for as many members there are in the object. For the purposes of this documentation, the term child will be used to refer to a single array element or object member.

If the array or object is empty, then the body of the loop will be skipped and execution will continue at the statement following the closing }.

The loop_label can be any string, but must not be the same as any other loop_label values in the same scope (ie: when nesting foreach loops, each loop must have a unique label). This label is used to uniquely identify any given loop level when loops are nested.

The foreach statement will execute the statements in the body of the loop once for every child. foreach loops can be nested, and at each iteration the loop_label can be used to extract values from an array or object in the current child using a . See the examples at the end of this article for a sample implementation showing this in action.

As the foreach loop iterates over the children, a number of variables are automatically created or updated as follows:

Examples

Basic looping

Consider the following JSON in a file called samples/json/array.json:

To generate a list of IDs and names from the items array, the following would be used:

Nested looping

To extract values from an array using nested loops:

Given the source JSON in a file called example.json, the following USE script:

will produce the following output:

The days_in_month statement sets a variable to contain the number of days in the specified month.

Syntax

get_last_day_ofyyyyMMasvarName

Details

The get_last_day_of statement will set the value of the variable called varName to contain the number of days in the month specified by yyyyMM where yyyy is a four-digit year and MM is a 2-digit month.

The statement will take leap years into account.

Example

The subroutine keyword is used to define a named subroutine.

Syntax

Details

Overview

A subroutine is a named section of code that can be executed multiple times on demand from anywhere in the script. When called (via the statement), the execution of the script jumps to the start of the specified subroutine. When the end of the code in the subroutine body is reached or a statement is encountered (whichever comes first), execution resumes at the statement following the most recent statement that was executed.

The code in the body of a subroutine statement is never executed unless the subroutine is explicitly called using . If a subroutine is encountered during normal linear execution of the script then the code in it will be ignored.

Subroutine Arguments

When invoked via the statement, arguments can be passed to the subroutine. These arguments are read-only but may be copied to normal variables if required.

Arguments are accessed using the same syntax as is used for variables as follows:

${SUBARG.COUNT} contains the number of arguments that were passed to the subroutine

${SUBARG_N} is the value of any given argument, where N is the number of the argument starting at 1

Every time a subroutine is called, any number of arguments may be passed to it. These arguments are local to the subroutine and will be destroyed when the subroutine returns. However, copying an argument to a standard variable will preserve the original value as follows:

After the subroutine above has been executed the return_value variable will retain the value it was set to.

Example

The following demonstrates using a subroutine to detect when another subroutine has been provided with an incorrect number of arguments:

The http statement initiates an HTTP session using any settings previously configured using the set statement. It can also be used for querying response headers.