Releases

Upgrading

Upgrading to the latest release is a straight-forward process. More information.

When upgrading to new major version (e.g. from version 1.x.x to version 2.x.x), also take note of these manual upgrade steps:

• Upgrading to version 2

Changelog

v2.3.1

January 23, 2019

New features

• The 'export' statement will no longer generate an error if asked to process an empty DSET while option mode = strict is set

• Implemented user-definable timeout setting when retrieving data from HTTP sources

  When retrieving data from HTTP sources, the number of seconds to wait for a server response before timing out can be defined using set http_timeout.

v2.3.0

January 17, 2019

New features

• Added editable labels for report levels

• Added the ability to mass delete services

• Invoice report is now called Summary report

• The 'finish' statement in Transcript can be made to cause the task to fail if the DSET it's given is empty

  Previously, when creating an RDF, the finish statement would perform no action if the DSET to create the RDF from was empty. This is still the case if 'option mode = permissive' is in force, but if 'option mode = strict' then the finish statement will now generate an error. The error will cause the task to fail for the current day, and if 'option continue' is not enabled, the task will be terminated, else the task will move onto the next date in the range of days being processed.

• When importing multiple files using the pattern option to import, any invalid files will be skipped When multiple files matching a pattern are imported, any of those files that are malformed or otherwise non-importable will be skipped.

• Added the ability to exit a subscript invoked via #include in a transform script The 'return' statement, when used in a Transform script, will now cause script execution to resume from the statement following the #include statement in a parent script that referenced the script containing the 'return' statement.

• Increased the performance of the correlate statement Correlation should now be significantly faster than it was previously

• Increased the performance of the aggregate statement

  Aggregation should now be significantly faster than it was previously

Bug fixes

• Fixed a bug where users couldn't update their own details (including their password)

• Fixed a bug where only 10 datasets were displayed when creating a new report

• Fixed a bug where the mail sender name was not persisted in the configuration

• Fixed an issue when importing CSV files containing quotes

  When importing a CSV file with two successive quote characters at the end of a field, Transcript would reject the file as invalid. This has now been fixed.

• Fixed an issue where deleting data let to GUI crashes on occasion When deleting data (RDFs) associated with a report, it could be that if one or more days had previously been overwritten, a stale database entry would cause issues after the RDFs were deleted. This has now been fixed.

• Fixed a bug whereby using terminate within the body of an if statement in a Transform script could cause an error Invoking terminate in an if block when running a transform script against a range of dates could cause the error The maximum number of nested blocks (32) is already in use. This has now been fixed.

v2.2.1

December 13, 2018

Bug fixes

• A bug was fixed which could lead to an error in the invoice report when using a rate with a minimum commit set

• Fixed an issue with minimum commit It was possible that when applying minimum commit to a service, that other services would be affected by that minimum commit. This has now been fixed.

• Fixed an issue when retrieving NULL fields from an ODBC query When using ODBC to collect data, the presence of NULL values in the results could cause USE to crash. This has been fixed.

v2.2.0

November 30, 2018

New features

• Added the ability to view instance level details on the invoice reports.

• Added the ability to customize the report exports (CSV format only) field delimiter and decimal separator. These settings are system-wide and available to administrators by navigating to Administration > Configuration > Formatting.

• Added the ability for users to reset their own passwords. This requires the email address of users to be set and a working server configuration for sending emails. This can be configured in Administration > System > Environment.

• Logfiles generated from workflow tasks now include a timestamp. This prevents logfiles from consecutive runs of the same task from being overwritten.

• Workflow status now automatically refreshes after a manual run.

• Added a new Environment tab in Administration > System. In this tab, information about the system the Exivity instance is running on can be filled out. In the future this will be expanded to include more configuration options.

• Invoice reports now include minimum commit uplifts as separate entries.

• Carriage-returns and line-feeds in data extracted using ODBC are now replaced with spaces. When extracting data with USE, the presence of newlines in the data could cause corrupt CSV output. Carriage Return and Line Feed characters in data extracted from ODBC are therefore now replaced with spaces.

• Enhanced expression support in the Extractor component. Conditional expressions have been enhanced in the Extractor component such that more complex conditions can be evaluated and additional operations can be performed. Additionally, it is now possible to set a variable value using an expression.

• Services can now be manually deleted using the GUI.

Bug fixes

• The datasets selector visible when creating a new report definition is now alphabetically sorted.

• Fixed a bug which caused the contents of extractor editor to not update after updating variables. The contents of the extractor script itself was always saved after updating variables, only those changes were not visible in the editor.

• Fixed a bug which caused the account depth selector to reset after performing an upgrade.

• Fixed a bug which could cause the interface to become unresponsive after preparing a report.

• Fixed an issue when running Transform scripts for days with 25 hours in them. When running a Transform script with a data-date representing a day where the clocks were adjusted such that the day had 25 hours in it, the script would be executed a second time automatically once the first had completed. This could lead to unexpected errors and log entries on occasion, and has now been fixed.

• When writing to CSV files in USE, embedded CR/LF characters are converted to spaces. USE will now automatically strip out embedded carriage-return and line-feed characters when writing data to CSV files. Each unique occurrence of one or more sequential CR/LF characters will be replaced with a single space.

• Fixed a bug whereby the body of an 'if' statement in the Transformer could terminate prematurely. In some cases, using an 'import' statement with an 'options' block within the body of an 'if' statement could cause statements following the 'import' to be skipped. This has now been fixed.

v2.1.5

November 22, 2018

Bug fixes

• Updated the documentation links in the header to point to our new documentation site.

• Fixed grouping behaviour in the details table of the accounts report. In some cases, accounts could appear grouped under the wrong parent account in the 'Detailed' table in the accounts report.

v2.1.4

October 31, 2018

Bug fixes

• Fixed an issue with incorrect quantities sometimes showing on reports. Occasionally, when running a report for a range of dates, the quantities on one or more services differed from the quantity for that service shown when a report was run for a different date range (or just the day in question). This issue has now been fixed.

v2.1.3

October 26, 2018

New features

• The USE 'basename' statement can now write its results to a new variable. Previously, the 'basename' statement would always modify the value of the variable whose name was supplied as the argument. It can now also accept a literal string and create a new, or update an existing, variable to hold the result.

• Archives in GZIP format can now be decompressed using USE. USE now supports the 'gunzip' statement which can be used to inflate GZIP'd data. Details of how to use this statement may be found at https://docs.exivity.com/diving-deeper/extract/language/gunzip

• FIxed an issue whereby when running a Transform script the Audit database would be locked for the duration of the task. Transcript now only opens the Audit database when it needs to, reducing the likelihood of encountering errors pertaining to a locked audit database in the logfile.

• A new system variable is now available containing the number of days in the current month. The existing dataMonth variable, which contains the yyyyMM of the current month is now supplemented with a new variable called dataMonthDays which contains the number of days in that month.

• Changed default service type to 'automatic' in the 'services' statement in Transcript. When creating services, if no 'type' parameter is provided then the default service type will now be be set to 'automatic'.

Bug fixes

• Fixed an issue whereby when creating a service, the audit indicated that the service creation failed. When a service definition is successfully created, Transcript will now correctly audit that event as opposed to indicating that the attempt failed.

• Fixed an issue whereby over-writing services could result in database errors in the logfile. Sometimes when overwriting services, a constraint error would be logged in the logfile and the service would not have any rate associated with it. This has been fixed.

v2.1.2

October 18, 2018

Bug fixes

• Fixed an issue that could cause database corruption.

  Fixed an issue that could cause database corruption due to the Aeon database being held open for long periods of time.

v2.1.1

October 10, 2018

New features

• Added a live preview feature when working with transforms. A new feature has been added which can display a live preview of the transformer output. Note: this feature is currently in beta and will be further updated in the next release.

• The code editor has been updated. The code editor for Extractor and Transformer scripts has been updated (it now uses the open source Monaco editor - https://microsoft.github.io/monaco-editor/) resulting in a significant improvement over our previous editor. This greatly enhances the user experience when editing scripts in the GUI. Note: This change also lays the foundation for more advanced features going forwards.

• Charges for monthly services now take quantity into consideration as well as price. If two or more days in a month have the same highest price then the one with the highest quantity will be reported. Previously, the first seen was reported which could lead to discrepancies between the reported quantity and price on the report.

• When running reports blank instance values are now displayed as a hyphen. When running reports against data with blank instance values in the usage data, the instance value will now be represented as a hyphen, which improves the aesthetics of the report.

• Added hardware information to Transcript log-files. Log-files created by Transcript now contain information about the CPU and RAM at the top of the log.

• Increased auditing information in Transcript. Events relating to service, rate and RDF changes are now audited

Bug fixes

• Removed COGS option for users without rights to view COGS information. In the services and instances report, users with no access to view COGS will no longer be able to select the COGS type in the details table. Note: This bug never allowed users without appropriate access rights to view the actual COGS data.

• Fixed a bug where the list of datasets on the report definition page was only showing the first 10 results. This could result in an inability to create new reports using datasets that were not included in those results

• A link to the instances report has been added to the search feature in the header.

• The service interval column in the instances report now contains data. Previously this column was always blank

• Fixed a bug where searching for units within a services reports leads to a GUI crash.

• Fixed an issue whereby very rarely a charge would not be included in reports. On very rare occasions, information in a record in the prepared report caches was not included in the output when a report was run. This has now been fixed.

• Fixed an issue that could cause Aeon database corruption. Fixed an issue that could cause database corruption (and workflows to fail) due to the Aeon database being held open for long periods of time.

• Fixed an issue whereby re-using an existing named buffer in USE for ODBC purposes could lead to unexpected results. Fixed an issue in USE whereby if an existing named buffer was re-used to store data retrieved from ODBC then a new buffer could have been created with the same name as the existing buffer, and attempts to reference it would return the old data.

• Fixed an issue when executing ODBC queries that return no data. Using the ODBC capability to execute a query that returns no data will no longer cause an extractor to return an error.

v2.0.6

September 05, 2018

New features

• Upgraded the underlying API framework For more information, please refer to the Laravel release notes.

Bug fixes

• Fixed an issue whereby 'append' could crash if one or other DSET was empty When executing the 'append' statement in a transformation script, if one or other of the DSETs involved in the operation was empty (having no data rows) then a crash could occur. This has now been fixed.

• Fixed an issue where an expression that evaluated as FALSE could show the wrong line number in a log message The DEBUG level logfile entry indicating an expression is true or false would contain a reference to the wrong line number if the expression evaluated to false. This has now been fixed.

• Fixed an issue whereby some comparisons would evaluate incorrectly in expressions Fixed and issue whereby in some cases where a value was quoted in an expression, the quotes would be considered part of the value itself.

• Fixed a condition where reports were not showing for non-admins

• Quantity metric is available again for the timeline chart on the services and instances reports

• Reports in the navigation menu dropdown are now alphabetically ordered

v2.0.5

August 28, 2018

New features

• Ability to filter data in report using search query: The search bar in Accounts, Services and Instances reports now supports the use of operators (for example >and <) to filter your results based on column values or strings.

• Add avg_unit_based_rate to report/run API endpoint Added the average per unit rate field to the report/run API endpoint and a placeholder for the average per interval rate which will be implemented later.

Bug fixes

• Fixed an issue where deleting services could lead to adjustments not displaying correctly

• Rate column in report details tables now use the configured rate precision setting

• Fixed an issue whereby scheduled tasks that output more than 4kb of data to the console could suspend execution and do nothing until they timed out

v2.0.4

August 22, 2018

New features

• Transcript can now normalise scientific decimal numbers to standard format: When processing data that contains numbers in scientific format (such as 2.1E-5) the normalise statement can now be used to convert these to standard decimal notation (0.000021 in the above case) using the form normalise columncolNameas standardwhere colNameis the column containing the values to convert. Any values already in decimal will be unchanged, except that any trailing zeros will be removed from them. Non-numeric values will be converted to 0.

• Support group and group_col as service parameters in Transcript: In the serviceand servicesstatements in Transcript, the parameters to define the service category are category and category_colThese parameters now have aliases of groupand group_col respectively, for those who prefer to use that terminology.

Bug fixes

• The replace statement in Transcript will no longer behave unexpectedly when given an empty string as the target to replace: When using replace to update substrings within the values in a column, if the target string (the text to replace) is empty then Transcript will generate a meaningful log entry explaining that it cannot be used to replace empty strings, and will no longer overwrite non-blank column values with multiple copies of the replacement text.

• The export statement in Transcript now supports backslashes as path delimiters: When specifying a relative path for the exportstatement, Transcript will automatically create any directories that do not exist in that path. Previously there was a bug whereby the auto-creation of those directories would only work if UNIX-style forward slashes were used as delimiters in the path. This has now been fixed and Windows or UNIX style delimiters may be used when specifying an export path.

• Fixed a bug in the scheduler that could cause schedules to fail: In some cases schedules could fail for no obvious reason. This has now been fixed.

v2.0.3

August 17, 2018

New features

• USE scripts can now be forced to terminate with an error result Previously, the 'terminate' statement could be used to cancel script execution, but its use would always indicate that the script ran successfully. This may not be appropriate in all cases (for example if an error is detected by the script itself but ultimately cannot be resolved satisfactorily). The 'terminate' statement will still cause a script to exit with a success result by default, but may now be invoked as 'terminate with error' such that an error status is returned instead.

• Added more service attributes as optional columns in the reports details table. The following extra service attributes can now be enabled as columns in the report details table: interval, charge type, cogs type and proration.

• Support 'group' and 'groupcol' as service parameters in Transcript In the 'service' and 'services' statements in Transcript, the parameters to define the service category are 'category' and 'category_col'. These parameters now have aliases of 'group' and 'group_col' respectively, for those who prefer to use that terminology.

• Reduced the chance of a 'database is locked' warning when preparing reports When preparing reports, on occasion it is possible for a warning to appear in the logfile pertaining to the global database being locked. When this warning happened, it could cause some days in the reporting period to remain unprepared. A known specific cause of this issue has been fixed, significantly reducing the likelihood of it happening.

Bug fixes

• Fixed an issue where an ODBC connection could cause a crash in USE When executing an ODBC-based collection in USE, under certain circumstances an incorrect direct connection string could cause a crash. This has been fixed. Additionally, when an ODBC error occurs the error written to the logfile contains more detail than in previous releases.

• The order of workflow steps in the status tab now corresponds to the order of workflow steps in the configuration tab.

• An issue has been fixed where old user preferences could conflict by updates in the GUI, leading to errors when loading the service and instance reports.

• An issue has been fixed where certain characters in a workflow status could lead to errors in the API. Sometimes, when running a scheduled task, the output written to the database contains non-printable characters. The API now re-encodes those characters, which means the GUI will now correctly show the status for those workflows.

• When selecting a reporting period that spans multiple months, the charts will now only show a single label for each month.

• Fixed a USE crash bug with certain combinations of conditional expressions Fixed an issue whereby if an expression with more than 2 parameters was followed later in the script by an expression with fewer parameters than the first, a crash would occur.

• Fixed issue where an extractor could crash when using a parslet after formatting some JSON A bug has been fixed whereby if the 'json format' statement was used to prettify some JSON in a named buffer, use of a parslet to extract data from the JSON could cause a crash.

• Fixed an issue where sometimes an XML parslet would cause an 'out of memory' error in USE When using an XML parslet, it was possible that an 'out of memory' error would be returned in the logfile and the script would fail, even on small input files. This has now been fixed.

v2.0.2

August 03, 2018

Bug fixes

• The 'export' statement in Transcript now supports backslashes as path delimiters

  When specifying a relative path for the 'export' statement, Transcript will automatically create any directories that do not exist in that path. Previously there was a bug whereby the auto-creation of those directories would only work if UNIX-style forward slashes were used as delimiters in the path. This has now been fixed and Windows or UNIX style delimiters may be used when specifying an export path.

• The 'replace' statement in Transcript will no longer behave unexpectedly when given an empty string as the target to replace

  When using 'replace' to update substrings within the values in a column, if the target string (the characters to replace) is empty then Transcript will generate a meaningful log entry explaining that it cannot be used to replace empty strings, and will no longer overwrite non-blank column values with multiple copies of the replacement text.

v2.0.1

July 25, 2018

New features

• Increased default timeout when retrieving data from HTTP servers

  Currently a USE script will fail if more than 3 minutes elapse without response when downloading data from an HTTP server. This has been increased to 5 minutes to cater for slow APIs.

v2.0.0

July 19, 2018

New features

• Transcript can now normalise scientific decimal numbers to standard format When processing data that contains numbers in scientific format (such as 2.1E-5) the 'normalise' statement can now be used to convert these to standard decimal notation (0.000021 in the above case) using the form 'normalise column colName as standard' where 'colName' is the column containing the values to convert. Any values already in decimal will be unchanged, except that any trailing zeros will be removed from them. Non-numeric values will be converted to 0.

• When accessing the GUI via http visitors will be redirected to https automatically

• Progress indicator in the report/run endpoint can be disabled To disable, set the progress parameter to 0. More information at our API documentation.

• Filter selectors show which items are present in the current report The service category selector in the services and instances report, and service selector in the instances report will show items not available in the current report grayed out.

• On-demand workflow execution Workflows can now be executed on demand. Also the schedule for a Workflow can be disabled.

• Single workflows can now have multiple schedules

• Workflows can now be scheduled in a specific timezone

• Added the average rate column to the reports details table

• Added the ability to show various totals in reports summary widget Summary widget now has the option to show all totals (previous behaviour) or only the totals for the current search results, or for the current pinned items.

• Added report shortcuts to the dashboard

• COGS, fixed COGS and fixed prices are now evaluated per instance when preparing reports Previously, if a service was created that used any of fixed_price_col, cogs_col or fixed_cogs_col to indicate that the rate in question should be obtained from the usage data for any given day, then the charge engine would use a single value from the specified column(s) and apply that to all instances of the services for the day. Now, each row of usage is individually consulted when preparing reports such that the specific value on that row is used (as is already the case when using 'rate_col' for pass-through rates)

• When extracting XML or JSON values, parse errors no longer cause the USE script to terminate Previously, when using a static parslet to extract XML or JSON values from the contents of a named buffer, if the buffer contained invalid JSON or XML then the USE script failed with an error in the log saying that the contents of the buffer could not be parsed. Now, if a named buffer contains data that is not valid JSON or XML, any attempt to extract a value from it using a static parslet will be expanded to the value EXIVITY_INVALID_XML or EXIVITY_INVALID_JSON.

• Improved performance and lowered memory requirements when running a report Previously, in some circumstances running a report could take longer than expected and consume large amounts of memory in the process. The performance and memory use of the report engine have both been improved.

• Reduced memory and increased performance when preparing reports Previously it was possible for some installations to use large amounts of memory and exhibit unreasonably slow performance when preparing reports. Preparing reports is not intended to be a realtime feature and will always incur some time overhead, but this time should now be significantly reduced in many cases, and the memory required to complete the process will be much less.

• New output format for the /report/run endpoint in the API Due to changes to the charge engine, the output format of the /report/run endpoint in the API has changed. An up-to-date overview of the attributes returned by this endpoint can be found at our API documentation.

• Free formatted ODBC connect strings are now supported in USE This exposes all ODBC driver options to the user, and avoid the requirement of creating manually DSN at the operating system level.

• The 'split' statement now supports discarding unwanted result columns When using the 'split' statement it is now possible to discard all but a selected range of the resulting new columns.

Bug fixes

• Fixed a Transcript crash when deleting a DSET Transcript will no longer crash in certain circumstances when deleting a DSET using the 'delete dset' statement.

• The Transcript 'export' statement now creates a path automatically When exporting data from Transcript, if a relative path is specified as part of the export filename, Transcript will automatically create the path if it does not exist. The path will be created relative to /exported

• Changed the behaviour of some columns in the report tables The optional _per unit charges and per interval charges columns on the report pages represent a fraction of the total charge and as such should be considered a subtotal rather than a rate._

• Allow users to see anonymous roll-up accounts even if they have no access When a user only has access to some children of a parent account, reports will now show the combined usage of those accounts grouped as an unknown account in the reports.

• Fixed a rare bug where incorrect character encoding in the data source could lead to reports not loading

• Currency symbol is no longer shown for quantity graphs

• An issue has been fixed which could lead to empty reports when there actually was report data In some cases, selecting certain combination of filters could lead to reports showing No data while there actually was report data for the current set of filters. This behaviour was observed mainly on the instances report page.

• Usernames are now allowed to contain special characters As a side effect of changing usernames to be case-insensitive, using special characters was no longer permitted since v1.8.1. This restriction is now removed.

• Changed the behaviour of clearing the charge engine caches Clearing the charge engine (Edify) caches unprepares all reports. The button on the About page now reflects this.

• It is now possible to use decimal values for adjustment amounts Previously this was only possible through the API. The GUI has been updated to also support this.

• Changing the date in the invoice report no longer resets the account selection Previously, when changing the date range on the invoice report screen, the current account selection (dropdown inside the invoice page) would automatically select the first account in the list. This has now been fixed to remember the selection when changing the date.

• Exivity now works correctly when installed in a directory containing spaces

• Transcript variables were not properly expanded when using in an import filter

• Export of consolidated invoice now contains data for all accounts Previously, selecting the CSV or Excel export of a consolidated invoice would only export data for the first account on the invoice.

• Fixed crash bug in the 'services' statement When creating services, Transcript will no longer crash if a blank interval or model value is encountered while building the service definitions.

Older release notes can be found here.