Upgrading to version 3

Please ensure to read and understand all subjects that are mentioned here. Implement the suggested changes (where applicable) before upgrading to version 3 to avoid unexpected behavior.

Upgrading to a v3.x.x will require upgrading to version v2.10.x first. The installer will verify this and displays a warning when this requirement is not satisfied.

PostgreSQL

The biggest single change is the use of a new database engine powering all application state, audit logs and processed report data. Upgrading to this new database engine is transparent and the installer will take care of installing the database server as part of the regular installation process. After or during upgrade, it is possible to leverage an external database server.

Windows Services

In version 2.x.x there were only two Exivity Services installed:

Windows Services with Exivity version 2.x.x

With Exivity version 3.0.0 up to version 3.4.3, assuming all components are installed on a single host system, there was a total of 4 different services:

Windows Services with Exivity version 3.0.0 and =< 3.4.3

As of version 3.5.0, the following Exivity services are to be considered when all components are deployed on a single host, there will be 7 different services:

Windows Services with Exivity version 3.5.0 and higher

In case you were using a service account in Exivity version 2.x.x for the Exivity Scheduling Service and the Exivity Web Service, you will have to reconfigure this service account for both services, as well as the Exivity API Service. In most cases, the Exivity Database Service may continue to run un the Local System account.

In case your current Exivity version 2.x.x installation runs inside an Active/Passive Windows Cluster, you will need to re-register the Cluster Roles for the Exivity Scheduling Service. Additionally, a new Cluster Role should be created for the Exivity Database Service, in case you decide to not use an external database host.

Default TCP ports

In v2.x.x, the default port for the Exivity GUI was 8001 and 8002 for the Proximity API. Both services were already available through port 443 (the default port used for https traffic, which means clients don't have to explicitly specify the port) and in v3.x.x this will be used by default:

Default using port 443 for both UI and API

This is achieved by shipping a web proxy configuration for nginx, which routes all requests starting with /v1/ to port 8002 and all other requests to port 8001. The recommended configuration is to not expose port 8001 to the public and only accept incoming traffic on port 443. Port 8002 may still be opened to external hosts, typically in a configuration where the Web and API/Backend components are deployed seperate hosts. In such a scenario it is advisable to allow communication from the Web node towards the API/Backend node on port 8002.

Default Security Settings

As of Exivity version 3 more strict security settings are applied by default. These can be found under Administration > Settings. One important item which should be considered when upgrading a multi node environment, is the use of CORS. It is required to list all possible front end UI nodes in the CORS origins field:

Administering security settings

Multiple hosts including https:// may be added, while seperating each URL using a , (comma) symbol. Wildcards may also be used as part of the hostname to match multiple URL's in one go such as: https://*.cors.exivity.io. An overview of all current security policies can be found here.

Transformer changes

@FILE_EXISTS and @FILE_EMPTY

The @FILE_EXISTS and @FILE_EMPTY functions in Transcript have been modified in a manner which may require changes to scripts that use them.

Previously, it would only check for files in the system and exported folders within the Exivity home directory, and if a specified path + filename did not start with system/ or exported/ then these would be prepended automatically before the check as done.

This behaviour has been changed in the following ways in v3.x.x:

• Any path + filename within the Exivity home directory can now be checked

• path + filenames are now accepted and treated as being relative to the Exivity home directory

• The folders system/ and exported/ are no longer automatically prepended

Consider a file somefile.csv in the %EXIVITY_HOME_PATH%/exported folder. Previously with version 2.x.x a user could check for the existence of this file using the following example:

v2: FILE_EXISTS statement auto-prepends exported / system

In version 3 it is required to include the entire path relative to the %EXIVITY_HOME_PATH% :

v3: Path must be relative to %EXIVITY_HOME_PATH%

This change may require modifications to existing Transformer scripts. This is because system and exported will no longer be automatically prepended.

Extractor changes

HTTP server certificate validation

The default behaviour of the HTTP subsystem in USE was changed to fully validate server SSL certificates, which may cause some USE scripts to fail. This typically applies to data Extractors, which are connecting to on premise data sources that use self signed SSL certificated. In version 2.x.x, the default behavior was to ignore these certificate errors. You can identify these errors in your Extractor logs:

Identifying Certificate Errors in the Extractor logs

It is highly recommended to use valid SSL certificates, properly signed by a trusted CA (Certificate Authority). However it is still possible to switch certificate validation off by specifying set http_secure no in an Extractor script before executing an HTTP request.

The above will apply to all of your Data Extractors where you are connecting to (most likely internal) data sources which are using self signed certificates. Make sure to apply this change before upgrading to v3 to avoid any data extraction errors.

Extractor & Transformer Schedules

In version 2.x.x it was still possible to schedule an Extractor or Transformer from the editor screen. This feature is removed from version 3.x.x. In case you still have schedules which are configured thru this interface, you should Unschedule these and create an approperiate Workflow instead.

Example v2.x.x Schedule which will be removed in version 3.x.x

GUI changes

There are some minor changes in the GUI which are not backwards-compatible:

• We removed the Excel export options from the reports. They were using the CSV format under the hood (i.e. they never actually produced valid Excel worksheets). In the future, we plan to implement proper Excel export formats for the reports, including a full summary report Excel export.

• Removed functionality which would take a custom API port from the #port=xxx location hash parameter on the login screen. Specifying a custom API port (and hostname) is still possible by setting up a config.json file on your system.

Changes to data processing for reports

Due to changes to the processing of reports, when making changes to either services, rates or adjustments, associated reports should be prepared. In v2.x.x this was already required when making changes to services or rates. Since v3.x.x this will be also required when making changes to adjustments. We've planned further improvements making this more transparent (i.e. handling preparation of reports automatically in the background).

API changes

Normalised date/time data in responses

Some endpoints were returning dates and timestamps in different formats. This has been normalised in such a way that all responses use the same serialisation for dates and timestamps:

• A date is always represented as ISO-8601 string: "yyyy-mm-dd", e.g. "2020-01-29"

• A date/time is always represented as ISO-8601 string in UTC: "yyyy-mm-ddThh:mm:ssZ", e.g. "2020-01-29T11:26:52Z". Note that the Z suffix denotes the UTC time standard.

This affects the responses (attributes) for the following group of API endpoints:

• /v1/audit

  • created_at

• /v1/budgetrevisions

  • effective_from

• /v1/dsets

  • earliest_rdf

  • latest_rdf

  • rdf_detail.created

  • rdf_detail.updated

• /v1/extractors

  • last_modified

• /v1/log

  • created

  • lines.date

• /v1/reports

  • data_status.first_date

  • data_status.last_date

  • data_status.status.date

• /v1/services

  • created_at

  • updated_at

• /v1/transformers

  • last_modified

• /v1/workflows

  • created_at

  • updated_at

• /v1/workflowschedules

  • start_time

  • next_run

Changed functionality

/v1/reports/{id}/run endpoint

• The JSON format in the response is simplified. See examples below.

This is an example response from v2.x.x:

{
   "report": [
      {
         "meta": {
            // ...
         },
         "error": false,
         "data": [
            {
               // ...
            }
         ]
      }
   ]
}

This is an example response from v3.x.x:

{
   "report": [
      {
         // ...
      }
   ]
}

Remove deprecated functionality

/v1/usergroups endpoint

Removed deprecated permission aliases.

• upload_files (use manage_files instead)

• manage_configuration (use manage_settings instead)

• manage_system (use manage_settings instead)

/v1/reports/{id}/run endpoint

Functionality deprecated in v2.x.x has been removed. If you were relying on any of the following functionality, please use the suggested replacement instead:

• The pdf/invoice option for the format query parameter has been removed. Please use pdf/summary instead.

• The invoice_options query parameter has been removed in favour of summary_options.

/v1/configuration endpoint

• Configuration keys prefixed with INVOICE_ are replaced by respective keys prefixed with SUMMARY_.

/v1/workflowsteplogs endpoint

Also relevant to other endpoints including workflowsteplogs .

• Removed the last_log attribute. Include the last log by specifying the query parameters include=steplogs&related[steplogs][limit]=1&related[steplogs][sort]=-start_ts.

• Removed the timestamp attribute. Use start_timestamp instead.

• Removed the message attribute. Use a combination of status and output instead.

/v1/file endpoint

Only applicable to POST requests to this endpoint.

• The filename in the response from this endpoint will no longer include the import/ prefix to better align for other requests in this endpoint. See example below.

{
-    "filename": "/import/generic/2020/01/31_uploaded_001.txt"
+    "filename": "/generic/2020/01/31_uploaded_001.txt"
}

/v1/usergroups endpoint

The following permissions has been removed in favour of their new counterparts:

• UPLOAD_FILES has become MANAGE_FILES

• MANAGE_CONFIGURATION has become MANAGE_SETTINGS

• MANAGE_SYSTEM has become MANAGE_SETTINGS