# Upgrading to version 3 {% hint style="warning" %} Upgrading to a v3.x.x will require upgrading to version v2.10.x first. The installer will verify this and displays a warning when this requirement is not satisfied. {% endhint %} ## PostgreSQL The biggest single change is the use of a new database engine powering all application states, audit logs and processed report data. Upgrading to this new database engine is transparent and the installer will take care of installing the database server as part of the regular installation process. After or during the upgrade, it is possible to leverage an [external database server](/getting-started/installation.md#database-server). ## Windows Services In version 2.x.x there were only two Exivity Services installed: ![Windows Services with Exivity version 2.x.x](/files/-M4O75g3MhYGY8spCl-X) With Exivity version 3.0.0 up to version 3.4.3, assuming all components are installed on a single host system, there was a total of 4 different services: ![Windows Services with Exivity version 3.0.0 and =< 3.4.3 ](/files/-M4O7s2pbzSTHht7csW4) As of version 3.5.0, the following Exivity services are to be considered when all components are deployed on a single host, there will be 7 different services: ![Windows Services with Exivity version 3.5.0 and higher](/files/-MjdkueNnn1oDmzrTbk3) In case you were using a service account in Exivity version 2.x.x for the **Exivity Scheduling Service** and the **Exivity Web Service**, you will have to reconfigure this service account for both services, as well as the **Exivity API Service**. In most cases, the **Exivity Database Service** may continue to run un the *Local System* account. {% hint style="warning" %} In case your current Exivity version 2.x.x installation runs inside an Active/Passive **Windows Cluster**, you will need to re-register the Cluster Roles for the **Exivity Scheduling Service**. Additionally, a new Cluster Role should be created for the **Exivity Database Service**, in case you decide to not use an external database host. {% endhint %} ## Default TCP ports In v2.x.x, the default port for the Exivity GUI was 8001 and 8002 for the Proximity API. Both services were already available through port 443 (the default port used for HTTPS traffic, which means clients don't have to explicitly specify the port) and in v3.x.x this will be used by default: ![Default using port 443 for both UI and API](/files/-M4sjGYSoIxOAsSSwpy1) This is achieved by shipping a web proxy configuration for Nginx, which routes all requests starting with `/v1/` to port 8002 and all other requests to port 8001. The recommended configuration is to not expose port 8001 to the public and only accept incoming traffic on port 443. Port 8002 may still be opened to external hosts, typically in a configuration where the Web and API/Backend components are deployed [as separate hosts](/getting-started/installation/on-premises/multi-node.md#multi-node-system-architecture). In such a scenario it is advisable to allow communication from the [Web node towards the API/Backend node on port 8002](/getting-started/installation/on-premises.md#configuring-a-separate-web-server-portal). ## Default Security Settings As of Exivity version 3, more strict security settings are applied by default. These can be found under *Administration > Settings.* One important item which should be considered when upgrading a multi-node environment is the use of CORS. It is required to list all possible front end UI nodes in the **CORS origins** field: ![Administering security settings](/files/-M4sguC47t_Z4Oig8xzC) Multiple hosts including **`https://`** may be added while separating each URL using a `,` (comma) symbol. Wildcards may also be used as part of the hostname to match multiple URLs in one go such as: `https://*.cors.exivity.io`. An overview of all current security policies can be found [here](/security/security.md). ## Transformer changes #### @FILE\_EXISTS and @FILE\_EMPTY The [@FILE\_EXISTS](https://docs.exivity.com/data-pipelines/transform/language/if#file_exists) and [@FILE\_EMPTY](https://docs.exivity.com/data-pipelines/transform/language/if#file_empty) functions in Transcript have been modified in a manner that may require changes to scripts that use them. Previously, it would only check for files in the `system` and `exported` folders within the Exivity home directory, and if a specified path + filename did not start with `system/` or `exported/` then these would be prepended automatically before the check was done. This behaviour has been changed in the following ways in v3.x.x: * Any path + filename within the Exivity home directory can now be checked * path + filenames are now accepted and treated as being relative to the Exivity home directory * The folders `system/` and `exported/` are no longer automatically prepended Consider a file `somefile.csv` in the `%EXIVITY_HOME_PATH%/exported` folder. Previously with version 2.x.x a user could check for the existence of this file using the following example: ![v2: FILE\_EXISTS statement auto-prepends exported / system](/files/-M56_VVWYSGB-4z8i51s) In version 3 it is required to include the entire path relative to the `%EXIVITY_HOME_PATH%` : ![v3: Path must be relative to %EXIVITY\_HOME\_PATH%](/files/-M56_hr19XXFCsHhDKzx) {% hint style="warning" %} This change may **require modifications** to existing **Transformer** scripts. This is because the `system` and `exported` will no longer be automatically prepended. {% endhint %} ## Extractor changes #### HTTP server certificate validation The default behaviour of the HTTP subsystem in USE was changed to fully validate server SSL certificates, which may cause some USE scripts to fail. This typically applies to data Extractors, which are connecting to on-premises data sources that use self-signed SSL certificates. In version 2.x.x, the default behavior was to ignore these certificate errors. You can identify these errors in your Extractor logs: ![Identifying Certificate Errors in the Extractor logs](/files/-M4O3upj9ZQrIGKWcqlJ) It is highly recommended to use valid SSL certificates, properly signed by a trusted CA (Certificate Authority). However, it is still possible to switch certificate validation off by specifying [**set http\_secure no**](/data-pipelines/extract/language/set.md#http_secure) in an Extractor script before executing an HTTP request. {% hint style="warning" %} The above will apply to all of your **Data Extractor**s where you are connecting to (most likely internal) data sources which are using **self-signed certificates**. Make sure to apply this change before upgrading to v3 to avoid any data extraction errors. {% endhint %} ## Extractor & Transformer Schedules In version 2.x.x it was still possible to schedule an Extractor or Transformer from the editor screen. This feature is removed from version 3.x.x. In case you still have schedules that are configured through this interface, you should *Unschedule* these and create an appropriate [**Workflow**](/data-pipelines/workflows.md#creating-a-workflow) instead. ![Example v2.x.x Schedule which will be removed in version 3.x.x](/files/-M4O08XTVBuo9FeqSi8o) ## GUI changes There are some minor changes in the GUI that are not backwards-compatible: * We removed the Excel export options from the reports. They were using the CSV format under the hood (i.e. they never actually produced valid Excel worksheets). In the future, we plan to implement proper Excel export formats for the reports, including a full summary report Excel export. * Removed functionality that would take a custom API port from the #port=xxx location hash parameter on the login screen. Specifying a custom API port (and hostname) is still possible by [setting up a `config.json` file](/administration/settings.md#config-json) on your system. ## Changes to data processing for reports Due to changes to the processing of reports, when making changes to either services, rates or adjustments, associated reports should be prepared. In v2.x.x this was already required when making changes to services or rates. Since v3.x.x this will be also required when making changes to adjustments. We've planned further improvements to make this more transparent (i.e. handling the preparation of reports automatically in the background). ## API changes ### Normalised date/time data in responses Some endpoints were returning dates and timestamps in different formats. This has been normalised in such a way that all responses use the same serialization for dates and timestamps: * A date is always represented as ISO-8601 string: `"yyyy-mm-dd"`, e.g. `"2020-01-29"` * A date/time is always represented as ISO-8601 string in UTC: `"yyyy-mm-ddThh:mm:ssZ"`, e.g. `"2020-01-29T11:26:52Z"`. Note that the Z suffix denotes the UTC time standard. This affects the responses (attributes) for the following group of API endpoints: * `/v1/audit` * created\_at * `/v1/budgetrevisions` * effective\_from * `/v1/dsets` * earliest\_rdf * latest\_rdf * rdf\_detail.created * rdf\_detail.updated * `/v1/extractors` * last\_modified * `/v1/log` * created * lines.date * `/v1/reports` * data\_status.first\_date * data\_status.last\_date * data\_status.status.date * `/v1/services` * created\_at * updated\_at * `/v1/transformers` * last\_modified * `/v1/workflows` * created\_at * updated\_at * `/v1/workflowschedules` * start\_time * next\_run ### Changed functionality #### `/v1/reports/{id}/run` endpoint * The JSON format in the response is simplified. See examples below. This is an example response from v2.x.x: ```javascript { "report": [ { "meta": { // ... }, "error": false, "data": [ { // ... } ] } ] } ``` This is an example response from v3.x.x: ```javascript { "report": [ { // ... } ] } ``` ### Remove deprecated functionality #### `/v1/usergroups` endpoint Removed deprecated permission aliases. * `upload_files` (use `manage_files` instead) * `manage_configuration` (use `manage_settings` instead) * `manage_system` (use `manage_settings` instead) #### `/v1/reports/{id}/run` endpoint Functionality deprecated in v2.x.x has been removed. If you were relying on any of the following functionality, please use the suggested replacement instead: * The `pdf/invoice` option for the `format` query parameter has been removed. Please use `pdf/summary` instead. * The `invoice_options` query parameter has been removed in favour of `summary_options`. #### `/v1/configuration` endpoint * Configuration keys prefixed with `INVOICE_` are replaced by respective keys prefixed with `SUMMARY_`. #### `/v1/workflowsteplogs` endpoint Also relevant to other endpoints including `workflowsteplogs` . * Removed the last\_log attribute. Include the last log by specifying the query parameters `include=steplogs&related[steplogs][limit]=1&related[steplogs][sort]=-start_ts`. * Removed the `timestamp` attribute. Use `start_timestamp` instead. * Removed the `message` attribute. Use a combination of `status` and `output` instead. #### `/v1/file` endpoint Only applicable to `POST` requests to this endpoint. * The `filename` in the response from this endpoint will no longer include the `import/` prefix to better align for other requests in this endpoint. See the example below: ```diff { - "filename": "/import/generic/2020/01/31_uploaded_001.txt" + "filename": "/generic/2020/01/31_uploaded_001.txt" } ``` #### `/v1/usergroups` endpoint The following permissions have been removed in favour of their new counterparts: * `UPLOAD_FILES` has become `MANAGE_FILES` * `MANAGE_CONFIGURATION` has become `MANAGE_SETTINGS` * `MANAGE_SYSTEM` has become `MANAGE_SETTINGS` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://olddocs.exivity.io/getting-started/releases/upgrading-to-version-3.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.