# Upgrading to version 3 {% hint style="warning" %} Upgrading to a v3.x.x will require upgrading to version v2.10.x first. The installer will verify this and displays a warning when this requirement is not satisfied. {% endhint %} ## PostgreSQL The biggest single change is the use of a new database engine powering all application state, audit logs and processed report data. Upgrading to this new database engine is transparent and the installer will take care of installing the database server as part of the regular installation process. After or during upgrade, it is possible to leverage an [external database server](https://olddocs.exivity.io/3.4.3/installation#database-server). ## Windows Services In version 2.x.x there were only two Exivity Services installed: ![Windows Services with Exivity version 2.x.x](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M4O5S33crT-JyPmd7Wt%2F-M4O75g3MhYGY8spCl-X%2Fimage.png?alt=media\&token=ae1c82cf-03f3-4d2d-b74c-4a58f1169272) With Exivity version 3.x.x, assuming all components are installed on a single host system, there will be total of 4 different services: ![Windows Services with Exivity version 3.x.x](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M4O5S33crT-JyPmd7Wt%2F-M4O7s2pbzSTHht7csW4%2Fimage.png?alt=media\&token=5605b040-ea4a-4cd8-8919-38c669fc746a) In case you were using a service account in Exivity version 2.x.x for the **Exivity Scheduling Service** and the **Exivity Web Service**, you will have to reconfigure this service account for both services, as well as the **Exivity API Service**. In most cases, the **Exivity Database Service** may continue to run un the *Local System* account. {% hint style="warning" %} In case your current Exivity version 2.x.x installation runs inside an Active/Passive **Windows Cluster**, you will need to re-register the Cluster Roles for the **Exivity Scheduling Service**. Additionally, a new Cluster Role should be created for the **Exivity Database Service**, in case you decide to not use an external database host. {% endhint %} ## Default TCP ports In v2.x.x, the default port for the Exivity GUI was 8001 and 8002 for the Proximity API. Both services were already available through port 443 (the default port used for https traffic, which means clients don't have to explicitly specify the port) and in v3.x.x this will be used by default: ![Default using port 443 for both UI and API](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M4sXC01x7rxHRBe2_Ra%2F-M4sjGYSoIxOAsSSwpy1%2Fimage.png?alt=media\&token=2b25a7d3-0e68-414c-95c3-74e04528b489) This is achieved by shipping a web proxy configuration for nginx, which routes all requests starting with `/v1/` to port 8002 and all other requests to port 8001. The recommended configuration is to not expose port 8001 to the public and only accept incoming traffic on port 443. Port 8002 may still be opened to external hosts, typically in a configuration where the Web and API/Backend components are deployed [seperate hosts](https://olddocs.exivity.io/3.4.3/installation/on-premises/multi-node#multi-node-system-architecture). In such a scenario it is advisable to allow communication from the [Web node towards the API/Backend node on port 8002](https://olddocs.exivity.io/3.4.3/installation/on-premises#configuring-a-separate-web-server-portal). ## Default Security Settings As of Exivity version 3 more strict security settings are applied by default. These can be found under *Administration > Settings.* One important item which should be considered when upgrading a multi node environment, is the use of CORS. It is required to list all possible front end UI nodes in the **CORS origins** field: ![Administering security settings](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M4sXC01x7rxHRBe2_Ra%2F-M4sguC47t_Z4Oig8xzC%2Fimage.png?alt=media\&token=81d27328-e88b-48e5-8c27-004de69dde4f) Multiple hosts including **`https://`** may be added, while seperating each URL using a `,` (comma) symbol. Wildcards may also be used as part of the hostname to match multiple URL's in one go such as: `https://*.cors.exivity.io`. An overview of all current security policies can be found [here](https://olddocs.exivity.io/3.4.3/advanced/security). ## Transformer changes #### @FILE\_EXISTS and @FILE\_EMPTY The [@FILE\_EXISTS](https://docs.exivity.com/data-pipelines/transform/language/if#file_exists) and [@FILE\_EMPTY](https://docs.exivity.com/data-pipelines/transform/language/if#file_empty) functions in Transcript have been modified in a manner which may require changes to scripts that use them. Previously, it would only check for files in the `system` and `exported` folders within the Exivity home directory, and if a specified path + filename did not start with `system/` or `exported/` then these would be prepended automatically before the check as done. This behaviour has been changed in the following ways in v3.x.x: * Any path + filename within the Exivity home directory can now be checked * path + filenames are now accepted and treated as being relative to the Exivity home directory * The folders `system/` and `exported/` are no longer automatically prepended Consider a file `somefile.csv` in the `%EXIVITY_HOME_PATH%/exported` folder. Previously with version 2.x.x a user could check for the existence of this file using the following example: ![v2: FILE\_EXISTS statement auto-prepends exported / system](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M56VZBOlY9nikQdfeX1%2F-M56_VVWYSGB-4z8i51s%2Fimage.png?alt=media\&token=35130b16-7a36-492b-830c-432ea016aa12) In version 3 it is required to include the entire path relative to the `%EXIVITY_HOME_PATH%` : ![v3: Path must be relative to %EXIVITY\_HOME\_PATH%](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M56VZBOlY9nikQdfeX1%2F-M56_hr19XXFCsHhDKzx%2Fimage.png?alt=media\&token=b4fdbe66-7563-4efd-8b74-ab63138f312b) {% hint style="warning" %} This change may **require modifications** to existing **Transformer** scripts. This is because `system` and `exported` will no longer be automatically prepended. {% endhint %} ## Extractor changes #### HTTP server certificate validation The default behaviour of the HTTP subsystem in USE was changed to fully validate server SSL certificates, which may cause some USE scripts to fail. This typically applies to data Extractors, which are connecting to on premise data sources that use self signed SSL certificated. In version 2.x.x, the default behavior was to ignore these certificate errors. You can identify these errors in your Extractor logs: ![Identifying Certificate Errors in the Extractor logs](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M4O2lvzV9hCrFUPPOsl%2F-M4O3upj9ZQrIGKWcqlJ%2Fimage.png?alt=media\&token=f75e8bee-e138-49e0-979c-366d398309c7) It is highly recommended to use valid SSL certificates, properly signed by a trusted CA (Certificate Authority). However it is still possible to switch certificate validation off by specifying [**set http\_secure no**](https://olddocs.exivity.io/3.4.3/data-pipelines/extract/language/set#http_secure) in an Extractor script before executing an HTTP request. {% hint style="warning" %} The above will apply to all of your **Data Extractor**s where you are connecting to (most likely internal) data sources which are using **self signed certificates**. Make sure to apply this change before upgrading to v3 to avoid any data extraction errors. {% endhint %} ## Extractor & Transformer Schedules In version 2.x.x it was still possible to schedule an Extractor or Transformer from the editor screen. This feature is removed from version 3.x.x. In case you still have schedules which are configured thru this interface, you should *Unschedule* these and create an approperiate [**Workflow**](https://olddocs.exivity.io/3.4.3/data-pipelines/workflows#creating-a-workflow) instead. ![Example v2.x.x Schedule which will be removed in version 3.x.x](https://3540922554-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHEKskLK6aXinV75Knl%2F-M4Nz1gdP_D75NmEmC94%2F-M4O08XTVBuo9FeqSi8o%2Fimage.png?alt=media\&token=cf7c88da-96f8-4c2a-832f-127fab6adab9) ## GUI changes There are some minor changes in the GUI which are not backwards-compatible: * We removed the Excel export options from the reports. They were using the CSV format under the hood (i.e. they never actually produced valid Excel worksheets). In the future, we plan to implement proper Excel export formats for the reports, including a full summary report Excel export. * Removed functionality which would take a custom API port from the #port=xxx location hash parameter on the login screen. Specifying a custom API port (and hostname) is still possible by [setting up a `config.json` file](https://olddocs.exivity.io/3.4.3/administration/settings#config-json) on your system. ## Changes to data processing for reports Due to changes to the processing of reports, when making changes to either services, rates or adjustments, associated reports should be prepared. In v2.x.x this was already required when making changes to services or rates. Since v3.x.x this will be also required when making changes to adjustments. We've planned further improvements making this more transparent (i.e. handling preparation of reports automatically in the background). ## API changes ### Normalised date/time data in responses Some endpoints were returning dates and timestamps in different formats. This has been normalised in such a way that all responses use the same serialisation for dates and timestamps: * A date is always represented as ISO-8601 string: `"yyyy-mm-dd"`, e.g. `"2020-01-29"` * A date/time is always represented as ISO-8601 string in UTC: `"yyyy-mm-ddThh:mm:ssZ"`, e.g. `"2020-01-29T11:26:52Z"`. Note that the Z suffix denotes the UTC time standard. This affects the responses (attributes) for the following group of API endpoints: * `/v1/audit` * created\_at * `/v1/budgetrevisions` * effective\_from * `/v1/dsets` * earliest\_rdf * latest\_rdf * rdf\_detail.created * rdf\_detail.updated * `/v1/extractors` * last\_modified * `/v1/log` * created * lines.date * `/v1/reports` * data\_status.first\_date * data\_status.last\_date * data\_status.status.date * `/v1/services` * created\_at * updated\_at * `/v1/transformers` * last\_modified * `/v1/workflows` * created\_at * updated\_at * `/v1/workflowschedules` * start\_time * next\_run ### Changed functionality #### `/v1/reports/{id}/run` endpoint * The JSON format in the response is simplified. See examples below. This is an example response from v2.x.x: ```javascript { "report": [ { "meta": { // ... }, "error": false, "data": [ { // ... } ] } ] } ``` This is an example response from v3.x.x: ```javascript { "report": [ { // ... } ] } ``` ### Remove deprecated functionality #### `/v1/usergroups` endpoint Removed deprecated permission aliases. * `upload_files` (use `manage_files` instead) * `manage_configuration` (use `manage_settings` instead) * `manage_system` (use `manage_settings` instead) #### `/v1/reports/{id}/run` endpoint Functionality deprecated in v2.x.x has been removed. If you were relying on any of the following functionality, please use the suggested replacement instead: * The `pdf/invoice` option for the `format` query parameter has been removed. Please use `pdf/summary` instead. * The `invoice_options` query parameter has been removed in favour of `summary_options`. #### `/v1/configuration` endpoint * Configuration keys prefixed with `INVOICE_` are replaced by respective keys prefixed with `SUMMARY_`. #### `/v1/workflowsteplogs` endpoint Also relevant to other endpoints including `workflowsteplogs` . * Removed the last\_log attribute. Include the last log by specifying the query parameters `include=steplogs&related[steplogs][limit]=1&related[steplogs][sort]=-start_ts`. * Removed the `timestamp` attribute. Use `start_timestamp` instead. * Removed the `message` attribute. Use a combination of `status` and `output` instead. #### `/v1/file` endpoint Only applicable to `POST` requests to this endpoint. * The `filename` in the response from this endpoint will no longer include the `import/` prefix to better align for other requests in this endpoint. See example below. ```diff { - "filename": "/import/generic/2020/01/31_uploaded_001.txt" + "filename": "/generic/2020/01/31_uploaded_001.txt" } ``` #### `/v1/usergroups` endpoint The following permissions has been removed in favour of their new counterparts: * `UPLOAD_FILES` has become `MANAGE_FILES` * `MANAGE_CONFIGURATION` has become `MANAGE_SETTINGS` * `MANAGE_SYSTEM` has become `MANAGE_SETTINGS`