The Workflows menu allows you to schedule various tasks and execute them at a specific date and time. This allows the execution of different Extractors and Transformers, so that they are tightly chained together.
To create a new Workflow, go to 'Data Pipelines' > 'Workflows', then click the green button labeled 'Create Workflow'. Then create one or more New Schedule intervals:
Then add one or more steps that should be executed:
Provide a meaningful Name for the Workflow
Optionally you can provide a detailed Description
Set the Type of the step using the drop-down menu
Provide the option that goes with the selected Type. This can be your Extractor, Transformer, Report or another name
Depending on the selected Step Type, you can provide an offset date. This value is used during the execution of that step. Typically this would be used for a From date offset (i.e. -1 for yesterday)
A To date offset can be provided for some step types (i.e. 0 for today)
In case multiple steps need to be executed in parallel, you may choise to remove the Wait checkbox.
Additional arguments can be sent to the step. This applies only to some Extractors and when executing a custom Command
You can delete a step using the bin icon in the right upper corner. To execute a workflow click the Run Now button:
Bear in mind that the selected date will be influenced by date offsets difined in any given step. To view historical Workflow results, click the Status tab.
Apart from adding Extractor, Transformer and Report steps, there are a few other special Workflow Step types:
This step type will trigger a Publish Report event, which can be consumed by the notification engine.
This step type will evaluate all budgets, which can be consumed by the notification engine.
The Execute step enables you to execute an external command, like a script:
As an example: you could run a Powershell script to obtains some data from a special data source that Exivity Extractors are not able or allowed to connect to. This script could be executed the following manner:
The above command calls the Powershell executable to run the special.ps1 script, with a dynamically generated parameter that is evaluated at run time. This particular example always provides yesterdays date in yyyyMMdd format as a parameter to the special.ps1 script. Many other variations and scripting languages are possible. Feel free to experiment.
