Exivity uses two main disk directories when it's installed. One to store the user data and one to store the software files. These directories are called the home and program directories, respectively.
On the system where Exivity is installed, the following environmental variables contain a absolute path reference to these directories: EXIVITY_HOME_DIRECTORY and EXIVITY_PROGRAM_DIRECTORY.
The home directory should preferably be located on a dedicated volume i.e. D:\exivity\home and it is recommended that it be located on an SSD drive.
The main program directory as it should be installed by the Exivity installer:
RDF stands for Reporting Database File. Exivity uses these to store usage data and configuration information.
A daily RDF stores the usage data from which Edify produces report data. Any given RDF contains a single DSET, along with internally managed metadata such as column types and prepared report data.
An RDF is created using the finish statement in a Transformer task. For any given DSET, there can be a single RDF per day, although there may be many RDF files per day in total (one RDF for each DSET).
RDFs are named according to the datadate and ID of the DSET they contain, and have a .rdf extension. For example a DSET with an ID of Azure.usage for the first day of the month will result in an RDF called 01_Azure.usage.rdf.
An RDF containing usage data is located at <home dir>/report/<yyyy>/<mm>/<dd>_<dset>.rdf where:
<yyyy> is the year of the datadate
<MM> is the month of the datadate
<dd> is the day of the datadate
<dset> is the DSET ID that was used to populate the usage data
The global database uses PostgreSQL (user data is stored at<home_dir>/system/pgdata) and contains system-wide configuration data including (but not necessary limited to):
Service definitions
Service rate revisions
Service rate adjustments
Service categories
User accounts
Report definitions (and related metadata such as sychronisation timestamps)
Metadata about RDF files created by Transcript
Usergroups
Security settings
Account information
Job Schedules
The Global database should never be manually modified unless this process is performed by, or under the guidance of, Exivity support staff. It is critical to make a backup copy of this file before any changes are made.
CSV files are a way of storing data in a table format. A table consists of one or more rows, each row containing one or more columns. There is no formal specification for CSV files although a proposed standard can be found at https://www.ietf.org/rfc/rfc4180.txt.
Although the 'C' in 'CSV' stands for the word comma, other characters may be used as the separator. TAB and semicolons are common alternatives. Exivity can import CSVs using any separator character apart from an dot or an ASCII NUL and uses the comma by default.
A Dataset is a CSV file, usually produced by USE, which can be imported for processing by a Transcript task. To qualify as a dataset a CSV file must meet the following requirements:
The first line in the file must define the column names
Every row in the file must contain the same number of fields
A field with no value is represented as a single separator character
The separator character must not be a dot (.)
There may be no ASCII NUL characters in the file (a NUL character has an ASCII value of 0)
When a Dataset is imported by a Transcript task any dots (.) contained in the a column name will be replaced with underscores (_). This is due to a dot being a reserved character used by Fully Qualified Column Names
When a DSET is exported during execution of a Transcript task, the exported CSV file will always be a Dataset, in that it can be imported by another Transcript task.
Although datasets are generated by USE as part of the extraction phase, additional information may be provided in CSV format for the purposes of enriching the extracted data. This additional CSV data must conform to the requirements above.
A DSET is the the data in a Dataset once it has been imported by a Transcript task. A DSET resides in RAM during the Transform phase and, if referenced by a finish statement, is then stored in a database file for long-term use.
A Transcript task may import more than one dataset during execution, in which case multiple DSETs will be resident in RAM at the same time. It is therefore necessary for any subsequent Transcript statement which manipulates the data in a DSET to identify which DSET (or in some cases DSETs) to perform the operation on. This is achieved using a DSET ID which is the combination of a Source tag and an Alias tag.
If multiple DSETs are present in memory, the first one that was created will be the default DSET. Column names that are not fully qualified are assumed to be located in the default DSET.
After a Dataset has been imported the resulting DSET is assigned a unique identifier such that it can be identified by subsequent statements. This unique identifier is termed the DSET ID and consists of two components, the Source and Alias tags.
The default alias tag is defined automatically and is the filename of the imported Dataset, minus the file extension. For example a Dataset called usage.csv will have an alias of usage.
The Transcript import statement can take one of two forms. Depending on which is used, the Source tag is determined automatically, or specified manually as follows:
By convention Datasets produced by USE are located in sub-directories within the directory <basedir>\collected. The sub-directories are named according to the data source and datadate associated with that data. The naming convention is as follows:
where:
<data_source> is a descriptive name for the external source from which the data was collected
<yyyy> is the year of the datadate as a 4-digit number
<MM> is the month of the datadate as a 2 digit number
<dd> is the day of the month of the datate as a 2 digit number
When importing one of these datasets using automatic source tagging, the source tag will be the name of the directory containing that dataset. Thus, assuming a datadate of 20160801 the following statement:
import usage from Azure
will import the Dataset <basedir>\collected\Azure\2016\08\01_usage.csv, and assign it a Source tag of Azure.
When importing a Dataset from a specified path, the Source tag is specified as part of the import statement. For example the statement:
import my_custom_data\mycosts.csv source costs
will import the Dataset <basedir>\my_custom_data\mycosts.csv and assign it a Source tag of costs. Checks are done during the import process to ensure that every imported Dataset has a unique source.alias combination.
When a column name is prefixed with a DSET ID in the manner described previously, it is said to be fully qualified. For example the fully qualified column name Azure.usage.MeterName explicitly refers to the column called MeterName in the DSET Azure.usage.
When a transformer runs, transcript.exe is executed and one of the command line arguments it requires is a date in yyyyMMdd format which is termed the data date. The activities performed by Transcript are associated with this date in several ways, most notably:
when importing using automatic source tagging which specific Dataset file to import from \collected
to determine the concept of 'today' when processing files containing data with timestamps spanning multiple days
to determine the output directory into which RDF files will be generated
to generate the filename of the RDF files generated
The data date is made available to a Transcript task through the automatic creation of the ${dataDate} variable
The following articles provide information regarding advanced concepts around which Exivity is built. These concepts are referenced repeatedly throughout the documentation and as such the below articles are recommended reading.