Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The buffer command is used to create and/or populate one of these named buffers with data.
buffername=protocol protocol_parameter(s)
The first argument to the buffer statement is the name of the buffer to create. If a buffer with this name already exists then any data it contains will be overwritten.
There must be whitespace on both sides of the 'equals' symbol following the buffer name.
The following protocols are supported:
bufferbuffername= filefilename
The file protocol imports a file directly into a buffer. This can be very useful when developing USE scripts, as the USE script for processing for a JSON file (for example) can be implemented without requiring access to a server.
If the specified buffer name already exists, then a warning will be logged and any data in it will be cleared before importing the file.
bufferbuffername= datastring
The data protocol populates the buffer with the literal text specified in string. This is useful when extracting embedded JSON. For example the JSON snippet below contains embedded JSON in the instanceData field:
In this case the instanceData field can be extracted using a parslet, placed into a new buffer and re-parsed to extract the values within it. Assuming the snippet is in a file called my_data.json this would be done as follows:
bufferbuffername= httpmethod url
!!! note For full details on the HTTP protocol and its parameters please refer to the http article.
Once the HTTP request has been executed, any data it returned will be contained in the named buffer, even if the data is binary in format (eg: images, audio files or anything else non-human readable).
If the HTTP request returned no data, one of the following will apply:
If the buffer does not already exist then the buffer will not be created
If the buffer already exists then it will be deleted altogether
For details of how to access the data in a named buffer, please refer to the USE script basics article.
bufferbuffername= odbcdsn [username password] query
username and password are optional, but neither or both must be specified
where:
dsn is the ODBC Data Source Name (this should be configured at the OS level)
username and password are the credentials required by the DSN
query is an SQL query
Once the query has been executed, the resulting data is located in the named buffer. It can subsequently be saved as a CSV file to disk using:
save {buffername} as filename.csv
The resulting CSV uses a comma (,) as the separator and double quotes (") as the quoting character. Any fields in the data which contain a comma will be quoted.
bufferbuffername= odbc_directquery
where query is an SQL query.
Executes SQL query against ODBC datasource that is described in set's odbc_connect parameter.
Once the query has been executed, the resulting data is located in the named buffer. It can subsequently be saved as a CSV file to disk using:
save {buffername} as filename.csv
The resulting CSV uses a comma (,) as the separator and double quotes (") as the quoting character. Any fields in the data which contain a comma will be quoted.
The following examples retrieve data from ODBC and HTTP sources:
The basename statement is used to extract the filename portion of a path + filename string
basenamevarName
basenamestringasvarName
Given a string describing the full path of a file, such as /extracted/test/mydata.csv the basename statement is used to identify the filename (including the file extension, if any) portion of that string only. If there are no path delimiters in the string then the original string is returned.
The basename statement supports both UNIX-style (forward slash) and Windows-style (backslash) delimiters.
When invoked as basename varName, the varName parameter must be the name of the variable containing the string to analyse. The value of the variable will be updated with the result so care should be taken to copy the original value to a new variable beforehand if the full path may be required later in the script.
As a convenience in cases where the full path needs to be retained, the result of the operation can be placed into a separate variable by using the form basename string as varName where string is the value containing the full path + filename and varName is the name of the variable to set as the result.
When invoked using basename string as varName if a variable called varName does not exist then it will be created, else its value will be updated.
The following script ...
... will produce the following output:
The following script ...
... will produce the following output:
The csv statement is used to create and populate CSV files. It is typically combined with foreach loops to write values extracted from an array in a JSON and/or XML document stored in a named buffer.
CSV files are produced via the use of multiple csv statements which perform the following functions:
Create a new empty CSV file
Define the headers
Finalise the headers
Write data to one or more rows of the file
Close the file
All CSV files created by the csv command use a comma - , - as the separator character and a double quote - " - as the quote character. Headers and data fields are automatically separated and quoted.
The following is used to create a new, empty CSV file:
csvlabel = filename
The label must not be associated with any other open CSV file. Up to 16 CSV files may be open simultaneously and the label is used by suesequent csv statements to determine which of the open files the statement should operate on. Labels are case sensitive and may be from 1 to 15 characters in length.
The specified filename is created immediately, and if it is the name of an existing file then it will be truncated to 0 bytes when opened.
The filename argument may contain a path component but the csv statement does not create directories, so any path component in the filename must already exist. The path, if specified, will be local to the Exivity home directory.
csv usage = "${exportdir}/azure_usage.csv"
This section refers to add_headers as the action, but either add_header or add_headers may be used. Both variants work in an identical fashion.
csv add_headerslabel header1 [header2 ... headerN]
All CSV files created by USE script must start with a header row which names the columns in the file. The number of columns can vary from file to file, but in any given file every data row must have the same number of columns as there are headers.
To create one or more columns in a newly created CSV file, the csv add_headers statement is used as shown above. The label must match the label previously associated with the file as described previously.
One or more header names can be specified as arguments to csv add_headers. Multiple instances of the csv add_headers statement may reference the same CSV file, as each statement will append additional headers to any headers already defined for the file.
No checks are done to ensure the uniqueness of the headers. It is therefore up to the script author to ensure that all the specified headers in any given file are unique.
csv add_headers usage username user_id subscription_id
This section refers to fix_headers as the action, but either fix_header or fix_headers may be used. Both variants work in an identical fashion
csv fix_headerslabel
After csv add_headers has been used to define at least one header, the headers are finalised using csv fix_headers statement. Once the headers have been fixed, no further headers can be added to the file and until the headers have been fixed, no data can be written to the file.
csv fix_headers usage
This section refers to write_fields as the action, but either write_field or write_fields may be used. Both variants work in an identical fashion
csv write_fieldslabel value1 [value2 ... valueN]
After the headers have been fixed, the csv write_fields statement is used to write one or more fields of data to the CSV file. Currently it is not possible to write a blank field using csv write_fields, however when extracting data from a buffer using a parslet, if the extracted value is blank then it will automatically be expanded to the string (no value).
USE keeps track of the rows and columns as they are populated using one or more csv write_fields statements, and will automatically write the fields from left to right starting at the first column in the first data row and will advance to the next row when the rightmost column has been written to.
It is the responsibility of the script author to ensure that the number of fields written to a CSV file is such that when the file is closed, the last row is complete, in order to avoid malformed files with one or more fields missing from the last row.
csv closelabel
Once all fields have been written to a CSV file, it must be closed using the csv close statement. This will ensure that all data is properly flushed to disk, and will free the label for re-use.
csv close usage
Consider the file "\examples\json\customers.json" representing two customers:
Using a combination of foreach loops and parslets, the information in the above JSON can be converted to CSV format as follows:
The resulting CSV file is as follows:
The escape statement is used to escape quotes in a variable value or the contents of a named buffer
escape quotes invarName|{bufferName}
If a variable value or named buffer contains quotes then it may be desirable to escape them, either for display purposes (to prevent USE from removing them before rendering the data as output) or in order to satisfy the requirements of an external API.
The escape statement will precede all occurrances of the character " with a backslash as shown in the example below. This operation is not just temporary - it will update the actual contents of the variable or named buffer.
The escape statement does not consider the context of any existing quotes in the data. Running it multiple times against the same data will add an additional escape character each time to each occurrence of a quote.
Given an input file called 'escapeme.txt' containing the following data:
The following script:
will produce the following output:
The encode statement is used to base16 or base64 encode the contents of a variable or a named buffer.
encode base16|base64varName|{buffer_name}
The encode statement will encode the contents of an existing variable or named buffer, replacing those contents with the encoded version.
The result of encoding the contents will increase their length. With base16 encoding the new length will be double the original. With base64 encoding the new length will be greater than the original but the exact size increase will depend on the contents being encoded.
When encoding a variable, if the size of the result after encoding exceeds the maximum allowable length for a variable value (8095 characters) then the USE script will fail and an error will be returned.
Encoding an empty variable or buffer will produce an empty result
The following script ...
... produces the following output:
The aws_sign_String statement is used to generate an AWS4-HMAC-SHA256 signature, used as the signature component of the Authorization HTTP header when calling the AWS API.
aws_sign_stringvarNameusingsecret_key date region service
The authentication method used by AWS requires the generation of an authorization signature which is derived from a secret key known to the client along with specific elements of the query being made to the API.
This is a fairly involved process and a full step-by-step walkthrough is provided by Amazon on the following pages (these should be read in the order listed below):
The aws_sign_string statement is used to generate the final signature as detailed on the calculate signature page listed above.
Note that in order to use this statement it is necessary to have the following strings available:
A string to sign, obtained by following the process creating a string to sign, containing meta-data about the request being made
A secret_key, obtained from Amazon which is used by any client application authorising against their API
The date associated with the API request, in YYYYMMDD format
The AWS region associated with the API request (for example eu-central-1)
The AWS service being accessed (for example s3)
The aws_sign_string statement will use these inputs to generate the HMAC-SHA256 signature which is a component of the Authorization header when connecting to the API itself.
The varName parameter is the name of a variable containing the string to sign. After executing aws_sign_string the contents of this same variable will have been updated to the base-16 encoded signature value.
If there are any errors in the string to sign, _date, AWS region or AWS service strings used as input to aws_sign_string then a signature will still be generated, but the AWS API will reject the request. In this case it is necessary to review the process by which these strings were created as per the AWS guide provided above.
The following is an example USE script that implements everything described above.
The functionality described in this article is not yet available. This notice will be removed when the appropriate release is made.
The gunzip statement is used to inflate a GZIP file
gunzip filename as filename
gunzip {bufferName} as filename
The gunzip statement can be used to extract the contents of a GZIP archive containing a single file. The GZIP archive may be a file on disk or may be the contents of a named buffer.
It is not possible to inflate GZIP data directly in memory, but the same effect can be achieved by extracting GZIP data in a named buffer to disk, and then loading the extracted data back into the named buffer as shown in the example below.
All paths and filenames are treated as relative to the Exivity home directory
This article assumes knowledge of variables.
The encrypt statement is used to conceal the value of a variable, such that it does not appear in plain text in a USE script.
encrypt varname = value_to_be_encrypted
The encrypt statement differs from other statements in that it takes effect before execution of a USE script begins. In this regard is is effectively a directive to the internal script pre-processor which prepares a script for execution.
Comments, quotes and escapes in the value to be encrypted are treated as literal text up until the end of the line.
White-space following the value to be encrypted will therefore be included in the encrypted result.
White-space preceding the value to be encrypted will be ignored and will not be included in the encrypted result.
Any variable prefixed with the word encrypt will be encrypted by the pre-processor and the script file itself will be modified as follows:
All text (including trailing white-space) from the word following the = character up to the end of the line is encrypted
The encrypted value is base64 encoded
The original variable value in the USE script is substituted with the result
The encrypt keyword for that variable is changed to encrypted
The USE script is overwritten on disk in this new form
This process is repeated for all variables preceded by the encrypt keyword.
As a side effect of the encryption process, it is not currently possible to encrypt a value that begins with a space or a tab. This functionality will be implemented in due course.
Once encrypted a variable can be used just as any other, the only requirement being that the encrypted keyword preceding its declaration is not removed or modified.
To change the value of an encrypted variable simply replace the declaration altogether and precede the new declaration with encrypt. Upon first execution, the USE script will be updated with an encrypted version of the variable as described above.
Encrypted values can only be used on the system that they were created on. If an encrypted value is moved or copied to a different installation of Exivity then any attempt to reference or decrypt it will result in something other than the original value.
Firstly, create the script as usual, with encrypt preceding any variables that are to be encrypted:
Secondly, run the script. Prior to execution the script will be automatically modified as shown below:
The hash statement is used to generate a base-16 or base-64 encoded hash of data stored in a variable or named buffer.
hash sha256 [HMACkey] target|{target}asresult[b16|b64]
hash md5target|{target}asresult[b16|b64]
The hash statement uses the contents of target as its input and places the final result into result. The SHA256 and MD5 hash algorithms are supported.
If target is surrounded with curly braces like {this} then it is taken to be the name of a memory buffer and the contents of the buffer will be used as input. Otherwise, it is treated as the name of the variable, the value of which will be hashed.
By default the resulting hash is base-16 encoded and the result placed into the variable specified by the result argument.
result is the name of the variable to put the output into, and not a reference to the contents of that variable. This is why it is not ${result}
If the optional HMACkey arguments are provided when the hash type is sha256 then the secret in key will be used to generate an HMAC-SHA-256 result.
If the optional b64 argument is used (base64 may also be specified), then the result will be encoded using base-64.
The optional b16 (base16 may also be used) is provided for completeness, but need not be specified as this is the default encoding to use.
Running the script:
results in the following output:
This article links to detailed descriptions of all the statements supported by USE script.
These descriptions assume knowledge of the USE script basics.
The foreach statement defines a block of zero or more statements and associates this block with multiple values. The block of statements is executed repeatedly, once for each value.
foreachparsletasloop_label{
# Zero or more USE script statements go here
}
The opening { may be placed on a line of its own if preferred
The foreach statement is used to iterate over the values in an array or object (identified by a parslet) within the data in a named buffer.
The loop will execute for as many elements as there are in the array, or for as many members there are in the object. For the purposes of this documentation, the term child will be used to refer to a single array element or object member.
If the array or object is empty, then the body of the loop will be skipped and execution will continue at the statement following the closing }.
The loop_label can be any string, but must not be the same as any other loop_label values in the same scope (ie: when nesting foreach loops, each loop must have a unique label). This label is used to uniquely identify any given loop level when loops are nested.
The foreach statement will execute the statements in the body of the loop once for every child. foreach loops can be nested, and at each iteration the loop_label can be used to extract values from an array or object in the current child using a dynamic parslet. See the examples at the end of this article for a sample implementation showing this in action.
As the foreach loop iterates over the children, a number of variables are automatically created or updated as follows:
Consider the following JSON in a file called samples/json/array.json:
To generate a list of IDs and names from the items array, the following would be used:
To extract values from an array using nested loops:
Given the source JSON in a file called example.json, the following USE script:
will produce the following output:
The days_in_month statement sets a variable to contain the number of days in the specified month
get_last_day_ofyyyyMMasvarName
The get_last_day_of statement will set the value of the variable called varName to contain the number of days in the month specifed by yyyyMM where yyyy is a four-digit year and MM is a 2-digit month.
The statement will take leap years into account.
The discard statement is used to delete a named .
discard{buffer_name}
The discard statement will delete the named buffer and free the memory used to store its contents. The statement takes immediate effect and any attempt to reference the buffer afterwards (at least until such time as another buffer with the same name is created) will cause the USE script to log an error and fail.
The gosub keyword is used to run a named subroutine
gosub subroutineName([argument1, ... argumentN])
The argument list may span multiple lines, so long as any given argument is contained on a single line and ends with a comma, eg:
The subroutineName provided to the gosub statement must be that of a subroutine defined elsewhere in the script using the statement.
If any argument contains white-space or a comma then it must be quoted:
gosub getfile("directory with spaces/filename.txt")
It is permitted to call a subroutine from within another subroutine, therefore gosub can be used within the body of a subroutine. This may be done up to 256 levels in depth.
The opening bracket after subroutineName may or may not be preceded with a space:
gosub getfile ("filename.txt")
To call a subroutine with no parameters, use empty brackets:
gosub dosomething()
The http statement initiates an HTTP session using any settings previously configured using the statement. It can also be used for querying response headers.
httpmethod url
http dump_headers
http get_headerheaderNameasvarName
The http statement performs an HTTP request against the server and resource specified in the url paramater. Any http-related settings previously configured using will be applied to the request.
The method argument determines the HTTP method to use for the request and must be one of GET, PUT, POST or DELETE.
The url argument must start with either http: or https:. If https: is used then SSL will be used for the request.
The url argument must also contain a valid IP address or hostname. Optionally, it may also contain a port number (preceded by a colon and appended to the IP address or hostname) and a resource.
The following defaults apply if no port or resource is specified:
To dump a list of all the response headers returned by the server in the most recent session use the statement:
http dump_headers
This will render a list of the headers to standard output, and is useful when implementing and debugging USE scripts. The intention of this statement is to provide a tool to assist in script development, and as such it would normally be removed or suppressed with a debug mode switch in production environments.
To retrieve the value of a specific header, use the statement:
http get_headerheaderNameasvarName
This will set the variable varName to be the value of the header headerName.
If headerName was not found in the response, then a warning will be written to the log-file. In this case varName will not be created but if it already exists then its original value will be unmodified.
The following shows the process of retrieving a header. The output of:
Takes the following form:
The exit_loop statement will terminate the current loop.
Either exit_loop or loop_exit may be used. Both variants work identically.
exit_loop
The exit_loop statement will immediately terminate the current loop and script execution will jump to the statement following the } at the end of the current loop.
This can be done even if the exit_loop statement is within one or more constructs inside the loop.
If no loop is in effect then an error will be logged and the script will terminate.
Please refer to the example in the documentation for the statement
The format of the http statement is identical when used in conjunction with the statement.
Statement
Description
Create an AWS4-HMAC-SHA256 signature value
Extract the filename from path + filename string
Create a named buffer
Delete any defined HTTP headers
Create a CSV file
Delete a named buffer
Base16 or base64 encode data
Encrypt a variable
Break out of a loop
Iterate over an array
Set a variable to contain the number of the last day of a specified month
Call a subroutine
Inflate GZIP data
Generate an SHA256 or HMACSHA256 hash
Execute an HTTP request
Conditionally execute statements
Format JSON data
Change the logging level
Execute statements repeatedly
Search using a regular expression
Suspend script execution
Echo text to standard output
Explicitly return from a subroutine
Save a named buffer to disk
Specify a protocol parameter
Define a subroutine
End script execution
Decompress ZIP data in a named buffer
URI (percent) encode a variable
Create or update a variable
Variable
Value
loop_label.COUNT
The number of times the loop has executed. If the object or array is empty then this variable will have a value of 0.
loop_label.NAME
The name of the current child
loop_label.VALUE
The value of the current child
loop_label.TYPE
The type of the current child
Field | Default |
port |
|
resource |
|
The clear statement is used to delete all HTTP headers previously configured using the set http_header statement.
clear http_headers
The clear statement will remove all the headers currently defined, after which a new set of headers can be specified using set http_header.
The if statement is used to conditionally execute one or more statements. In conjunction with an optional else statement it can cause one or other of two blocks of statements to be executed depending on whether an expression is true or false.
if(expression){
# Statements
} [else {
# Statements
}]
If the condition evaluates to true, then the first block of statements is executed, and the second block (if present) is skipped over. If the condition evaluates to false then the first block of statements is skipped and the second block (if present) is executed.
The opening { character at the start of each block may be placed on a line of its own if preferred but the closing } must be on a line of its own.
Multiple conditions can be used in a single expression and combined with the boolean operators && or || (for AND and OR respectively) so long as each condition is enclosed in braces. For example:
Given the source JSON in a file called example.json, the following USE script:
will produce the following output:
The json statement is used to format JSON in a named buffer.
json format{buffername}
In many cases an API or other external source will return JSON in a densely packed format which is not easy for the human eye to read. The json statement is used to re-format JSON data that has been previously loaded into a named buffer (via the buffer statement) into a form that is friendlier to human eyes.
Given the following single packed line of JSON in a named buffer called myJSON:
The following USE script fragment:
will result in the following output:
While executing a USE script, various messages are written to a logfile. The loglevel option determines the amount of detail recorded in that logfile.
loglevelloglevel
The table below shows the valid values for the loglevel argument. Either the numeric level or the label can be specified. If the label is used then it must be specified in CAPITAL LETTERS.
The log levels are cumulative, in that higher log-level values include lower level messages. For example a level of INFO will cause FATAL, ERROR, WARN and INFO level messages to be written to the log.
The loglevel statement takes immediate effect and may be used multiple times within a USE script in order to increase or decrease the logging level at any time.
The match statement is to used search either a specified string or the contents of a named buffer using a regular expression.
matchlabel expression target
The three parameters serve the following purposes:
The label associates a meaningful name to the search. Once the match has been attempted, two variables will be created or updated as follows:
These variables can be checked after the match in order to determine the result status and access the results.
The regular expression must contain one or more characters enclosed in brackets - ( ... ) - the contents of which are termed a subgroup. If a successful match is made then the portion of the target text that was matched by the subgroup will be returned in the label.RESULTvariable.
The target determines whether a supplied string or the contents of a named buffer are searched. By default the parameter will be treated as a string.
If the string contains white-space then it must be enclosed in double quotes
If the target argument is surrounded with curly braces - { ... } - then it is taken to be the name of a buffer and the expression will be applied to the contents of that buffer.
Regular expressions are generally used for searching ASCII data. Searching binary data is possible but may be of limited usefulness.
Search the contents of a variable for the text following the word 'connection:' with or without a capital 'C':
Search a text file previously retrieved from a HTTP request to locate the word 'Error' or 'error'
The return statement is used to exit a subroutine at an arbitrary point and return to the calling location
return
A will automatically return to the location it was called from when the end of its body is reached. However it may be desirable to explicitly exit the subroutine at some other point in which case the return statement is used.
The return statement cannot be used to return a value to the calling code (this should be done via the use of variables as described in the statement documentation)
The subroutine keyword is used to define a named subroutine
A subroutine is a named section of code that can be executed multiple times on demand from anywhere in the script. When called (via the statement), execution of the script jumps to the start of the specified subroutine. When the end of the code in the subroutine body is reached or a statement is encountered (whichever comes first), execution resumes at the statement following the most recent statement that was executed.
The code in the body of a subroutine statement is never executed unless the subroutine is explicitly called using . If a subroutine is encountered during normal linear execution of the script then the code in it will be ignored.
Subroutines in USE do not return any values, but any variables that are set within the subroutine can be accessed from anywhere in the script and as such they should be used for returning values as needed.
When invoked via the statement, arguments can be passed to the subroutine. These arguments are read-only but may be copied to normal variables if required.
Arguments are accessed using the same syntax as is used for variables as follows:
${SUBARG.COUNT} contains the number of arguments that were passed to the subroutine
${SUBARG_N} is the value of any given argument, where N is the number of the argument starting at 1
Every time a subroutine is called, any number of arguments may be passed to it. These arguments are local to the subroutine and will be destroyed when the subroutine returns. However, copying an argument to a standard variable will preserve the original value as follows:
After the subroutine above has been executed the return_value variable will retain the value it was set to.
It is not permitted to nest subroutine statements. If used within the body of a subroutine statement, a subroutine statement will cause the script to terminate with an error.
The following demonstrates using a subroutine to detect when another subroutine has been provided with an incorrect number of arguments:
The loop statement executes one or more statements multiple times.
looplabel [count][timeouttimelimit]{
# Statements
}
The opening { may be placed on a line of its own if preferred but the closing } must be on a line of its own
The loop statement will loop indefinitely unless one of three exit conditions cause it to stop. These are as follows:
The number of loops specified by the count parameter are completed
At least as many milliseconds as are specified by the timelimit parameter elapse
An statement explicitly exits the loop
In all three cases when the loop exits, execution of the script will continue from the first statement after the closing } marking the end of the loop.
In the event that both count and timelimit parameters are specified, the loop will exit as soon as one or other of the limits have been reached, whichever comes first.
Both the count and timeout parameters are optional. If omitted then the default for both of them will be infinite.
The loop statement will automatically create and update a variable called loop_label.COUNT which can be referenced to determine how many times the loop has executed (as shown in the example below). This variable is not deleted when the loop exits which means that it is possible to know how many times any given loop executed, even after the loop has exited.
Any specified timeout value is evaluated at the end of each execution of the loop and as such the actual time before the loop exits is likely to be a short time (typically a few milliseconds) greater than the specified value. In practice this should be of no consequence.
The loop shown above will result in the following output:
The print statement is used to display text to standard output while a USE script is executing.
print [-n]word|{buffer_name} [... word|{buffer_name]
The print statement enables user-defined output to be generated during the execution of a USE script. When retrieving data from external sources it may take some time for a lengthy series of operations to complete, so one use of the print statement is to provide periodic status updates during this time.
The print statement will process as many arguments as it is given, but at least one argument is required. If the first argument is -n then no newline will be output after the last argument has been echoed to standard output, else a newline is output after the last argument.
Arguments that are normal words will be sent to standard output followed by a space. Arguments referencing a named buffer will result in the contents of the buffer being displayed.
Note that print will stop output of data from a named buffer as soon as a NUL (ASCII value 0) character is encountered
The set statement is used to configure a setting for use by a subsequent or statements.
setsetting value
A protocol such as offers a number of configuration options. Any given option is either persistent or transient:
The following settings can be configured using set:
set http_progress yes|no
Persistent. If set to yes then dots will be sent to standard output to indicate that data is downloading when an HTTP session is in progress. When downloading large files if a lengthy delay with no output is undesirable then the dots indicate that the session is still active.
set http_usernameusername
Persistent. Specifies the username to be used to authenticate the session if the http_authtype setting is set to anything other than none. If the username contains any spaces then it should be enclosed in double quotes.
set http_passwordpassword
Persistent. Specifies the password to be used to authenticate the session if the http_authtype setting is set to anything other than none. If the password contains any spaces then it should be enclosed in double quotes.
set http_authtypetype
Persistent. Specifies the type of authentication required when initiating a new connection. The type parameter can be any of the following:
set http_authtargettarget
Valid values for target are:
server (default) - authenticate against a hostname directly
proxy - authenticate against the proxy configured at the Operating System level
set http_header"name: value"
Persistent. Used to specify a single HTTP header to be included in subsequent HTTP requests. If multiple headers are required, then multiple set http_header statements should be used.
An HTTP header is a string of the form name: value.
There must be a space between the colon at the end of the name and the value following it, so the header should be enclosed in quotes
Example: set http_header "Accept: application/json"
Typically these will be an Accept: header for GET requests and an Accept: and a Content-Type: header for POST requests. However there is no hard and fast standard so the documentation for any API or other external endpoint that is being queried should be consulted in order to determine the correct headers to use in any specific scenario.
Headers are not verified as sane until the next HTTP connection is made
set http_body datastring - use the specified string as the body of the request
set http_body filefilename - send the specified file as the body of the request
set http_body{named_buffer} - send the contents of the named buffer as the body of the request
Transient. By default no data other than the headers (if defined) is sent to the server when an HTTP request is made. The http_body setting is used to specify data that should be sent to the server in the body of the request.
When using http_body a Content-Length: header will automatically be generated for the request. After the request this Content-Length: header is discarded (also automatically). This process does not affect any other defined HTTP headers.
After the request has been made the http_body setting is re-initialised such that the next request will contain no body unless another set http_body statement is used.
set http_savefilefilename
If no response is received from the next request after using set http_savefile then the setting will be ignored and no file will be created.
Regardless of whether the server sent a response or not after the HTTP request has completed, the http_savefile setting is re-initialised such that the next request will not cause the response to be saved unless another set http_savefile statement is used.
No directories will be created automatically when saving a file, so if there is a pathname component in the specified filename, that path must exist.
set http_savemodemode
Persistent.
If mode is overwrite (the default) then if the filename specified by the set http_savefile statement already exists it will be overwritten if the server returns any response data. If no response data is sent by the server, then the file will remain untouched.
If mode is append then if the filename specified by the set http_savefile statement already exists any data returned by the server will be appended to the end of the file.
set http_timeoutseconds
Persistent. After a connection has been made to a server it may take a while for a response to be received, especially on some older or slower APIs. By default, a timeout of 5 minutes (300 seconds) is endured before an error is generated.
This timeout may be increased (or decreased) by specifying a new timeout limit in seconds, for example:
The minimum allowable timeout is 1 second.
set odbc_connectconnection_string
A DSN connection string must contain a DSN attribute and optional UID and PWD attributes. A non-DSN connection string must contain a DRIVER attribute, followed by driver-specific attributes.
Please refer to the documentation for the database to which you wish to connect to ensure that the connection string is well formed.
An example connection string for Microsoft SQL Server is:
The save statement is used to write the contents of a to disk.
save{buffer_name}asfilename
The save statement will write the contents of a to filename. As well as providing a means of direct-to-disk downloading this can be useful for retrieving server responses and capturing them for later examination, whether it be for analysis, debugging or audit purposes.
If the destination file already exists then it will be overwritten.
If the filename argument contains a path component, then any directories not present in the path will be created. If creation of the path destination file is not successful then an error will be logged and the USE script will fail.
The save statement is similar in effect to the option supported by , in that data from a server is written to disk. There is one important distinction however:
When has been used to specify a file to save, the next request will stream data to the file as it is received from the server
When a statement is used to capture the server response, and a subsequent save statement is used to write it to disk, all the buffered data will be written to the file immediately
The pause statement is used to suspend execution of a USE script for a specified time.
pausedelaytime
The delaytime parameter is the number of milliseconds to wait before continuing. A value of 0 is allowed, in which case no delay will occur.
The pause statement may be useful in cases where an external data source imposes some form of rate limiting on the number of queries that can be serviced in a given time-frame, or to slow down execution at critical points when debugging a long or complex script.
This example makes use of script parameters which are provided when USE is executed. For more information on script parameters please refer to the .
The uri statement is used to encode the contents of a variable such that it does not contain any illegal or ambiguous characters when used in an request.
uri encodevarname
uri component-encodevarname
uri aws-object-encodevarname
As well as uri component-encode you can use uri encode-component (the two are identical in operation). Similarly, uri aws-object-encode and aws-encode-object are aliases for each other.
When sending a request to an HTTP server it is necessary to encode certain characters such that the server can accurately determine their meaning in context. The encoding involves replacing those characters with a percent symbol - % - followed by two hexadecimal digits representing the ASCII value of that character.
Note that the last parameter to the uri statement is a variable name, so to encode the contents of a variable called my_query the correct statement would be uri encode my_query and not uri encode ${my_query} (The latter would only be correct if the value of my_query was the name of the actual variable to encode)
USE script provides the following methods for encoding the contents of a variable:
uri encodevarname
This method will encode all characters except for the following:
This is typically used to encode a URI which contains spaces (spaces encode to %20) but doesn't contain any query parameters.
uri encode-componentvarname
This method will encode all characters except for the following:
This is typically used to encode query components of a URI, such as usernames and other parameters. Note that this method will encode the symbols =, & and ? and as such a URL of the form:
server.com/resource?name=name_value&domain=domain_value
is usually constructed from its various components using the values of the parameters as shown in the example below.
uri aws-object-encodevarname
This method is specifically implemented to support the encoding of object names when downloading from Amazon S3 buckets. Amazon S3 buckets appear much like shared directories, but they do not have a heirarchical filesystem.
The 'files' in buckets are termed objects and to assist in organising the contents of a bucket, object prefixes may be used to logically group objects together.
These prefixes may include the forward slash character, making the resulting object name appear identical to a conventional pathname (an example might be billing_data/20180116_usage.csv). When downloading an object from S3 the object name is provided as part of the HTTP query string.
When referencing an S3 object name there is an explicit requirement not to encode any forward slashes in the object name. USE therefore provides the aws-object-encode method to ensure that any S3 object names are correctly encoded. This method will encode all characters except for the following:
URI encode every byte. UriEncode() must enforce the following rules:
URI encode every byte except the unreserved characters: 'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '', and '~'._
The space character is a reserved character and must be encoded as "%20" (and not as "+").
Each URI encoded byte is formed by a '%' and the two-digit hexadecimal value of the byte.
Letters in the hexadecimal value must be uppercase, for example "%1A".
Encode the forward slash character, '/', everywhere except in the object key name. For example, if the object key name is photos/Jan/sample.jpg, the forward slash in the key name is not encoded.
The usr-object-encode method is compliant with the above requirements. For most trivial cases it should not be necessary to encode the AWS object name as it is relatively straightforward to do it by hand. However using uri aws-object-encode to URI-encode the object name may be useful for object names that contain a number of characters not listed above or for cases where the object name is provided as a parameter to the USE script.
The above script will output:
The unzip statement is used to unzip the data in a .
unzip{buffer_name}
The unzip statement will extract a single file from a zip archive stored in a named buffer. In order for this to succeed, the buffer must have been previously populated using the statement, and the data within the buffer must be a valid ZIP file.
Only ZIP files are supported. To extract GZIP files, use
A warning will be logged, the buffer left intact and the script will continue to execute if any of the following conditions arise:
The buffer is empty or does not contain a valid ZIP archive
The ZIP archive is damaged or otherwise corrupted
More than 1 file is present within the archive
After the unzip statement completes, the buffer will contain the unzipped data (the original ZIP archive is discarded during this process).
The filename of the unpacked file is also discarded, as the resulting data is stored in the buffer and can subsequently be saved using an explicit filename as shown in the example below.
Persistent. Specifies whether any authentication configured using the http_authtype setting should be performed against a proxy or the hostname specified in the URL.
Headers configured using set http_header will be used for all subsequent HTTP connections. If a different set of headers is required during the course of a USE script then the statement can be used to remove all the configured headers, after which set http_header can be used to set up the new values.
By default, no headers at all will be included with requests made by the statement. For some cases this is acceptable, but often one or more headers need to be set in order for a request to be successful.
Transient. If set, any response returned by the server after the next HTTP request will be saved to the specified filename. This can be used in conjunction with the statement, in which case the response will both be cached in the named buffer and saved to disk.
Persistent. Sets the ODBC connection string for use by the statement's odbc_direct protocol. The connection string may reference an ODBC DSN or contain full connection details, in which case a DSN doesn't need to be created.
More information may be found at where it states:
Level
Label
Meaning
0
DEBUGX
Extended debugging information
1
DEBUG
Debugging information
2
INFO
Standard informational messages
3
WARN
Warnings and non-fatal errors
4
ERROR
Run-time errors
5
FATAL
Non-recoverable errors
Parameter
Value
label
A unique name to associate with this match
expression
The regular expression to apply to the target
target
The data to search using the expression
Variable
Possible values
Notes
label.STATUS
MATCH
NOMATCH
ERROR
The result of applying the expression (ERROR infers an invalid expression)
label.RESULT
(A string) (Empty value)
The text matched by the subgroup in the expression, if any
Value | Meaning |
| no authentication is required or should be used |
| use basic authentication |
| use NTLM authentication |
| use passport authentication |
| use digest authentication |
| automatically selects between NTLM and Kerberos authentication |
Type | Meaning |
Persistent | The setting remains active indefinitely and will be re-used over successive HTTP calls |
Transient | The setting only applies to a single HTTP call, after which it is automatically reset |
The terminate statement will exit the USE script immediately.
terminate [with error]
Normally a USE script will finish execution when an error is encountered or when the end of the script file is reached, whichever comes first.
When the terminate statement is encountered, the script will finish at that point. No statements after the terminate statement will be executed.
By default, the script will exit with a success status, however it may be useful to exit deliberately when an error such as an invalid or unexpected response from an HTTP session is detected. Adding the keywords with error to the statement will cause it to exit with an error status.
The var statement is used to create or update a variable which can subsequently be referenced by name in the USE script.
[public] varname [ = value]
[public] varname operator number
[public] encrypt varname = value
For details on encrypted variables please refer to the encrypt article
Variables are created in one of two ways:
Manually via the var command
Automatically, as a consequence of other statements in the script
If the word public precedes a variable declaration then the variable will be shown in, and its value can be updated from, the Exivity GUI. Only variables prefixed with the word public appear in the GUI (all others are only visible in the script itself). To make an automatic variable public, re-declare it with a value of itself as shown below:
A variable is a named value. Once defined, the name can be used in place of the value for the rest of the script. Amongst other things this permits configuration of various parameters at the top of a script, making configuration changes easier.
The = value portion of the statement is optional, but if used there must be white-space on each side of the = character. To use spaces in a variable value it should be quoted with double quotes.
Once a variable has been defined it can be referenced by prefixing its name with ${ and post-fixing it with a }. For example a variable called outputFile can be referenced using ${outputFile}. If no value is specified, then the variable will be empty, eg:
will result in the output:
Variable names are case sensitive, therefore ${variableName} and ${VariableName} are different variables.
If there is already a variable called name then the var statement will update the value.
There is no limit to the number of variables that can be created, but any given variable may not have a value longer than 8095 characters
Variables that contain a numeric value can have the arithmetic operations performed on them. This is done using the following syntax:
varname operator number
The operator must be surrounded by white-space and following values are supported:
For example the statement var x += 10 will add 10 to the value of x.
When performing arithmetic operations on a variable, any leading zeros in the value of that variable will be respected:
Attempting to perform an arithmetic operation on a variable that does not contain a valid number will result in an error being logged, and the script will terminate.
Currently, only integer arithmetic is supported.
Automatic variables are referenced in exactly the same way as manually created ones; the only difference is in the manner of creation.
The following variables are automatically created during the execution of a USE script:
To derive the short versions of the day and month names, use a match statement to extract the first 3 characters as follows:
match day "(...)" ${DAY_NAME_UTC}
var short_day = ${day.RESULT}
On occasion it may be useful to determine the length (in characters) of the value of a variable. This can be done by appending the suffix .LENGTH to the variable name when referencing it. For example if a variable called result has a value of success then ${result.LENGTH} will be replaced with 7 (this being the number of characters in the word 'success').
A variable with no value will have a length of 0, therefore using the .LENGTH suffix can also be used to check for empty variables as follows:
myvar.LENGTH is not a variable in its own right. The .LENGTH suffix merely modifies the manner in which the myvar variable is used.
Operator
Meaning
+=
Addition
-=
Subtraction
*=
Multiplication
/=
Division
Variable
Details
${ARGC}
The number of parameters passed to the script
${ARG_N}
For each parameter passed to the script a variable called ${ARG_N}, where N is a number greater than or equal to 1, will be created whose value is the argument value associated with that parameter
${DAY}
The day of the current local date, padded to 2 digits if necessary
${DAY_NAME}
The full English name of the current day of the week
${DAY_UTC}
The day of the current date in UTC, padded to 2 digits if necessary
${DAY_NAME_UTC}
The full English name of the current day o fthe week in UTC
${GET_TIME}
The current local time in 'friendly' format, eg Tue Jan 16 14:04:32 2018
${loop_label.COUNT}
A foreach loop creates this variable (where loop_name is the name of the loop). The value of the variable is updated every time the loop executes, with a value of 1 on the first loop. If no loops are performed, then the variable will have a value of 0
${loop_label.NAME}
${loop_label.VALUE}
When iterating over the children of a JSON object (not an array) using foreach, these variables are updated with the name and value respectively of the current child every time the loop is executed (either may be blank if the child has no name or value respectively)
${loop_label.TYPE}
When iterating over the children of a JSON object (not an array) using foreach, this variable is updated to reflect the type of the current child every time the loop is executed. The type will be one of boolean, number, string, array, object or null.
${HOUR}
The hour of the current local time, padded to 2 digits if necessary
${HOUR_UTC}
The hour of the current time in UTC, padded to 2 digits if necessary
${HTTP_STATUS_CODE}
The HTTP status code returned by the server in response to the most recent http request executed
${MINUTE}
The minute of the current local time, padded to 2 digits if necessary
${MINUTE_UTC}
The minute of the current time in UTC, padded to 2 digits if necessary
${MONTH}
The month of the current local date, padded to 2 digits if necessary
${MONTH_NAME}
The full English name of the current month of the year
${MONTH_UTC}
The month of the current date in UTC, padded to 2 digits if necessary
${MONTH_NAME_UTC}
The full English name of the current month of the year in UTC
${NEWLINE}
A newline (0x0A) character. Example use:
var twolines = "This string${NEWLINE}contains two lines of text"
${SECOND}
The second of the current local time, padded to 2 digits if necessary
${SECOND_UTC}
The second of the current time in UTC, padded to 2 digits if necessary
${MSEC}
The milliseconds of the current local time, padded to 3 digits if necessary
${MSEC_UTC}
The milliseconds of the current time in UTC, padded to 3 digits if necessary
${SCRIPTNAME}
The filename of the script being executed
${OSI_TIME_UTC}
The current UTC time in YYYYMMDD'T'HHMMSS'Z' format, eg: 20180116T140432Z
${YEAR}
The year of the current local date as a 4 digit number
${YEAR_UTC}
The year of the current date in UTC as a 4 digit number