Templates and payload
Look up the five preset templates, the custom template object and its Data Set format, and the payload settings of a Standard HTTP/HTTPS POST endpoint.
A template is the Data Stream object that decides which variables each log line carries and under which keys. Its Data Set is a JSON object: each key is a name in the log line, and each value is a variable of the stream’s data source. The Functions Event Collector preset has this data set:
A stream names one template in transform[render_template].attributes.template, as described in Stream settings. The API path is /v4/workspace/stream/templates, and one template is /v4/workspace/stream/templates/<template-id>. Every request carries the header Authorization: Token [TOKEN VALUE]. The settings in Payload belong to the Standard HTTP/HTTPS POST endpoint, not to the template.
Preset templates
Azion provides five preset templates. The API lists them with custom: false, last_editor support@azion.com, and created_at null. Each preset is named for the data source whose variables it carries, and the template object has no data source field.
| Preset | ID | Data source | Keys | Use for |
|---|---|---|---|---|
| Activity History Collector | 251 | Activity History | 20 | Account events: what changed, who changed it, and from which address. |
| Applications Event Collector | 2 | Applications | 36 | Request, response, cache, and upstream data of each request. |
| Applications + WAF Event Collector | 184 | Applications | 51 | The request data plus WAF, session, TLS, and server address variables. |
| Functions Event Collector | 86 | Functions | 8 | The messages functions log, with their level and request ID. |
| WAF Event Collector | 4 | WAF Events | 21 | Requests that WAF evaluated, with the attack family, action, and score. |
A preset’s variables are fixed. In the Console, the Data Set field of the Render Template section is read-only, and for a preset, Duplicate Template opens the Create Custom Template drawer filled with the preset’s data set. To send other variables, create a custom template. Each variable is described on Data sources and variables.
Activity History Collector
The Activity History Collector preset, ID 251, uses each variable name as its key: $comment, $user_ip, $request_data, $resource_name, $user_id, $account_id, $referer_header, $title, $author_email, $parent_resource_id, $author_name, $resource_id, $parent_resource_name, $client, $user_agent, $parent_resource_type, $time, $type, $remote_port, and $resource_type.
An S3 endpoint with Content Type plain/text receives this preset’s log lines as one JSON object per line. One line, trimmed:
Applications Event Collector
The Applications Event Collector preset, ID 2, uses each variable name as its key: $http_user_agent, $ssl_protocol, $server_protocol, $waf_attack_action, $upstream_cache_status, $request_time, $upstream_status, $state, $version, $request_method, $ssl_cipher, $scheme, $tcpinfo_rtt, $status, $sent_http_x_original_image_size, $request_length, $sent_http_content_type, $time, $requestQuery, $host, $http_referrer, $upstream_local_addr, $configuration, $request_uri, $proxy_status, $requestPath, $country, $remote_addr, $bytes_sent, $upstream_header_time, $upstream_bytes_received, $client, $upstream_response_time, $remote_port, $upstream_connect_time, and $waf_attack_family.
Applications + WAF Event Collector
The Applications + WAF Event Collector preset, ID 184, carries every variable of the Applications Event Collector except $upstream_local_addr and $version. It adds 17 variables: $session_id, $stream, $upstream_addr, $upstream_bytes_sent, $server_port, $server_addr, $waf_learning, $waf_block, $waf_total_processed, $waf_total_blocked, $waf_score, $waf_match, $waf_headers, $asn, $ssl_session_reused, $ssl_server_name, and $request_id. Its keys match the variable names, except $http_referrer, which it sends under the key http_referer.
Functions Event Collector
The Functions Event Collector preset, ID 86, has eight keys. Six of them differ from the variable name: configuration carries $global_id, edgeFunctionID carries $edge_function_id, requestID carries $request_id, messageSource carries $message_source, logLevel carries $log_level, and logMessage carries $log_message. The keys time and client carry $time and $client.
WAF Event Collector
The WAF Event Collector preset, ID 4, uses each variable name as its key: $version, $time, $client, $configuration, $host, $remote_addr, $server_protocol, $country, $waf_server, $waf_uri, $waf_learning, $blocked, $waf_score, $waf_match, $waf_attack_family, $waf_attack_action, $truncated_body, $waf_args, $requestPath, and $requestQuery. Its 21st key, headers, holds the literal - instead of the $headers variable. To send $headers, use a custom template.
Custom templates
A custom template is a template the account creates, and the API lists it with custom: true in the same list as the presets. A stream that uses a custom template sends only the variables in its data set. In the Console, the Template dropdown groups custom templates under Custom Templates and presets under Azion’s Templates.
| Console | API field | Type | Required | Default | Values |
|---|---|---|---|---|---|
| Name | name | string | Yes | none | 1 to 100 characters. |
| Data Set | data_set | string | Yes | none | 1 to 65,535 characters. A JSON object whose entries are "<key>": "$<variable>", sent as a string with its quotes escaped. |
| none | active | boolean | No | true | true or false. |
The API adds the read-only fields id, custom, last_editor, created_at, and last_modified. A custom template in the template list reads like this one, which copies the data set of the Functions Event Collector:
A POST to /v4/workspace/stream/templates creates a custom template from name and data_set. On /v4/workspace/stream/templates/<template-id>, a GET reads the template, a PATCH changes only the keys you send, a PUT takes the same required keys as a create, and a DELETE removes it.
The key of each entry is yours to choose, and the value is the variable. This data set keeps ten Applications variables under their own names, two of which, $session_id and $server_port, only the Applications + WAF Event Collector preset carries:
A custom template can also carry a variable that no preset carries. The data set {"time": "$time", "traceback": "$traceback"} sends the rules that ran on each request of an application. A Functions data set can keep the variable names as keys, such as {"time": "$time", "global_id": "$global_id", "edge_function_id": "$edge_function_id", "request_id": "$request_id", "log_level": "$log_level", "log_message": "$log_message"}.
Templates have no Console page of their own. In the Render Template section of a stream, Create Custom Template opens a drawer with the sections General, holding Name, and Data Set, a JSON editor. For a custom template, Edit Template opens the Edit Custom Template drawer, where Danger area holds Delete template. For the steps, refer to Create a custom template.
Payload
The payload settings decide how a Standard HTTP/HTTPS POST endpoint receives the log lines of each request. They are fields of the endpoint in outputs[0].attributes, not of the template, and no other endpoint type has them. The Console labels the endpoint field Connector and shows these settings in the Output section when Connector is Standard HTTP/HTTPS POST.
| Console | API field | Type | Required | Default | Values |
|---|---|---|---|---|---|
| Payload Format | outputs[0].attributes.payload_format | string | No in the API, Yes in the Console | $dataset | 1 to 250 characters. $dataset is replaced by the log lines of the request, joined by the separator. Text around $dataset is sent as written. |
| Payload Log Line Separator | outputs[0].attributes.log_line_separator | string | No in the API, Yes in the Console | \n | 1 to 100 characters. The string between two log lines. \n puts each log line on its own line. |
| Payload Max Size | outputs[0].attributes.max_size | integer, bytes | No | null in the API, 1000000 in the Console | 1,000,000 to 2,147,483,647. The maximum size of a data packet. A value below 1000000 is refused with 10050 Min Value. |
A Standard HTTP/HTTPS POST endpoint receives a batch when it reaches 2,000 log lines, 60 seconds, or the Payload Max Size, whichever comes first. For how batches form, refer to How Data Stream works.
With the defaults, the endpoint receives NDJSON: one log line per line, with no enclosing brackets and no comma between lines. NDJSON suits a receiver that processes one record at a time. A JSON array, with brackets and commas, is read as a single record. Both examples below use this data set:
With Payload Log Line Separator \n and Payload Format $dataset, the request body is NDJSON:
With Payload Log Line Separator , and Payload Format [$dataset], the request body is one JSON array:
To customize the payload of an Activity History stream that sends to a Standard HTTP/HTTPS POST endpoint, Azion specifies Payload Log Line Separator \n and this Payload Format:
This format names $time_iso8601 and $clientid, which are not in the Activity History variables. That data source names the time $time and the client $client.
The payload settings have no compression option. Among the endpoint types, only Simple Storage Service (S3) offers a compressed Content Type, application/gzip. For every endpoint field, including Custom Headers, refer to Endpoints.