Data sources and variables

A data source is where the logs of a Data Stream stream come from. Each stream reads one data source, and that choice decides which variables its template can place in a log line. A variable holds one piece of information about an event and belongs to a specific data source.

Azion Console names each data source differently from its API slug. The Console sets it in the Data Source field of the Input section, and the API in inputs[0].attributes.data_source:

In a template’s Data Set, a variable is written with a leading $, such as $request_id. The In preset column of each table below names the preset templates whose Data Set carries the variable, or reads none. In the list of streams, the Source column shows the API slug rather than the Console option.

The API lists the data sources with a GET request:

Each result carries the slug you send in a stream, the API name, and active. The API name reads Workloads and Functions Console where the Console shows Applications and Functions. A data_source outside the four slugs is refused with 10039, as listed in Stream settings.

Activity History

Activity History carries the account’s Activity History: each change a user makes on the account in Azion Console. These events belong to the account rather than to a workload, so a stream on this data source uses sampling, as described in Stream settings.

Applications

Applications carries the requests made to your Applications. Its API slug is workloads, and a stream on it collects from every workload on the account or from a list of workloads, set in Stream settings.

Two presets read this data source. Applications Event Collector carries 36 variables. Applications + WAF Event Collector carries 51. It adds seven WAF variables and $asn, $request_id, $server_addr, $server_port, $session_id, $ssl_server_name, $ssl_session_reused, $stream, $upstream_addr, and $upstream_bytes_sent. It leaves out $upstream_local_addr and $version. To correlate a log line with other records of the same request, use $request_id, the unique identifier of the request, which the Functions data source also carries.

Several values in one field

Six variables can hold more than one value, one per connection the request opened: $upstream_bytes_received, $upstream_cache_status, $upstream_connect_time, $upstream_header_time, $upstream_response_time, and $upstream_status. A request opens several connections when it is redirected internally, or when Load Balancer chooses another origin. $upstream_addr lists the servers the same way. Two separators split the values:

• A comma separates the servers contacted while processing the request, such as 192.168.1.1:80, 192.168.1.2:80.
• A colon marks an internal redirect from one server group to another, started by X-Accel-Redirect or by Error Responses, such as 192.168.1.1:80, 192.168.1.2:80, unix:/tmp/sock : 192.168.10.1:80, 192.168.10.2:80.

When no server can be selected, $upstream_addr keeps the name of the server group, and $upstream_status keeps 502.

Read the values as transitions of the connection: the last value is usually the one that matters most. With Error Responses set for status 502 on an application, the upstream fields carry two values, typically 502 : 200. The first try to fetch the content from the origin returned 502, so Azion requested the URI set in Error Responses and delivered that page with 200. Every upstream field keeps the values in the same positions.

Values served from cache

When the response comes from cache, $proxy_status, $upstream_connect_time, $upstream_header_time, $upstream_response_time, and $upstream_status hold -. $upstream_connect_time holds 0 for a KeepAlive connection, and $upstream_bytes_received counts bytes only when the content is not cached.

Rules a request ran

No preset carries $traceback. To receive it, use a custom template on the Applications data source with $traceback in its Data Set, and turn on Debug Rules on the application.

The value groups the rule names by where they ran. edge_application_request holds the Request Phase rules of the application, edge_application_response holds its Response Phase rules, and edge_firewall holds the rules of the firewall. A request that passed through a firewall and an application can carry all three keys. A firewall rule that applies WAF lets the other rules of the firewall run alongside it, so the value can list more firewall rules for a request WAF blocked, and the block still applies.

Functions

Functions carries the requests your Functions handle, with the messages each function writes to the log. Its API slug is functions_console, and the data source needs Functions on the account.

The Functions Event Collector preset carries all eight variables, but places six of them under keys with other names, such as edgeFunctionID for $edge_function_id. For each key, refer to Templates and payload.

WAF Events

WAF Events carries the requests WAF analyzed, so you can map the score each request received, the WAF rules it matched, and the reason it was blocked. Its API slug is waf, and the data source needs Firewall with WAF on the account.

The WAF Event Collector preset does not carry $headers: it sends the literal - under its headers key.

Related resources