Customize the HTTP POST payload
Set the payload format, log line separator, maximum size, and custom headers of a Standard HTTP/HTTPS POST endpoint, in Azion Console or with the Azion API.
You can customize the payload that a Standard HTTP/HTTPS POST endpoint receives from a stream, from Azion Console or with the Azion API. To create a stream that sends to an HTTP endpoint, with its data source and template, refer to Send logs to an HTTP endpoint.
The payload is the body of each POST request that Data Stream sends. Four fields of the endpoint shape it: Payload Format, Payload Log Line Separator, Payload Max Size, and Custom Headers. No other endpoint type has them. In the stream form, the endpoint is set in the field labeled Connector. The template decides which variables each log line carries. For every payload field and its bounds, refer to Templates and payload.
Select your interface once. The prerequisites and every task below show only that path.
Prerequisites
- An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
- A workload on the account that receives requests.
- An HTTP or HTTPS URL that accepts
POSTrequests, and the headers it requires, such as an authentication token.
- Access to Azion Console. To sign in, refer to How to access Azion Console.
- An account with fewer than 3,000 workloads. At 3,000 workloads or more, the Console blocks the stream forms, and you manage streams through the API only.
Set the payload fields
The fields are in the Output section of the stream form, on a stream you create or one you edit. With the defaults, the endpoint receives NDJSON: one log line per line, with no brackets and no comma between lines.
To set the payload fields with the API, send them in outputs[0].attributes of the create request. This example collects the requests of one workload with the Applications data source and template 2, Applications Event Collector. Replace <workload-id> with the ID of your workload and [TOKEN] with the token your endpoint expects:
A 201 carries the stream, with the payload fields as stored:
The stream exists with the payload you sent. Because it uses a workload filter, not sampling, creating it deactivates no other stream.
In JSON, "\\n" is the escape sequence \n, the separator the Console shows by default. headers is required, and {} sends no header. A request that leaves out the other fields stores $dataset, \n, and a max_size of null. To change the payload of an existing stream, send a PATCH request to /v4/workspace/stream/streams/<stream-id> with the whole outputs array.
A change to the payload of an existing stream takes a few minutes to propagate. Saving checks the format of the fields, not the endpoint, so a wrong URL or header shows up later as failed sends.
Send the log lines as one JSON array
A receiver that reads each request body as a single JSON document needs the log lines in a JSON array, with brackets around them and a comma between them. Two fields change: Payload Format wraps $dataset in brackets, and the separator becomes a comma. For the NDJSON and JSON array bodies side by side, refer to Templates and payload.
To send a JSON array with the API, set payload_format to [$dataset] and log_line_separator to , in the endpoint of the stream. The outputs item reads:
Send this item in the outputs array of a create request, or of a PATCH request to /v4/workspace/stream/streams/<stream-id>. Each request body is then one JSON array of log lines.
Confirm the delivery
Real-Time Events records every send of a stream, accepted or not, with the status code your endpoint returned. Send a few requests to the workload, then wait about a minute. A batch leaves after 60 seconds, or sooner at 2,000 log lines or at the Payload Max Size.
To read the sends with the API, query the dataStreamedEvents dataset of the Real-Time Events GraphQL API. Replace the dates with a range that covers the save of the stream:
The API answers 200 with one record for each send, the latest first:
A statusCode of 405 means the endpoint at url refused the batch, and streamedLines gives the number of log lines in it. A statusCode of 200 means your endpoint accepted the batch. An empty dataStreamedEvents list means the stream has not sent in the range. For every field, refer to Real-Time Events GraphQL fields.
A status other than 200, such as 405, is the answer of your endpoint. A 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.