Send logs to Amazon S3

Create a stream that writes the logs of your applications as objects in an Amazon S3 bucket, in Azion Console or with the Azion API, and confirm the delivery.

Last updated

View as Markdown Agent setup

You can send the logs of a stream to an Amazon S3 bucket from Azion Console or with the Azion API. To send them to an Azion Object Storage bucket, whose credential needs other capabilities, refer to Send Data Stream data to Object Storage.

Data Stream writes each batch of log lines as one object in the bucket. In the stream form, the endpoint is set in the field labeled Connector, and Amazon S3 is its Simple Storage Service (S3) option. For every field and its bounds, refer to Endpoints.

The example collects the requests of one workload with the Applications data source.

Select your interface once. The prerequisites and every task below show only that path.

Prerequisites

  • An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
  • A workload on the account that receives requests.
  • An Amazon S3 bucket. The bucket must exist before the stream sends to it. It can use server-side encryption with Amazon S3 managed keys (SSE-S3).
  • The region code of the bucket, such as us-east-1.
  • An AWS Identity and Access Management (IAM) credential with an access key ID and a secret access key. The credential needs the s3:ListBucket permission, to list the objects of the bucket, and the s3:PutObject permission, to write objects in it.

Create the stream

The stream collects from the workload you choose, through a workload filter. Unlike sampling, a workload filter leaves your other streams active.

To create the stream with the API, send a POST request to https://api.azion.com/v4/workspace/stream/streams. Replace [TOKEN VALUE] with your personal token, <workload-id> with the ID of your workload, and the bucket values with your own:

Shell
01curl -X POST 'https://api.azion.com/v4/workspace/stream/streams' \
02 -H 'Content-Type: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]' \
04 -d {
05 "name": "logs-to-s3",
06 "active": true,
07 "inputs": [
08 { "type": "raw_logs", "attributes": { "data_source": "workloads" } }
09 ],
10 "transform": [
11 { "type": "filter_workloads", "attributes": { "workloads": [<workload-id>] } },
12 { "type": "render_template", "attributes": { "template": 2 } }
13 ],
14 "outputs": [
15 {
16 "type": "s3",
17 "attributes": {
18 "host_url": "https://s3.us-east-1.amazonaws.com",
19 "bucket_name": "<your-bucket>",
20 "region": "us-east-1",
21 "access_key": "[ACCESS KEY]",
22 "secret_key": "[SECRET KEY]",
23 "object_key_prefix": "azion/logs",
24 "content_type": "application/gzip"
25 }
26 }
27 ]
28 }

The workloads data source is Applications in the Console, and template 2 is Applications Event Collector. The API answers 201 with the stored stream:

JSON
01{
02 "state": "executed",
03 "data": {
04 "id": 12349,
05 "name": "logs-to-s3",
06 "last_editor": "user@example.com",
07 "created": "2026-01-01T12:10:26.000000Z",
08 "last_modified": "2026-01-01T12:10:26.000000Z",
09 "product_version": "1.0",
10 …
11 "outputs": [
12 {
13 "type": "s3",
14 "attributes": {
15 "access_key": "[ACCESS KEY]",
16 "secret_key": "[SECRET KEY]",
17 "region": "us-east-1",
18 "object_key_prefix": "azion/logs",
19 "bucket_name": "<your-bucket>",
20 "content_type": "application/gzip",
21 "host_url": "https://s3.us-east-1.amazonaws.com"
22 }
23 }
24 ]
25 }
26}

Keep the id: it identifies the stream in every later request, such as /v4/workspace/stream/streams/12349. For every key of the body, refer to Stream settings.

To create the stream in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Select + Stream
Name the stream

In the General section, enter a Name. For example: logs-to-s3.

Select the data source

In the Input section, select Applications in Data Source.

Turn off Sampling

In the Transform section, turn off Sampling while Option is still All Current and Future Workloads, its starting value. A stream cannot carry sampling and a workload filter together.

Choose the workload

In the Transform section, set Option to Filter Workloads. In Available Workload, select your workload and move it to Chosen Workload with the arrow.

Select the template

In the Render Template section, select Applications Event Collector in Template.

Select the S3 endpoint

In the Output section, select Simple Storage Service (S3) in Connector.

Enter the bucket location

Enter the S3 host of the bucket’s region in URL, such as https://s3.us-east-1.amazonaws.com. Enter the bucket name in Bucket Name and its region code in Region.

Enter the credential

Enter the access key ID in Access Key and the secret access key in Secret Key. The Console masks both fields.

(Optional) Enter the object prefix

In Object Key Prefix, enter the start of each object name. For example: azion/logs.

Select the content type

In Content Type, select plain/text or application/gzip. With plain/text, each object holds one log line per line.

Keep the stream active

In the Status section, keep Active turned on.

Select Save

The Console shows Your data stream has been created. The stream appears in the Data Stream list with Amazon S3 in the Connector column and the Active status.

The API and the Console save the stream without contacting the bucket. A wrong URL, region, or credential surfaces only when the stream sends. An activation takes effect after one to two minutes.

Confirm the delivery

Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute: a stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines.

To read the sends with the API, query the dataStreamedEvents dataset of the Real-Time Events GraphQL API. Replace the dates with a range that covers the activation of the stream:

Shell
01curl -X POST 'https://api.azion.com/v4/events/graphql' \
02 -H 'Content-Type: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]' \
04 -d '{"query":"query { dataStreamedEvents(limit: 20, filter: {tsRange: {begin: \"2026-01-01T12:00:00\", end: \"2026-01-01T12:45:00\"}}, orderBy: [ts_DESC]) { ts endpointType statusCode streamedLines dataStreamed } }"}'

The API answers 200 with one record for each send, the latest first:

JSON
01{
02 "data": {
03 "dataStreamedEvents": [
04 {
05 "ts": "2026-01-01T12:02:04Z",
06 "endpointType": "S3",
07 "statusCode": 200,
08 "streamedLines": 2,
09 "dataStreamed": 2797
10 }
11 ]
12 }
13}

A statusCode of 200 means the bucket accepted the batch, and streamedLines and dataStreamed give its size in log lines and bytes. An empty dataStreamedEvents list means the stream has not sent in the range. For every field, refer to Real-Time Events GraphQL fields.

To find the sends in Azion Console:

Open Real-Time Events

Access Azion Console > Real-Time Events.

Select the Data Stream data source
Read the latest sends

Each row is one send. Find the rows with S3 in Endpoint Type, and read their Status Code.

A Status Code of 200 means the bucket accepted the batch. Streamed Lines gives the number of log lines in the batch.

A status other than 200 is the answer of the endpoint, and 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.

In the bucket, each object name is the Object Key Prefix, a /, the date and time of the send in the YYYY/MM/DD/hh/mm/ format, and a UUID. With the prefix azion/logs, an object name starts with a path such as azion/logs/2026/01/01/12/02/, followed by the UUID.

Next steps

EndpointsEvery field of the S3 endpoint, with its type, bounds, and API name.Send Data Stream data to Object StorageSend the logs to an Azion Object Storage bucket through the same S3 endpoint.Edit, stop, or delete a streamChange the bucket or the credential, pause the stream, or remove it.Troubleshoot Data StreamFind what to change when a send returns a status other than 200.