Send Data Stream data to Object Storage

Point a Data Stream stream at an Object Storage bucket through the S3 endpoint, in Azion Console or with the Azion API, and confirm the delivery.

Last updated

View as Markdown Agent setup

You can send the logs of a stream to an Object Storage bucket from Azion Console or with the Azion API. To create a stream that writes to a bucket from the start, refer to the Data Stream quickstart. To send the logs to an Amazon S3 bucket, refer to Send logs to Amazon S3.

Data Stream reaches the bucket through the S3 endpoint and writes each batch of log lines as one object. In the stream form, the endpoint is set in the field labeled Connector, and its option is Simple Storage Service (S3). For every field and its bounds, refer to Endpoints.

Select your interface once. The prerequisites and every task below show only that path.

Prerequisites

  • The Edit Data Stream permission. For the permissions, refer to Stream settings.
  • A stream on the account. To create one, refer to the Data Stream quickstart.
  • An Object Storage bucket. To create one, refer to the Object Storage quickstart.
  • An Object Storage credential that reaches the bucket, with four capabilities: List Files, Write Files, List All Bucket Names, and List Buckets. In the API, these are listFiles, writeFiles, listAllBucketNames, and listBuckets. Without the two bucket-list capabilities, every send fails with status 503, and no object reaches the bucket. The secret key shows only once, when the credential is created. To create the credential, refer to S3 protocol compatibility credentials.
  • A personal token. To create one, refer to How to manage a personal token.
  • curl.
  • The ID of the stream. The list response of GET /v4/workspace/stream/streams carries it in the id field of each stream.

Point the stream at the bucket

The stream keeps its data source, its filters, and its template. Only the endpoint changes, with the bucket’s S3 address, its region, and the key pair of the credential.

To point the stream at the bucket with the API, send a PATCH request with the outputs key only. Replace <stream-id> with the ID of your stream, <your-bucket> with the name of your bucket, and the two keys with the keys of your credential:

Shell
01curl -X PATCH 'https://api.azion.com/v4/workspace/stream/streams/<stream-id>' \
02 -H 'Content-Type: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]' \
04 -d {
05 "outputs": [
06 {
07 "type": "s3",
08 "attributes": {
09 "host_url": "https://s3.us-east-005.azionstorage.net",
10 "bucket_name": "<your-bucket>",
11 "region": "us-east-005",
12 "access_key": "[ACCESS KEY]",
13 "secret_key": "[SECRET KEY]",
14 "object_key_prefix": "activity",
15 "content_type": "plain/text"
16 }
17 }
18 ]
19 }

The API answers 200 with the stored stream:

JSON
01{
02 "state": "executed",
03 "data": {
04 "id": 12345,
05 "name": "activity-to-bucket",
06 "last_editor": "user@example.com",
07 "created": "2026-01-01T11:30:47.000000Z",
08 "last_modified": "2026-01-01T11:58:54.000000Z",
09 "product_version": "1.0",
10 …
11 "outputs": [
12 {
13 "type": "s3",
14 "attributes": {
15 "access_key": "[ACCESS KEY]",
16 "secret_key": "[SECRET KEY]",
17 "region": "us-east-005",
18 "object_key_prefix": "activity",
19 "bucket_name": "<your-bucket>",
20 "content_type": "plain/text",
21 "host_url": "https://s3.us-east-005.azionstorage.net"
22 }
23 }
24 ]
25 }
26}

The outputs entry holds the bucket, and last_modified moves forward. The data source and the transforms stay as they were. For every key of the body, refer to Stream settings.

To point the stream at the bucket in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Open the stream

In the list, select the row of the stream.

Select the S3 endpoint

In the Output section, select Simple Storage Service (S3) in Connector.

Enter the bucket location

Enter the values of your bucket:

  • URL: https://s3.us-east-005.azionstorage.net
  • Bucket Name: the name of your bucket
  • Region: us-east-005
Enter the credential

Enter the access key in Access Key and the secret key in Secret Key. The Console masks both fields. On an account with View Data Stream only, a lock icon replaces the reveal icon.

On an edit, the Console does not require Secret Key.

(Optional) Enter the object prefix

In Object Key Prefix, enter the start of each object name. For example: activity. Without a prefix, the objects go to the root of the bucket.

Select the content type

In Content Type, select plain/text or application/gzip. With plain/text, each object holds one log line per line.

Keep the stream active

In the Status section, keep Active turned on.

Select Save
Confirm the sampling warning

If the Attention dialog opens, select Confirm.

The Console shows Your data stream has been updated. In the Data Stream list, the Connector column shows Amazon S3 for the stream.

Saving checks the format of each field and does not contact the bucket. A wrong key or a missing capability shows only as a failed send. An activation takes effect after one to two minutes. If the stream uses sampling, saving it as active deactivates every other stream on the account; a workload filter leaves them active.

Confirm the delivery

A stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines. Two checks confirm the delivery: the object in the bucket, and the send that Real-Time Events records.

Find the object in the bucket

Each object name is the Object Key Prefix, a /, the date and time of the send in the YYYY/MM/DD/hh/mm/ format, and a UUID. Data Stream adds the / after the prefix.

To list the objects in the bucket with the API, send a GET request to the bucket. Replace <your-bucket> with the name of your bucket:

Shell
01curl -X GET 'https://api.azion.com/v4/workspace/storage/buckets/<your-bucket>/objects' \
02 -H 'Accept: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]'

The API answers 200 with one entry per object, each with its key, its time, and its size in bytes:

JSON
01{
02 "continuation_token": null,
03 "results": [
04 {
05 "key": "activity/2026/01/01/12/02/11111111-1111-1111-1111-111111111111",
06 "last_modified": "2026-01-01T12:02:03.000000Z",
07 "size": 2797,
08 "is_folder": false
09 },
10 …
11 ]
12}

The key starts with the prefix activity, followed by the date and time of the send, which confirms that the stream delivered a batch.

To find the object in Azion Console:

Open the bucket list

Access Azion Console > Object Storage > Buckets.

Select the bucket

Select the bucket the stream sends to.

Find the object under the prefix

With the prefix activity, the object is listed under a key that starts with activity/.

The object is in the bucket, which confirms that the stream delivered a batch.

Read the send in Real-Time Events

Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned.

To read the sends with the API, query the dataStreamedEvents dataset of the Real-Time Events GraphQL API. Replace the dates with a range that covers the change of the stream:

Shell
01curl -X POST 'https://api.azion.com/v4/events/graphql' \
02 -H 'Content-Type: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]' \
04 -d '{"query":"query { dataStreamedEvents(limit: 20, filter: {tsRange: {begin: \"2026-01-01T12:00:00\", end: \"2026-01-01T12:45:00\"}}, orderBy: [ts_DESC]) { ts endpointType statusCode streamedLines dataStreamed } }"}'

The API answers 200 with one record for each send, the latest first:

JSON
01{
02 "data": {
03 "dataStreamedEvents": [
04 {
05 "ts": "2026-01-01T12:02:04Z",
06 "endpointType": "S3",
07 "statusCode": 200,
08 "streamedLines": 2,
09 "dataStreamed": 2797
10 }
11 ]
12 }
13}

A statusCode of 200 means the bucket accepted the batch, and streamedLines and dataStreamed give its size in log lines and bytes. An empty dataStreamedEvents list means the stream has not sent in the range. For every field, refer to Real-Time Events GraphQL fields.

To find the sends in Azion Console:

Open Real-Time Events

Access Azion Console > Real-Time Events.

Select the Data Stream data source
Read the latest sends

Each row is one send. Find the rows with S3 in Endpoint Type, and read their Status Code.

A Status Code of 200 means the bucket accepted the batch. Streamed Lines gives the number of log lines in the batch.

An object under the prefix and a send with status 200 together confirm the delivery. A status of 503 means Data Stream found the endpoint unavailable. With Object Storage, check first that the credential carries listAllBucketNames and listBuckets. For the causes, refer to Troubleshoot Data Stream.

Next steps

Data Stream quickstartCreate a stream that writes to a bucket and confirm its first logs.EndpointsEvery field of the S3 endpoint, with its type, bounds, and API name.Object StorageManage the bucket, its objects, and the credentials that reach it.Troubleshoot Data StreamFind what to change when a send returns a status other than 200.