Data Stream limits
Check the batching, delivery, sampling, and field bounds of Data Stream, what each one returns past its value, and the usage each plan includes.
Data Stream applies default limits to how a stream batches and delivers log lines, to the fields of a stream, and to the usage each plan includes every month. Azion can raise a default limit based on your plan. To request an increase, contact the technical support team.
Default limits
Every value in this table applies to each stream or to the account that holds it. The API answers a request past a field bound with 400 and an errors array, and creates nothing.
| Scope | Limit | Past the limit |
|---|---|---|
| Batch of log lines | 2,000 records or 60 seconds, whichever comes first. A record is one log line. For Standard HTTP/HTTPS POST, the batch also closes at Payload Max Size, in bytes. | The stream sends the batch to the endpoint. A stream with little traffic sends small batches. |
| Batch of log lines for AWS Kinesis Data Firehose | 500 records or 60 seconds, whichever comes first | The stream sends the batch to the endpoint. |
Payload Max Size, outputs[0].attributes.max_size on a standard endpoint | 1,000,000 bytes or more. The Console fills in 1000000. | A value below 1000000 is refused with 10050 Min Value. |
| Send timeout for an HTTP POST endpoint | 20 seconds | When the endpoint passes the availability check but does not receive the batch within 20 seconds, the send ends with status 504. |
| Endpoint availability check | Once a minute. Every Azion server must report the endpoint available. | While the endpoint is unavailable, Data Stream does not send to it and discards the logs of that interval. Real-Time Events records the attempt with status 503. The check runs again the next minute. |
| Endpoints per stream | 1 | A second entry in outputs is dropped without an error. The API answers 201 and keeps the first entry. |
| Active streams while one of them uses sampling | 1 | Saving an active stream with a sampling item deactivates every other stream on the account, with no API error. A rate of 100 counts as sampling. The Console warns before saving. |
Stream name | 1 to 100 characters | A longer name is refused with 10046 Max Length. |
Sampling rate, transform[sampling].attributes.rate | 1 to 100, an integer percentage | 0 is refused with 10050 Min Value, and 101 with 10068 Max Value. |
Workload filter, transform[filter_workloads].attributes.workloads | 1 to 600 workload IDs, each a workload on the account | A list outside the range is refused. An ID that is not a workload on the account is refused with 32003 Workloads Not Belong Account. |
| Workloads on the account, for Azion Console | Fewer than 3,000 | At 3,000 workloads or more, the Console blocks the create and edit forms, and the account manages its streams through the API only. |
A change of active state takes effect after one to two minutes, and during that window the previously active stream keeps sending. To run several streams at once, give each one a filter_workloads item instead of sampling. For the error codes and their fixes, refer to Stream settings.
Further bounds apply:
- Delivery of the logs to your infrastructure takes up to 3 minutes.
- On a
standardendpoint,payload_formattakes up to 250 characters,log_line_separatorup to 100, andmax_sizeup to 2,147,483,647 bytes. - The Console accepts up to five Custom Headers on a Standard HTTP/HTTPS POST endpoint.
- A template’s
data_settakes up to 65,535 characters, and itsnameup to 100. - An AWS Kinesis Data Firehose
stream_nametakes up to 64 characters. - An Apache Kafka
kafka_topicandbootstrap_servers, and an S3bucket_name, each take up to 150 characters.
For the bounds of every endpoint field, refer to Endpoints.
Included usage per plan
Data Stream is billed on two metrics: Requests and Data Transfer. Each plan includes the monthly amounts below.
| Metric | Hobby | Pro | Enterprise |
|---|---|---|---|
| Requests | 1M / month | 2M / month | Custom |
| Data Transfer | 10 GB / month | 20 GB / month | Custom |
The plan comparison is at Azion pricing. For the per-unit rates, refer to Pricing.
Real-Time Events records each send of a stream with the number of log lines in streamedLines and the size in bytes in dataStreamed. The size of a log line depends on the template and the event. To read the account’s consumption, refer to Query Data Stream usage data.