Troubleshoot Data Stream

This page lists the symptoms of a stream in Data Stream, each with its cause and its fix. Delivery symptoms come first, read from your endpoint and from Real-Time Events. Symptoms in the log lines, in API responses, and in Azion Console follow. In the Console, the endpoint field is labeled Connector.

A stream is active but nothing reaches the endpoint

Your endpoint receives no log line, while the stream list shows the stream with the Status Active.

Either the stream has not sent anything yet, or it sends and the endpoint does not accept the batches. Real-Time Events tells the two cases apart, because it records every send, delivered or not.

To find the case that applies:

1. In Real-Time Events, open the Data Stream data source.
2. Find the records whose url is your endpoint. Each record is one send, with its statusCode and its streamedLines.
3. When records exist, read statusCode. A 200 is a delivered batch. For 503, 504, or another status, follow the entry for it below.
4. When no record exists, wait. A change of active state takes one to two minutes, and delivery takes up to 3 minutes.
5. Check that the Status column still reads Active. A stream with sampling saved after yours deactivates it, as Another stream stopped sending after you saved one explains.
6. Check that the data source produced an event. An Activity History stream sends only after an action in the account, such as an edit in Azion Console.

For an Object Storage bucket, list the objects of the bucket as well:

Each object holds one batch, named after the Object Key Prefix, the time, and a unique ID:

To watch the volume of your streams over time, read the Data Stream tab of the Real-Time Metrics Observe dashboards. The GraphQL API serves the same data and the raw records.

A record with statusCode 200 and an object under the prefix together confirm that the endpoint receives the stream.

Real-Time Events shows status 503 and nothing reaches the endpoint

The records of the stream carry statusCode 503, and no log line reaches the endpoint.

Data Stream checks each endpoint once a minute and discards the batches of an endpoint it marks unavailable. One Azion server that reports the endpoint unavailable is enough, and which server reported it cannot be known.

• Check the endpoint: confirm that the service at the url of the record is running and reachable.
• Give an Object Storage credential the bucket capabilities: the credential needs listAllBucketNames, listBuckets, listFiles, and writeFiles. Without the first two, every send is recorded with 503. Create a credential with the four, and set its keys in Access Key and Secret Key. For the fields, refer to Azion Object Storage.
• Recover the missed interval elsewhere: the lines of a 503 send are never delivered later. Query that interval in Real-Time Events, which keeps the raw events for 7 days.

Once the edit takes effect, the sends are recorded with statusCode 200, and their batches reach the endpoint.

Real-Time Events shows status 504

The records of an HTTP POST endpoint carry statusCode 504.

The endpoint passed the availability check, but did not receive the batch within the send timeout of 20 seconds.

• Check the response time of the endpoint: it must receive each batch within 20 seconds, per Data Stream limits.
• Locate the slow send: url names the endpoint, and streamedLines and dataStreamed give the size of the batch.

Sends that the endpoint receives within 20 seconds are recorded with statusCode 200.

Real-Time Events shows the endpoint’s own error status, such as 405

The records of the stream carry a status other than 200, 503, or 504, such as 405.

The endpoint received the batch and refused it, and the record keeps the status the endpoint returned. For example, a URL that accepts no POST answers 405 to every send.

• Fix the receiving side: look up the status in the documentation or the logs of your endpoint. Then correct the URL, the credential, or what the endpoint accepts.
• Check where the batch went: the url field of the record shows the destination the stream used.
• Expect a gap: Data Stream does not send the lines of a refused batch again.

Once the endpoint accepts the batches, the records of the next sends show statusCode 200.

Another stream stopped sending after you saved one

After you save a stream, another stream of the account shows the Status Inactive and sends nothing.

Saving an active stream with sampling, at any rate including 100, deactivates every other stream on the account. The API returns no error, and the Console warns before it saves.

• Use a workload filter on streams that run together: in Transform, select Option › Filter Workloads and pick the workloads. A stream without sampling leaves the other streams active. For the steps, refer to Associate workloads with a stream.
• Reactivate the stopped stream: turn on Active in its Status section, and select Save. For the steps, refer to Edit, stop, or delete a stream.
• Keep one sampled stream per account: an Activity History stream uses sampling, so saving it active stops the others.

Each filtered stream shows Active in the list once saved, and its sends appear in Real-Time Events within one to two minutes.

The endpoint receives fewer log lines than requests

An Applications stream delivers fewer log lines than the requests your workloads served.

Each event the stream collects becomes one log line, so the missing lines are events outside its scope. Two settings narrow the scope: a sampling rate below 100, and a workload filter that leaves a workload out.

• Raise the sampling rate: set Sampling Rate (%) to 100 to collect every event. The Console states that sampling is statistical and not absolutely precise. It also states: When multiple Data Streams have different sampling rates, the system uses the lowest percentage.
• Add the missing workload to the filter: the stream skips a workload created later until you add it. All Current and Future Workloads covers later workloads, but it uses sampling. For its effect on other streams, refer to Another stream stopped sending after you saved one.

Sends recorded with 503 or with an endpoint error also lose their lines, as the entries above explain. At a rate of 100 over every workload in scope, the stream sends one log line per request.

Batches arrive every minute with only a few lines

The endpoint receives a batch about once a minute, with one or two log lines in each.

A batch closes at 2,000 log lines or after 60 seconds, whichever comes first. A quiet stream reaches 60 seconds first, so it sends the lines it holds.

• Read small batches as low traffic: this is the expected behavior, not an error.
• Expect no setting to change it: no field changes the 2,000-line count or the 60-second interval. On a Standard HTTP/HTTPS POST endpoint, Payload Max Size only closes a batch earlier. The bounds are in Data Stream limits.

As the traffic grows, the batches fill toward 2,000 log lines and leave before the 60 seconds pass.

Some fields of a log line show a dash

Some keys of the delivered log lines hold - instead of a value.

The variable behind the key has no value for that event. Four cases account for it.

• Read upstream fields of cached responses as empty: on a cache hit, $upstream_status, $proxy_status, and the upstream timings hold -. For the full list, refer to Values served from cache.
• Turn on Debug Rules for the rules a request ran: no preset carries $traceback. It needs a custom template and Debug Rules on the application, as Rules a request ran explains.
• Expect headers only on blocked requests: $headers and $waf_headers hold the request headers only when WAF blocked the request. The WAF Event Collector preset always sends - under its headers key. For the variables, refer to WAF Events.
• Read $truncated_body as empty: the variable is deprecated and always holds -.

Each remaining key holds the value of its variable for the event.

The API refuses a stream with 400

A POST to /v4/workspace/stream/streams returns 400 with an errors array, and the API creates nothing. The code and the source.pointer of each error name the cause.

• 400 32002 Workloads Must Be Provided: transform has neither a sampling nor a filter_workloads item. Add one of them.
• 400 32007 Sampling And Workloads Are Exclusive: transform has both. Keep one of them.
• 400 32008 Template Must Be Provided: transform has no render_template item. Add one with a template ID.
• 400 10059 Required Field at /data/outputs/0/headers: a standard endpoint has no headers. Send {} for none.

Every code, with its cause and its fix, is in Stream settings. With the field corrected, the API answers 201 and returns the stream.

Only one endpoint is kept after you save a stream

You sent two entries in outputs, the API answered 201, and the stream holds only the first one.

A stream sends to one endpoint. The API drops a second outputs entry without an error.

• Create one stream per endpoint: give the second stream the same data source and template, and the other endpoint.
• Keep both streams active: give each one a workload filter, not sampling, or the second save stops the first.

Each stream then shows its own endpoint in the Connector column, and its own sends in Real-Time Events.

A wrong endpoint credential shows only after the stream saves

The stream saved without an error, but Real-Time Events records its sends with 503 or an endpoint error.

Saving checks the format of the fields, not the endpoint. The API accepts a wrong credential or an unreachable URL, and the problem shows at the first send.

• Read the first records after each save: the save response confirms the format only. Real-Time Events shows whether the endpoint accepts the credential.
• Correct the credential and save the stream: for the fields each endpoint takes, refer to Endpoints. For an Azion Object Storage bucket, see Real-Time Events shows status 503 and nothing reaches the endpoint.
• Read a failed save as a format error: it returns 400 with a code. For the common codes, refer to The API refuses a stream with 400.

Once the change takes effect, the sends of the stream are recorded with statusCode 200.

You cannot create a stream or change its fields

+ Stream is disabled, the fields of a stream cannot be changed, or a template’s Data Set is read-only.

The account lacks the edit permission or holds too many workloads, or the template is a preset. A data source can also depend on a product the account does not have.

• Ask for the edit permission: View Data Stream shows streams only. Creating, editing, and deleting need Edit Data Stream. For how permissions are granted, refer to Teams and permissions.
• Use the API on large accounts: at 3,000 workloads or more, the Console blocks the stream forms. For the bound, refer to Data Stream limits.
• Duplicate a preset to change its variables: Duplicate Template opens the Create Custom Template drawer with the preset’s data set. For custom templates, refer to Custom templates.
• Activate the product behind the data source: Functions needs Functions, and WAF Events needs Firewall with WAF.

With the permission and the products in place, the form accepts your changes, and Save stores them.

Related resources