Configure sampling on a stream

Send a share of a stream's events with Data Stream sampling, in Azion Console or with the Azion API, and check which streams stay active.

Last updated

View as Markdown Agent setup

You can configure sampling on a Data Stream stream from Azion Console or with the Azion API. To limit a stream to the workloads you choose, which leaves the other streams active, refer to Associate workloads with a stream instead.

Sampling sends a percentage of the events a stream collects, picked at random. A lower rate reduces the volume, and the cost, of the data you collect and analyze. A stream with sampling covers every workload on the account, including workloads created later. For the bounds of each field, refer to Stream settings.

Select your interface once. The prerequisites and every task below show only that path.

Prerequisites

  • Access to Azion Console. To sign in, refer to Access Azion Console.
  • An account with fewer than 3,000 workloads. At 3,000 workloads or more, the Console blocks the stream forms, and you manage streams through the API only.
  • A personal token and curl.
  • The ID of the stream and the ID of its template. The list response of GET /v4/workspace/stream/streams?fields=id,name,active,transform carries both: the stream’s id, and template in the render_template item of its transform.

Set the sampling rate

The rate is a whole number from 1 to 100: the percentage of events the stream sends. A rate of 100 sends every event, and it still counts as sampling.

To set the rate with the API, send a PATCH request with the transform array of the stream. Set rate in the sampling item, and keep the render_template item with the template ID of your stream. This example keeps template 2, Applications Event Collector. Replace <stream-id> with the ID of your stream:

Shell
01curl -X PATCH 'https://api.azion.com/v4/workspace/stream/streams/<stream-id>' \
02 -H 'Content-Type: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]' \
04 -d {
05 "transform": [
06 { "type": "sampling", "attributes": { "rate": 50 } },
07 { "type": "render_template", "attributes": { "template": 2 } }
08 ]
09}

The API answers 200 with the stream, which now carries the new rate:

JSON
01{
02 "state": "executed",
03 "data": {
04 "id": 12361,
05 "name": "sampled-stream",
06 …
07 "transform": [
08 { "type": "sampling", "attributes": { "rate": 50 } },
09 { "type": "render_template", "attributes": { "template": 2 } }
10 ],
11 …
12 }
13}

The transform you send replaces the stored one, so send every item the stream keeps. A rate outside 1 to 100 is refused with 400: 0 with 10050 Min Value, and 101 with 10068 Max Value. A valid rate returns no warning that the other streams are deactivated.

To set the sampling rate in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Open the stream

In the list, select the row of the stream. The create form, opened with + Stream, has the same Transform section.

Select all workloads

In the Transform section, set Option to All Current and Future Workloads. The Sampling switch shows only with this option.

Turn on Sampling

If Sampling is off, turn it on. Sampling Rate (%) appears.

Enter the rate

In Sampling Rate (%), enter the percentage of events to send, for example 60.

Select Save
Confirm the sampling warning

The Attention dialog shows After activating and saving these settings, all other Data Streams will be disabled. Select Confirm.

The Console shows Your data stream has been updated.

Data Stream picks the sampled events at random. The Attention dialog also states Sampling percentage is statistical and not absolutely precise. When multiple Data Streams have different sampling rates, the system uses the lowest percentage.

A change of active stream takes effect after one to two minutes, and other edits take a few minutes to propagate. During that window, a stream that the save deactivated can keep sending.

Check which streams are active

After the save, the stream with sampling is the only active stream on the account. Wait two minutes, then check the status of each stream.

To check the streams with the API, list them with their active state and their transform:

Shell
01curl -X GET 'https://api.azion.com/v4/workspace/stream/streams?fields=id,name,active,transform' \
02 -H 'Authorization: Token [TOKEN VALUE]'

The API answers 200 with one entry for each stream:

JSON
01{
02 "count": 4,
03 "total_pages": 1,
04 "page": 1,
05 "page_size": 10,
06 "next": null,
07 "previous": null,
08 "results": [
09 {
10 "id": 12345,
11 "name": "activity-to-bucket",
12 "active": false,
13 "transform": [
14 { "type": "sampling", "attributes": { "rate": 100 } },
15 { "type": "render_template", "attributes": { "template": 251 } }
16 ]
17 },
18 …
19 {
20 "id": 12346,
21 "name": "sampled-stream",
22 "active": true,
23 "transform": [
24 { "type": "sampling", "attributes": { "rate": 100 } },
25 { "type": "render_template", "attributes": { "template": 251 } }
26 ]
27 },
28 …
29 ]
30}

The stream saved last with sampling is the only one with "active": true, such as 12346 in the sample above. Its sampling item carries the rate it uses.

To check the streams in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Read the Status column

The stream you saved reads Active. Every other stream reads Inactive.

Open the stream

Select the row of the stream. Sampling Rate (%) shows the stored rate.

Only the sampled stream is Active, at the rate you set.

To see what the stream sends, read its sends in Real-Time Events. Each send is recorded with the status code of the endpoint and the number of log lines it carried.

Next steps

Stream settingsLook up the bounds of the sampling rate and the workload filter, and their errors.Associate workloads with a streamLimit a stream to chosen workloads, so several streams can run at once.How Data Stream worksCompare what sampling and a workload filter collect, and what each one costs.Troubleshoot Data StreamRestore a stream that stopped sending after a sampled stream was saved.