# Stream settings A stream is the [Data Stream](/en/documentation/platform/data-stream/) object that collects the logs of one data source, shapes each log line with a template, and sends the lines to one endpoint. Azion Console and the Azion API write the same object, so each table on this page names the Console label beside the API field. API fields are written as paths inside the request body, such as `inputs[0].attributes.data_source`. --- ## Stream object The JSON body below is a complete stream. It sends every Activity History event, shaped by the *Activity History Collector* template, to a bucket over the S3 protocol: ```json { "name": "activity-to-bucket", "active": true, "inputs": [ { "type": "raw_logs", "attributes": { "data_source": "activity_history" } } ], "transform": [ { "type": "sampling", "attributes": { "rate": 100 } }, { "type": "render_template", "attributes": { "template": 251 } } ], "outputs": [ { "type": "s3", "attributes": { "host_url": "https://s3.us-east-005.azionstorage.net", "bucket_name": "", "region": "us-east-005", "access_key": "[ACCESS KEY]", "secret_key": "[SECRET KEY]", "object_key_prefix": "activity", "content_type": "plain/text" } } ] } ``` The three arrays carry the parts of the stream: - `inputs` holds one data source. - `transform` holds one `render_template` item and exactly one of `sampling` and `filter_workloads`. Each item has a `type` and `attributes`, so `transform[sampling].attributes.rate` names the `attributes.rate` key of the item whose `type` is `sampling`. - `outputs` holds one endpoint. The API keeps the first entry and drops any other entry without an error. The API path is `/v4/workspace/stream/streams`, and one stream is `/v4/workspace/stream/streams/`. Every request carries the header `Authorization: Token [TOKEN VALUE]`, a personal token of the account. A `POST` answers `201` with `"state": "executed"` and the stored object, which adds the read-only fields `id`, `last_editor`, `created`, `last_modified`, and `product_version`. A `PATCH` changes only the keys you send: `{"active": true}` activates a stream and returns it with its data source, transforms, and endpoint unchanged. A `PUT` takes the same required keys as a create. For every operation, refer to the [Azion API reference](https://api.azion.com/v4). For the token and the first request, refer to [Azion API](/en/documentation/devtools/api/). A saved stream has no deployment step. A change of active state takes effect after one to two minutes, and during that window the previously active stream keeps sending. An account with 3,000 workloads or more manages its streams through the API only, because the Console blocks the create and edit forms at that count. --- ## General The **General** section names the stream. The API requires `name`, `inputs`, `transform`, and `outputs` on every create. | Console | API field | Type | Required | Default | Values | | -------- | --------- | ------ | -------- | ------- | ----------------------------------------------------------- | | **Name** | `name` | string | Yes | none | 1 to 100 characters. A longer name is refused with `10046`. | --- ## Input The **Input** section picks the data source the stream collects from. Each data source has its own variables, listed on [Data sources and variables](/en/documentation/platform/data-stream/data-sources-and-variables/). | Console | API field | Type | Required | Default | Values | | --------------- | ---------------------------------- | ---- | -------- | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | **Data Source** | `inputs[0].attributes.data_source` | enum | Yes | none in the API, *Applications* in the Console | `activity_history`, `workloads`, `functions_console`, or `waf`. Any other value is refused with `10039`. | | none | `inputs[0].type` | enum | Yes | none | `raw_logs`, the only value. | The Console names each data source differently from its API slug: | Console option | API slug | Variables | | ------------------ | ------------------- | ------------------------------------------------------------------------------------------------------- | | *Activity History* | `activity_history` | [Activity History](/en/documentation/platform/data-stream/data-sources-and-variables/#activity-history) | | *Applications* | `workloads` | [Applications](/en/documentation/platform/data-stream/data-sources-and-variables/#applications) | | *Functions* | `functions_console` | [Functions](/en/documentation/platform/data-stream/data-sources-and-variables/#functions) | | *WAF Events* | `waf` | [WAF Events](/en/documentation/platform/data-stream/data-sources-and-variables/#waf-events) | *Activity History* carries the account's [Activity History](/en/documentation/fundamentals/activity-history/). *Functions* needs [Functions](/en/documentation/platform/functions/) on the account, and *WAF Events* needs [Firewall](/en/documentation/platform/firewall/) with WAF. --- ## Transform The **Transform** section sets which [workloads](/en/documentation/platform/workloads/) the stream collects from and what share of their events it sends. The **Option** radio picks the scope. Each choice sends a different item: | **Option** | What the stream collects | API item | | ----------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------ | | *All Current and Future Workloads*, the default | Every workload on the account, including workloads created later. Shows **Sampling**. | `sampling` | | *Filter Workloads* | Only the workloads you move to **Chosen Workload**. | `filter_workloads` | A stream carries exactly one of the two items. Without either, the API refuses it with `32002`, whatever its data source. With both, the API refuses it with `32007`. The API sets no default for the fields below; the defaults are the Console's. | Console | API field | Console default | Values | | --------------------- | -------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Workloads** | `transform[filter_workloads].attributes.workloads` | None | The workloads in **Chosen Workload**: an array of 1 to 600 workload IDs, each a workload on the account. An ID from another account is refused with `32003`. | | **Sampling** | `transform[sampling]` | On | On sends a `sampling` object. Off sends none, and resets **Sampling Rate (%)** to `100`. | | **Sampling Rate (%)** | `transform[sampling].attributes.rate` | `100` | Integer from 1 to 100, required in a `sampling` item: the percentage of events the stream sends. `0` is refused with `10050`, and `101` with `10068`. | Activity History events belong to the account rather than to a workload, so an Activity History stream uses sampling. On some accounts, the Console reads *All Current and Future Domains* and *Filter Domains* instead. Saving an active stream with a `sampling` item deactivates every other stream on the account, and the API returns no error. A rate of `100` counts as sampling. Before saving, the Console shows `After activating and saving these settings, all other Data Streams will be disabled.` and adds `When multiple Data Streams have different sampling rates, the system uses the lowest percentage.` To run several streams at once, give each one a `filter_workloads` item instead of sampling. --- ## Render Template The **Render Template** section picks the template that turns each event into a log line. The Console groups the **Template** options under *Azion's Templates* and *Custom Templates*, and shows the variables of the selected template in the read-only **Data Set** field. | Console | API field | Type | Required | Default | Values | | ------------ | ------------------------------------------------ | -------------------- | -------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Template** | `transform[render_template].attributes.template` | integer, template ID | Yes | none in the API, the first template by name in the Console | The ID of a preset or of a custom template on the account. Without the item, the request is refused with `32008`, and an ID that does not exist with `32004`. | Azion provides five preset templates: | Preset | `template` | | ------------------------------------ | ---------- | | *Activity History Collector* | `251` | | *Applications Event Collector* | `2` | | *Applications + WAF Event Collector* | `184` | | *Functions Event Collector* | `86` | | *WAF Event Collector* | `4` | **Create Custom Template** opens a drawer that creates a custom template on the account, listed under *Custom Templates*. For the variables of each preset and the custom template format, refer to [Templates and payload](/en/documentation/platform/data-stream/templates-and-payload/#preset-templates). --- ## Output The **Output** section sets the endpoint that receives the log lines. The Console labels the field **Connector**, and the API carries the choice in `outputs[0].type`. | Console | API field | Type | Required | Default | Values | | ------------- | ----------------- | ---- | -------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------ | | **Connector** | `outputs[0].type` | enum | Yes | none in the API, *Standard HTTP/HTTPS POST* in the Console | One of the 11 values below. The endpoint's own fields go in `outputs[0].attributes`. | | Console option | API `type` | Fields | | ----------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------- | | *Standard HTTP/HTTPS POST* | `standard` | [Standard HTTP/HTTPS POST](/en/documentation/platform/data-stream/endpoints/#standard-httphttps-post) | | *Apache Kafka* | `kafka` | [Apache Kafka](/en/documentation/platform/data-stream/endpoints/#apache-kafka) | | *Simple Storage Service (S3)* | `s3` | [Simple Storage Service (S3)](/en/documentation/platform/data-stream/endpoints/#simple-storage-service-s3) | | *Google BigQuery* | `big_query` | [Google BigQuery](/en/documentation/platform/data-stream/endpoints/#google-bigquery) | | *Elasticsearch* | `elasticsearch` | [Elasticsearch](/en/documentation/platform/data-stream/endpoints/#elasticsearch) | | *Splunk* | `splunk` | [Splunk](/en/documentation/platform/data-stream/endpoints/#splunk) | | *AWS Kinesis Data Firehose* | `aws_kinesis_firehose` | [AWS Kinesis Data Firehose](/en/documentation/platform/data-stream/endpoints/#aws-kinesis-data-firehose) | | *Datadog* | `datadog` | [Datadog](/en/documentation/platform/data-stream/endpoints/#datadog) | | *IBM QRadar* | `qradar` | [IBM QRadar](/en/documentation/platform/data-stream/endpoints/#ibm-qradar) | | *Azure Monitor* | `azure_monitor` | [Azure Monitor](/en/documentation/platform/data-stream/endpoints/#azure-monitor) | | *Azure Blob Storage* | `azure_blob_storage` | [Azure Blob Storage](/en/documentation/platform/data-stream/endpoints/#azure-blob-storage) | --- ## Status The **Status** section turns the stream on or off. The list of streams shows each one as *Active* or *Inactive*. | Console | API field | Type | Required | Default | Values | | ---------- | --------- | ------- | -------- | ----------------------------- | ------------------------------------------------------------------------ | | **Active** | `active` | boolean | No | `true`, and on in the Console | `false` stops the stream and keeps its settings. `true` starts it again. | A change of active state takes effect after one to two minutes. Activating a stream that has a `sampling` item deactivates every other stream on the account. --- ## Permissions Account permissions decide who can change a stream. For how permissions are granted, refer to [Teams and permissions](/en/documentation/fundamentals/teams-permissions/). **View Data Stream** shows the account's streams but does not allow creating, editing, or deleting them. Creating, editing, and deleting need **Edit Data Stream**. With **View Data Stream** only, the Console disables **+ Stream** and every form field, and the edit page shows no **Save**. Six key fields are masked, with an icon that reveals the value: S3 **Access Key**, **Secret Key**, and **Object Key Prefix**; AWS Kinesis Data Firehose **Access Key** and **Secret Key**; and Azure Monitor **Shared Key**. Without the edit permission, a lock icon replaces the reveal icon. --- ## Errors The API refuses each request below with HTTP `400`, except `10002` with `401`, and creates nothing. The `errors` array carries the code, the title, and a `source.pointer` to the field; the number in a pointer is the item's position in its array. | Code | Title | Pointer | Cause | What to do | | ------------------- | -------------------------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | `10002` | `Not Authenticated` | `source.headers` `Authorization` | The request has no token: `Authentication credentials were not provided.` | Send `Authorization: Token [TOKEN VALUE]`. | | `10046` | `Max Length` | `/data/name` | `name` is longer than 100 characters. | Shorten the name to 100 characters or fewer. | | `10039` | `Invalid Choice` | `/data/inputs/0/attributes/data_source` | `data_source` is not a known slug, such as `"nope"`. | Send `activity_history`, `workloads`, `functions_console`, or `waf`. | | `32002` | `Workloads Must Be Provided` | `/data/transform` | `transform` has neither a `sampling` nor a `filter_workloads` item: `If sampling is disabled, workloads must be provided.` | Add a `sampling` item, or a `filter_workloads` item with workload IDs. | | `32007` | `Sampling And Workloads Are Exclusive` | `/data/transform` | `transform` has both a `sampling` and a `filter_workloads` item: `If sampling is enabled, workloads must not be provided.` | Keep one of them: sampling for every workload, or the workload filter. | | `32003` | `Workloads Not Belong Account` | `/data/transform/0/attributes/workloads` | A workload ID is not on the account. `meta.invalid_workloads` lists the IDs, such as `[1]`. | Send IDs of workloads on the account. | | `10050` | `Min Value` | `/data/transform/0/attributes/rate` | `rate` is `0`. | Send a rate from 1 to 100. | | `10068` | `Max Value` | `/data/transform/0/attributes/rate` | `rate` is above `100`. | Send a rate from 1 to 100. | | `32008` | `Template Must Be Provided` | `/data/transform` | `transform` has no `render_template` item. | Add a `render_template` item with a template ID. | | `32004` | `Template Does Not Exist` | `/data/transform/1/attributes/template` | `template` is not the ID of a preset or of a custom template on the account. | Send an ID from the template list. | | `10050` | `Min Value` | `/data/outputs/0/max_size` | A `standard` endpoint has a `max_size` below `1000000`. | Send `1000000` or more, or leave `max_size` out. | | `10059` | `Required Field` | `/data/outputs/0/headers` | A `standard` endpoint has no `headers`. | Send `headers`, or `{}` for none. | | `10059` | `Required Field` | `/data/outputs/0/kafka_topic` | A `kafka` endpoint has no `kafka_topic`. | Send `kafka_topic`. | | `32006` and `10032` | `Invalid URL Scheme` and `Invalid Url` | `/data/outputs/0/url` | A `standard` endpoint has a `url` that is not an `http` or `https` URL, such as `not-a-url`. | Send a full URL that starts with `http://` or `https://`. | Two behaviors return no error. A second entry in `outputs` is dropped, and the stream keeps the first one. Saving an active stream with a `sampling` item deactivates every other stream on the account. --- ## Related resources - [Data sources and variables](/en/documentation/platform/data-stream/data-sources-and-variables.md): The variables each data source carries, with an example value for each one. - [Templates and payload](/en/documentation/platform/data-stream/templates-and-payload.md): The variables of each preset template, and how a custom template shapes a log line. - [Endpoints](/en/documentation/platform/data-stream/endpoints.md): The fields, values, and bounds of each of the 11 endpoint types. - [How Data Stream works](/en/documentation/platform/data-stream/how-it-works.md): The path an event follows from its data source to the endpoint, and how the stream batches log lines.