--- name: azion-send-logs-to-apache-kafka description: >- Create a stream that sends the logs of your applications as messages to an Apache Kafka topic, in Azion Console or with the Azion API. --- # Send logs to Apache Kafka You can send the logs of a stream to a topic of an [Apache Kafka](https://kafka.apache.org/) cluster from Azion Console or with the Azion API. To send the WAF events of a firewall to a SIEM, refer to [Stream WAF events to a SIEM](/en/documentation/guides/application-security/firewall-and-waf/integrate-siems/). [Data Stream](/en/documentation/platform/data-stream/) sends the log lines as messages to one topic of the cluster. In the stream form, the endpoint is set in the field labeled **Connector**, and *Apache Kafka* is one of its options. For every field and its bounds, refer to [Endpoints](/en/documentation/platform/data-stream/endpoints/#apache-kafka). The example collects the requests of one workload with the *Applications* data source. --- Select your interface once. The prerequisites and every task below show only that path. ## Prerequisites - An Azion account with the **Edit Data Stream** permission. For the permissions, refer to [Stream settings](/en/documentation/platform/data-stream/stream-settings/#permissions). - A [workload](/en/documentation/platform/workloads/) on the account that receives requests. - An Apache Kafka cluster and the host and port of the servers for the initial connection. You need only these servers, not every server of the cluster. - The name of the topic that receives the messages. The stream sends to one topic. - To encrypt the data with Transport Layer Security (TLS), a digital certificate on the receiving servers, issued by a trusted certificate authority (CA) such as IdenTrust, DigiCert, Sectigo, GoDaddy, GlobalSign, or Let's Encrypt. The Apache Kafka endpoint has no credential field: the stream connects with the server addresses, the topic, and the TLS setting. **Console** - Access to Azion Console. To sign in, refer to [How to access Azion Console](/en/documentation/guides/platform/account-and-billing/how-to-access-azion-console/). **API** - A personal token. To create one, refer to [How to manage a personal token](/en/documentation/guides/platform/account-and-billing/personal-tokens/). - The ID of the workload. - `curl`. --- ## Create the stream The stream collects from the workload you choose, through a workload filter. Unlike sampling, a workload filter leaves your other streams active. **Console** To create the stream in Azion Console: 1. **Open Data Stream** Access [Azion Console](https://console.azion.com/) > **Data Stream**. 2. **Select + Stream** 3. **Name the stream** In the **General** section, enter a **Name**. For example: `logs-to-kafka`. 4. **Select the data source** In the **Input** section, select *Applications* in **Data Source**. 5. **Turn off Sampling** In the **Transform** section, turn off **Sampling** while **Option** is still *All Current and Future Workloads*, its starting value. A stream cannot carry sampling and a workload filter together. 6. **Choose the workload** In the **Transform** section, set **Option** to *Filter Workloads*. In **Available Workload**, select your workload and move it to **Chosen Workload** with the arrow. 7. **Select the template** In the **Render Template** section, select *Applications Event Collector* in **Template**. 8. **Select the Apache Kafka endpoint** In the **Output** section, select *Apache Kafka* in **Connector**. 9. **Enter the bootstrap servers** In **Bootstrap Servers**, enter each server as `host:port`, separated by a comma and no space. For example: `kafka1.example.com:9092,kafka2.example.com:9092`. 10. **Enter the topic** In **Kafka Topic**, enter the name of one topic. For example: `azion.logs`. 11. **(Optional) Turn on TLS** Turn on **Enable Transport Layer Security (TLS)** to send the data encrypted. The receiving servers need a certificate from a trusted CA. 12. **Keep the stream active** In the **Status** section, keep **Active** turned on. 13. **Select Save** The Console shows `Your data stream has been created`. The stream appears in the **Data Stream** list with `Kafka` in the **Connector** column and the **Active** status. **API** To create the stream with the API, send a `POST` request to `https://api.azion.com/v4/workspace/stream/streams`. Replace `[TOKEN VALUE]` with your personal token, `` with the ID of your workload, and the cluster values with your own: ```bash curl -X POST 'https://api.azion.com/v4/workspace/stream/streams' \ -H 'Content-Type: application/json' \ -H 'Authorization: Token [TOKEN VALUE]' \ -d '{ "name": "logs-to-kafka", "active": true, "inputs": [ { "type": "raw_logs", "attributes": { "data_source": "workloads" } } ], "transform": [ { "type": "filter_workloads", "attributes": { "workloads": [] } }, { "type": "render_template", "attributes": { "template": 2 } } ], "outputs": [ { "type": "kafka", "attributes": { "bootstrap_servers": "kafka1.example.com:9092,kafka2.example.com:9092", "kafka_topic": "azion.logs", "use_tls": true } } ] }' ``` The `workloads` data source is *Applications* in the Console, and template `2` is *Applications Event Collector*. The API requires `use_tls`: send `true` to encrypt the data with TLS, or `false` to send it unencrypted. The API answers `201` with the stored stream: ```json { "state": "executed", "data": { "id": 12348, "name": "logs-to-kafka", "last_editor": "user@example.com", "created": "2026-01-01T12:10:24.000000Z", "last_modified": "2026-01-01T12:10:24.000000Z", "product_version": "1.0", … "outputs": [ { "type": "kafka", "attributes": { "bootstrap_servers": "kafka1.example.com:9092,kafka2.example.com:9092", "kafka_topic": "azion.logs", "use_tls": true } } ] } } ``` Keep the `id`: it identifies the stream in every later request, such as `/v4/workspace/stream/streams/12348`. For every key of the body, refer to [Stream settings](/en/documentation/platform/data-stream/stream-settings/#stream-object). The API and the Console save the stream without contacting the cluster. A wrong server address or topic surfaces only when the stream sends. An activation takes effect after one to two minutes. --- ## Confirm the delivery [Real-Time Events](/en/documentation/platform/real-time-events/data-sources/#data-stream) records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute. A stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines. **Console** To find the sends in Azion Console: 1. **Open Real-Time Events** Access [Azion Console](https://console.azion.com/) > **Real-Time Events**. 2. **Select the Data Stream data source** 3. **Read the latest sends** Each row is one send. Find the rows with `KAFKA` in **Endpoint Type**, and read their **Status Code**. A **Status Code** of `200` means the cluster accepted the batch. **Streamed Lines** gives the number of log lines in the batch. **API** To read the sends with the API, query the `dataStreamedEvents` dataset of the Real-Time Events GraphQL API. Replace the dates with a range that covers the activation of the stream: ```bash curl -X POST 'https://api.azion.com/v4/events/graphql' \ -H 'Content-Type: application/json' \ -H 'Authorization: Token [TOKEN VALUE]' \ -d '{"query":"query { dataStreamedEvents(limit: 20, filter: {tsRange: {begin: \"2026-01-01T12:00:00\", end: \"2026-01-01T12:45:00\"}}, orderBy: [ts_DESC]) { ts endpointType statusCode streamedLines dataStreamed } }"}' ``` The API answers `200` with one record for each send, the latest first: ```json { "data": { "dataStreamedEvents": [ { "ts": "2026-01-01T12:02:04Z", "endpointType": "KAFKA", "statusCode": 200, "streamedLines": 2, "dataStreamed": 2797 } ] } } ``` A send to Apache Kafka carries `KAFKA` in `endpointType`. A `statusCode` of `200` means the endpoint accepted the batch, and `streamedLines` and `dataStreamed` give its size in log lines and bytes. An empty `dataStreamedEvents` list means the stream has not sent in the range. For every field, refer to [Real-Time Events GraphQL fields](/en/documentation/devtools/graphql/gql-real-time-events-fields/#datastreamedevents-data-stream). A status other than `200` is the answer of the endpoint, and `503` means Data Stream found the endpoint unavailable. For the causes, refer to [Troubleshoot Data Stream](/en/documentation/platform/data-stream/troubleshooting/). --- ## Next steps - [Endpoints](/en/documentation/platform/data-stream/endpoints.md#apache-kafka): Every field of the Apache Kafka endpoint, with its type, bounds, and API name. - [Best practices for Data Stream](/en/documentation/platform/data-stream/best-practices.md#list-more-than-one-apache-kafka-bootstrap-server): List more than one bootstrap server and turn on TLS for the cluster. - [Edit, stop, or delete a stream](/en/documentation/guides/platform/observability/delete-data-stream.md): Change the servers or the topic, pause the stream, or remove it. - [Troubleshoot Data Stream](/en/documentation/platform/data-stream/troubleshooting.md): Find what to change when a send returns a status other than 200.