Send logs to Apache Kafka

Create a stream that sends the logs of your applications as messages to an Apache Kafka topic, in Azion Console or with the Azion API.

Last updated

View as Markdown Agent setup

You can send the logs of a stream to a topic of an Apache Kafka cluster from Azion Console or with the Azion API. To send the WAF events of a firewall to a SIEM, refer to Stream WAF events to a SIEM.

Data Stream sends the log lines as messages to one topic of the cluster. In the stream form, the endpoint is set in the field labeled Connector, and Apache Kafka is one of its options. For every field and its bounds, refer to Endpoints.

The example collects the requests of one workload with the Applications data source.

Select your interface once. The prerequisites and every task below show only that path.

Prerequisites

  • An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
  • A workload on the account that receives requests.
  • An Apache Kafka cluster and the host and port of the servers for the initial connection. You need only these servers, not every server of the cluster.
  • The name of the topic that receives the messages. The stream sends to one topic.
  • To encrypt the data with Transport Layer Security (TLS), a digital certificate on the receiving servers, issued by a trusted certificate authority (CA) such as IdenTrust, DigiCert, Sectigo, GoDaddy, GlobalSign, or Let’s Encrypt.

The Apache Kafka endpoint has no credential field: the stream connects with the server addresses, the topic, and the TLS setting.

Create the stream

The stream collects from the workload you choose, through a workload filter. Unlike sampling, a workload filter leaves your other streams active.

To create the stream with the API, send a POST request to https://api.azion.com/v4/workspace/stream/streams. Replace [TOKEN VALUE] with your personal token, <workload-id> with the ID of your workload, and the cluster values with your own:

Shell
01curl -X POST 'https://api.azion.com/v4/workspace/stream/streams' \
02 -H 'Content-Type: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]' \
04 -d {
05 "name": "logs-to-kafka",
06 "active": true,
07 "inputs": [
08 { "type": "raw_logs", "attributes": { "data_source": "workloads" } }
09 ],
10 "transform": [
11 { "type": "filter_workloads", "attributes": { "workloads": [<workload-id>] } },
12 { "type": "render_template", "attributes": { "template": 2 } }
13 ],
14 "outputs": [
15 {
16 "type": "kafka",
17 "attributes": {
18 "bootstrap_servers": "kafka1.example.com:9092,kafka2.example.com:9092",
19 "kafka_topic": "azion.logs",
20 "use_tls": true
21 }
22 }
23 ]
24 }

The workloads data source is Applications in the Console, and template 2 is Applications Event Collector. The API requires use_tls: send true to encrypt the data with TLS, or false to send it unencrypted. The API answers 201 with the stored stream:

JSON
01{
02 "state": "executed",
03 "data": {
04 "id": 12348,
05 "name": "logs-to-kafka",
06 "last_editor": "user@example.com",
07 "created": "2026-01-01T12:10:24.000000Z",
08 "last_modified": "2026-01-01T12:10:24.000000Z",
09 "product_version": "1.0",
10 …
11 "outputs": [
12 {
13 "type": "kafka",
14 "attributes": {
15 "bootstrap_servers": "kafka1.example.com:9092,kafka2.example.com:9092",
16 "kafka_topic": "azion.logs",
17 "use_tls": true
18 }
19 }
20 ]
21 }
22}

Keep the id: it identifies the stream in every later request, such as /v4/workspace/stream/streams/12348. For every key of the body, refer to Stream settings.

To create the stream in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Select + Stream
Name the stream

In the General section, enter a Name. For example: logs-to-kafka.

Select the data source

In the Input section, select Applications in Data Source.

Turn off Sampling

In the Transform section, turn off Sampling while Option is still All Current and Future Workloads, its starting value. A stream cannot carry sampling and a workload filter together.

Choose the workload

In the Transform section, set Option to Filter Workloads. In Available Workload, select your workload and move it to Chosen Workload with the arrow.

Select the template

In the Render Template section, select Applications Event Collector in Template.

Select the Apache Kafka endpoint

In the Output section, select Apache Kafka in Connector.

Enter the bootstrap servers

In Bootstrap Servers, enter each server as host:port, separated by a comma and no space. For example: kafka1.example.com:9092,kafka2.example.com:9092.

Enter the topic

In Kafka Topic, enter the name of one topic. For example: azion.logs.

(Optional) Turn on TLS

Turn on Enable Transport Layer Security (TLS) to send the data encrypted. The receiving servers need a certificate from a trusted CA.

Keep the stream active

In the Status section, keep Active turned on.

Select Save

The Console shows Your data stream has been created. The stream appears in the Data Stream list with Kafka in the Connector column and the Active status.

The API and the Console save the stream without contacting the cluster. A wrong server address or topic surfaces only when the stream sends. An activation takes effect after one to two minutes.

Confirm the delivery

Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute. A stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines.

To read the sends with the API, query the dataStreamedEvents dataset of the Real-Time Events GraphQL API. Replace the dates with a range that covers the activation of the stream:

Shell
01curl -X POST 'https://api.azion.com/v4/events/graphql' \
02 -H 'Content-Type: application/json' \
03 -H 'Authorization: Token [TOKEN VALUE]' \
04 -d '{"query":"query { dataStreamedEvents(limit: 20, filter: {tsRange: {begin: \"2026-01-01T12:00:00\", end: \"2026-01-01T12:45:00\"}}, orderBy: [ts_DESC]) { ts endpointType statusCode streamedLines dataStreamed } }"}'

The API answers 200 with one record for each send, the latest first:

JSON
01{
02 "data": {
03 "dataStreamedEvents": [
04 {
05 "ts": "2026-01-01T12:02:04Z",
06 "endpointType": "KAFKA",
07 "statusCode": 200,
08 "streamedLines": 2,
09 "dataStreamed": 2797
10 }
11 ]
12 }
13}

A send to Apache Kafka carries KAFKA in endpointType. A statusCode of 200 means the endpoint accepted the batch, and streamedLines and dataStreamed give its size in log lines and bytes. An empty dataStreamedEvents list means the stream has not sent in the range. For every field, refer to Real-Time Events GraphQL fields.

To find the sends in Azion Console:

Open Real-Time Events

Access Azion Console > Real-Time Events.

Select the Data Stream data source
Read the latest sends

Each row is one send. Find the rows with KAFKA in Endpoint Type, and read their Status Code.

A Status Code of 200 means the cluster accepted the batch. Streamed Lines gives the number of log lines in the batch.

A status other than 200 is the answer of the endpoint, and 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.

Next steps

EndpointsEvery field of the Apache Kafka endpoint, with its type, bounds, and API name.Best practices for Data StreamList more than one bootstrap server and turn on TLS for the cluster.Edit, stop, or delete a streamChange the servers or the topic, pause the stream, or remove it.Troubleshoot Data StreamFind what to change when a send returns a status other than 200.