--- name: azion-send-logs-to-elasticsearch description: >- Create a stream that sends the logs of your applications to an Elasticsearch index, in Azion Console or with the Azion API, and confirm the delivery. --- # Send logs to Elasticsearch You can send the logs of a stream to an [Elasticsearch](https://www.elastic.co/elasticsearch/) index from Azion Console or with the Azion API. [Data Stream](/en/documentation/platform/data-stream/) sends each batch of log lines to the index that ends the URL of the endpoint. In the stream form, the endpoint is set in the field labeled **Connector**, and *Elasticsearch* is one of its options. For every field and its bounds, refer to [Endpoints](/en/documentation/platform/data-stream/endpoints/#elasticsearch). The example collects the requests of one workload with the *Applications* data source. --- Select your interface once. The prerequisites and every task below show only that path. ## Prerequisites - An Azion account with the **Edit Data Stream** permission. For the permissions, refer to [Stream settings](/en/documentation/platform/data-stream/stream-settings/#permissions). - A [workload](/en/documentation/platform/workloads/) on the account that receives requests. - An Elasticsearch instance. It can run on any cloud platform, such as an Elastic Cloud deployment. - The URL of the instance followed by the name of the index, such as `https://elasticsearch.example.com/azion-logs`. On Elastic Cloud, the **Copy endpoint** action of the deployment gives the address of the instance. - An Elasticsearch API key. When Elasticsearch creates the key, it returns an `encoded` value in Base64. Data Stream takes that `encoded` value. On Elastic Cloud, you create the key in the **API console** of the deployment. **Console** - Access to Azion Console. To sign in, refer to [How to access Azion Console](/en/documentation/guides/platform/account-and-billing/how-to-access-azion-console/). **API** - A personal token. To create one, refer to [How to manage a personal token](/en/documentation/guides/platform/account-and-billing/personal-tokens/). - The ID of the workload. - `curl`. --- ## Create the stream The stream collects from the workload you choose, through a workload filter. Saving a stream with sampling deactivates your other streams, and a workload filter does not. **Console** To create the stream in Azion Console: 1. **Open Data Stream** Access [Azion Console](https://console.azion.com/) > **Data Stream**. 2. **Select + Stream** 3. **Name the stream** In the **General** section, enter a **Name**. For example: `logs-to-elasticsearch`. 4. **Select the data source** In the **Input** section, select *Applications* in **Data Source**. 5. **Turn off Sampling** In the **Transform** section, turn off **Sampling** while **Option** is still *All Current and Future Workloads*, its starting value. A stream cannot carry sampling and a workload filter together. 6. **Choose the workload** In the **Transform** section, set **Option** to *Filter Workloads*. In **Available Workload**, select your workload and move it to **Chosen Workload** with the arrow. 7. **Select the template** In the **Render Template** section, select *Applications Event Collector* in **Template**. 8. **Select the Elasticsearch endpoint** In the **Output** section, select *Elasticsearch* in **Connector**. 9. **Enter the index URL** In **URL**, enter the URL of the instance followed by the index. For example: `https://elasticsearch.example.com/azion-logs`, where `azion-logs` is the index. 10. **Enter the API key** In **Encoded API Key**, paste the Base64 `encoded` value of your Elasticsearch API key. 11. **Keep the stream active** In the **Status** section, keep **Active** turned on. 12. **Select Save** The Console shows `Your data stream has been created`. The stream appears in the **Data Stream** list with `Elasticsearch` in the **Connector** column and the **Active** status. **API** To create the stream with the API, send a `POST` request to `https://api.azion.com/v4/workspace/stream/streams`. Replace `[TOKEN VALUE]` with your personal token, `` with the ID of your workload, `[API KEY]` with the `encoded` value of your API key, and the URL with your own: ```bash curl -X POST 'https://api.azion.com/v4/workspace/stream/streams' \ -H 'Content-Type: application/json' \ -H 'Authorization: Token [TOKEN VALUE]' \ -d '{ "name": "logs-to-elasticsearch", "active": true, "inputs": [ { "type": "raw_logs", "attributes": { "data_source": "workloads" } } ], "transform": [ { "type": "filter_workloads", "attributes": { "workloads": [] } }, { "type": "render_template", "attributes": { "template": 2 } } ], "outputs": [ { "type": "elasticsearch", "attributes": { "url": "https://elasticsearch.example.com/azion-logs", "api_key": "[API KEY]" } } ] }' ``` The `workloads` data source is *Applications* in the Console, and template `2` is *Applications Event Collector*. The API answers `201` with the stored stream: ```json { "state": "executed", "data": { "id": 12351, "name": "logs-to-elasticsearch", "last_editor": "user@example.com", "created": "2026-01-01T12:10:31.000000Z", "last_modified": "2026-01-01T12:10:31.000000Z", "product_version": "1.0", … "outputs": [ { "type": "elasticsearch", "attributes": { "url": "https://elasticsearch.example.com/azion-logs", "api_key": "[API KEY]" } } ] } } ``` The `id` identifies the stream in every later request, such as `/v4/workspace/stream/streams/12351`. For every key of the body, refer to [Stream settings](/en/documentation/platform/data-stream/stream-settings/#stream-object). The API and the Console save the stream without contacting the instance. A wrong URL, index, or API key surfaces only when the stream sends. An activation takes effect after one to two minutes. --- ## Confirm the delivery [Real-Time Events](/en/documentation/platform/real-time-events/data-sources/#data-stream) records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute: a stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines. **Console** To find the sends in Azion Console: 1. **Open Real-Time Events** Access [Azion Console](https://console.azion.com/) > **Real-Time Events**. 2. **Select the Data Stream data source** 3. **Read the latest sends** Each row is one send. Find the rows with `ELASTICSEARCH` in **Endpoint Type**, and read their **Status Code**. A **Status Code** of `200` means the instance accepted the batch. **Streamed Lines** gives the number of log lines in the batch. **API** To read the sends with the API, query the `dataStreamedEvents` dataset of the Real-Time Events GraphQL API. Replace the dates with a range that covers the activation of the stream: ```bash curl -X POST 'https://api.azion.com/v4/events/graphql' \ -H 'Content-Type: application/json' \ -H 'Authorization: Token [TOKEN VALUE]' \ -d '{"query":"query { dataStreamedEvents(limit: 20, filter: {tsRange: {begin: \"2026-01-01T12:00:00\", end: \"2026-01-01T12:45:00\"}}, orderBy: [ts_DESC]) { ts endpointType statusCode streamedLines dataStreamed } }"}' ``` The API answers `200` with one record for each send, the latest first: ```json { "data": { "dataStreamedEvents": [ { "ts": "2026-01-01T12:02:04Z", "endpointType": "ELASTICSEARCH", "statusCode": 200, "streamedLines": 2, "dataStreamed": 2797 } ] } } ``` For a stream that sends to Elasticsearch, `endpointType` reads `ELASTICSEARCH`. A `statusCode` of `200` means the endpoint accepted the batch, and `streamedLines` and `dataStreamed` give its size in log lines and bytes. An empty `dataStreamedEvents` list means the stream has not sent in the range. For every field, refer to [Real-Time Events GraphQL fields](/en/documentation/devtools/graphql/gql-real-time-events-fields/#datastreamedevents-data-stream). A status other than `200` is the answer of the endpoint, and `503` means Data Stream found the endpoint unavailable. For the causes, refer to [Troubleshoot Data Stream](/en/documentation/platform/data-stream/troubleshooting/). --- ## Next steps - [Endpoints](/en/documentation/platform/data-stream/endpoints.md#elasticsearch): Every field of the Elasticsearch endpoint, with its type, bounds, and API name. - [Edit, stop, or delete a stream](/en/documentation/guides/platform/observability/delete-data-stream.md): Change the URL or the API key, pause the stream, or remove it. - [Troubleshoot Data Stream](/en/documentation/platform/data-stream/troubleshooting.md): Find what to change when a send returns a status other than 200. - [Stream WAF events to a SIEM](/en/documentation/guides/application-security/firewall-and-waf/integrate-siems.md): Send the WAF events of a firewall to the SIEM of your choice.