Send logs to Elasticsearch

You can send the logs of a stream to an Elasticsearch index from Azion Console or with the Azion API.

Data Stream sends each batch of log lines to the index that ends the URL of the endpoint. In the stream form, the endpoint is set in the field labeled Connector, and Elasticsearch is one of its options. For every field and its bounds, refer to Endpoints.

The example collects the requests of one workload with the Applications data source.

Prerequisites

Console

An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
A workload on the account that receives requests.
An Elasticsearch instance. It can run on any cloud platform, such as an Elastic Cloud deployment.
The URL of the instance followed by the name of the index, such as https://elasticsearch.example.com/azion-logs. On Elastic Cloud, the Copy endpoint action of the deployment gives the address of the instance.
An Elasticsearch API key. When Elasticsearch creates the key, it returns an encoded value in Base64. Data Stream takes that encoded value. On Elastic Cloud, you create the key in the API console of the deployment.
Access to Azion Console. To sign in, refer to How to access Azion Console.

Create the stream

The stream collects from the workload you choose, through a workload filter. Saving a stream with sampling deactivates your other streams, and a workload filter does not.

Console

To create the stream in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Select + Stream

Name the stream

In the General section, enter a Name. For example: logs-to-elasticsearch.

Select the data source

In the Input section, select Applications in Data Source.

Turn off Sampling

In the Transform section, turn off Sampling while Option is still All Current and Future Workloads, its starting value. A stream cannot carry sampling and a workload filter together.

Choose the workload

In the Transform section, set Option to Filter Workloads. In Available Workload, select your workload and move it to Chosen Workload with the arrow.

Select the template

In the Render Template section, select Applications Event Collector in Template.

Select the Elasticsearch endpoint

In the Output section, select Elasticsearch in Connector.

Enter the index URL

In URL, enter the URL of the instance followed by the index. For example: https://elasticsearch.example.com/azion-logs, where azion-logs is the index.

Enter the API key

In Encoded API Key, paste the Base64 encoded value of your Elasticsearch API key.

Keep the stream active

In the Status section, keep Active turned on.

Select Save

The Console shows Your data stream has been created. The stream appears in the Data Stream list with Elasticsearch in the Connector column and the Active status.

The API and the Console save the stream without contacting the instance. A wrong URL, index, or API key surfaces only when the stream sends. An activation takes effect after one to two minutes.

Confirm the delivery

Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute: a stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines.

Console

To find the sends in Azion Console:

Open Real-Time Events

Access Azion Console > Real-Time Events.

Select the Data Stream data source

Read the latest sends

Each row is one send. Find the rows with ELASTICSEARCH in Endpoint Type, and read their Status Code.

A Status Code of 200 means the instance accepted the batch. Streamed Lines gives the number of log lines in the batch.

A status other than 200 is the answer of the endpoint, and 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.

Next steps

EndpointsEvery field of the Elasticsearch endpoint, with its type, bounds, and API name.Edit, stop, or delete a streamChange the URL or the API key, pause the stream, or remove it.Troubleshoot Data StreamFind what to change when a send returns a status other than 200.Stream WAF events to a SIEMSend the WAF events of a firewall to the SIEM of your choice.