Using Elasticsearch to receive data from Azion Data Streaming
Data Streaming is one of Azion’s Observe products designed to help you access your content and application data in real time. To successfully use Data Streaming with connectors, you first need to set up the endpoints.
After completing the initial setup, you can use Elasticsearch to receive data from Azion’s Data Streaming. Once you finish connecting the endpoint, you can improve your monitoring and use other Azion products to continue exploring information on your data.
Continue reading this hands-on guide to see step by step of how to connect Elasticsearch endpoint to Data Streaming.
You can run an Elasticsearch instance on any cloud computing platform. In that case, to set it up, look for the specific documentation for your chosen platform. For the next steps, you’ll use the Elasticsearch cloud. To do so, you’ll have to create an account at Elasticsearch Cloud.
Using the Elasticsearch Cloud demands a few more steps before you return to Azion’s Real-Time Manager (RTM) and configure your Data Streaming endpoint.
Follow the next steps to continue your Elasticsearch Cloud configuration:
- After creating an account on the Elasticsearch stack, create a cloud deployment — this process takes about five minutes.
To create a deploy, you have to provide the following information:
- A name.
- A cloud provider.
- A region for your cloud.
- A hardware profile.
- The version you’ll use.
- Wait for the process to complete.
- Go to Dashboard on Elastic Cloud.
- You can now see the deployment you just finished with the name you’ve chosen.
- Click on the gear on the right corner of the deploy to access the configuration page for the deployment.
- Create an API key following the Elasticsearch instructions.
2. Configuring the new endpoint in Azion DataStreaming
Next, you’ll follow these steps to configure the new endpoint you created in Elasticsearch in your Azion Data Streaming.
- If you are a new user of Data Streaming, access the Account Menu > Billing & Subscriptions in Real-Time Manager (RTM) and enable the product in the Subscriptions tab.
- On the upper left corner of the page, select Products Menu > Data Streaming.
- Click the Add Streaming button.
- Type a name for your Data Streaming.
- In the Data Source dropdown list, select an option:
- Edge Applications
- Edge Functions
- WAF Events
For more information on creating Edge Applications, see the documentation page.
- In Template, choose the option that fits your Data Source choice:
- WAF Event Collector
- Edge Functions Event Collector
- Edge Applications + WAF Event Collector
- Custom Template
In case you select Custom Template, you’ll need to provide information in a JSON format in the Data Set field.
In Options, you can:
- Select Filter Domains and add the domains you want to receive the logs by selecting them in the Available Domains box.
- Select All Domains.
- In the Destination configurations, select Elasticsearch from the dropdown list.
- Fill the following fields:
- Elasticsearch URL: The URL plus the index of the Elasticsearch to receive the collected data from Data Streaming. For example: https://elasticsearch-domain.com/index
- API Key: API Key in base64 used for Elasticsearch authorization.
- Make sure the Active switch is on.
- Click the Save button.
After saving the configurations, you can keep track of the calls made by Elasticsearch to Data Streaming in the Real-Time Events product, available at the Products Menu at RTM. To do so, select tData Source > Data Streaming and choose the filters options as you wish.
Elasticsearch is a registered trademark of Elasticsearch B.V. in the United States and other countries.
Didn’t find what you were looking for? Open a support ticket.