Using Amazon Kinesis Data Firehose to receive data from Azion Data Streaming
Data Streaming is one of Azion’s Observe products designed to help you access your content and application data in real time. To successfully use Data Streaming with connectors, you first need to set up the endpoints.
After completing the initial setup, you can use Amazon Kinesis Data Firehose to receive data from Azion Data Streaming. Once you finish connecting the endpoint, you can improve your monitoring and use other Azion products to continue exploring information on your data.
Continue reading this hands-on guide to see step by step of how to connect Amazon Kinesis Data Firehose endpoint to Data Streaming.
To start using Amazon Kinesis Data Firehose, follow the steps:
- Create an account on Amazon AWS.
- In the AWS dashboard, create an instance.
- You will need to fill in the requested information. Each piece of information requires a different path to obtain, as exemplified next:
- Stream Name:
- Go to the Amazon Kinesis homepage.
- Select Kinesis Data Firehose as an option.
- When you click the create delivery stream button, you’ll be redirected to your stream creation page.
- In the first drop-down list, select Amazon Kinesis Data Stream as the source.
- In the second drop-down list, select Amazon S3 as the destination.
- These actions will load the rest of the configuration options.
- For Kinesis Data Stream, click browse and choose the previously created data stream from the list that will appear in the pop-up.
- Name the delivery stream according to what you find most convenient.
- Keep the other options with the default value.
- For destination settings, click browse and select a previously created S3 bucket.
- Click on Create a delivery flow.
- This is the region where your Amazon Kinesis instance is running.
- On the top bar of your dashboard, right next to the help button with the ’?’ icon, you’ll find a drop-down menu where you can see the name of the region, as well as its code.
- Use the code to fill in this field, which will look similar to this:
- Access Key and Secret Key:
- Go to the Identify and Access Management (IAM) panel and click the Create new access key button.
- A pop-up will appear on the screen with your key information.
- Click on Display Access Key to display your key information as indicated below:
- Access Key ID, which has a format similar to
AKIA5LVCRTER524RAIA3, will be the requested later on.
- Secret Access Key, which is the information you’ll later fill in the Secret Key field. It should look similar to:
- Save the two keys information in a safe place to fill in their respective fields on Azion Data Streaming later.
- Access Key ID, which has a format similar to
Note: you can also download the EMP file with the information for these keys.
2. Configuring the new endpoint in Azion Data Streaming
Next, you’ll follow these steps to configure the new endpoint you created in Amazon Kinesis Data Firehose in your Azion Data Streaming.
- If you are a new user of Data Streaming, select the Account Menu > Billing & Subscriptions in Real-Time Manager (RTM) and enable the product in the Subscriptions tab.
- On the upper left corner of the page, select Products Menu > Data Streaming product.
- Click the Add Streaming button.
- Type a name for your Data Streaming.
- In the Data Source dropdown list, select an option:
- Edge Applications
- Edge Functions
- WAF Events
For more information on creating Edge Applications, see the documentation page.
- In Template, choose the option that fits your Data Source choice:
- WAF Event Collector
- Edge Functions Event Collector
- Edge Applications + WAF Event Collector
- Custom Template
In case you select Custom Template, you’ll need to provide information in a JSON format in the Data Set field.
In Options, you can:
- Select Filter Domains in the domains box and add the domains you want to receive your logs by selecting them in the Available Domains box.
- Select All Domains.
In the Destination configurations, select AWS Kinesis Data Firehose from the dropdown list. Fill the following fields: Stream Name: Name of your delivery Kinesis Firehose stream. Region: Region of your Kinesis Firehose stream. Access Key: Access key of your Kinesis Firehose stream. Secret Key: Secret key of your Kinesis Firehose stream.
Read more about the fields on the Data Streaming documentation.
- Make sure the Active switch is on.
- Click the Save button.
After saving the configurations, you can keep track of the calls made by Data Streaming to Amazon Kinesis Data Firehose in the Real-Time Events available on Products Menu at RTM. To do so, select the Data Source > Data Streaming and choose the filters options as you wish.
Amazon Web Services, AWS, Amazon Kinesis and affiliates are registered trademarks of Amazon Inc in the United States and other countries.
Didn’t find what you were looking for? Open a support ticket.