While configuring data streamings, you need to set up a specific endpoint to stream your Azion data.
Continue reading for a step by step on how to connect a Google BigQuery endpoint to receive data from Data Streaming.
Google BigQuery requirementsSection titled Google BigQuery requirements
To get started with Google BigQuery, you must follow a few steps:
- Create a Google Cloud Platform account.
- Create a project on the Google Cloud Platform. The Project ID will be given while you create it, make sure to save it.
- Create a service account on the Google Platform.
- The service account must have the BigQuery Admin permissions. Make sure you select that option in the Role dropdown list.
For more details about the standard permissions assigned to the role of data editor of BigQuery, access BigQuery Roles.
You’ll also need to create and configure the following information:
- Enable the BigQuery API
- A dataset
- A service account key
Enabling BigQuery APISection titled Enabling BigQuery API
Next, you must access the API Manager and enable the BigQuery API.
The BigQuery API supports an endpoint to stream rows into a table. However, this endpoint isn’t supported in the Free Tier version. To use it, it’s necessary to enable the full version of the platform with the billing configuration.
Find more details on this step in the documentation of Billing management on projects. You can consult the fees for this API in the Streaming Inserts section’s price table.
Creating a datasetSection titled Creating a dataset
After enabling the API, you’ll need to create a dataset. To do so, you must first have created a project in the Google Cloud Console. By default, BigQuery is already enabled in new projects.
After creating the project, follow these steps:
- On your Google Cloud console, open the BigQuery page.
- Select the project you want to create a dataset on.
- Click the Actions option, with the vertical ellipsis > Create dataset.
- Fill in the required information. After choosing a Dataset ID, make sure to save it.
- Click Create dataset.
After creating a dataset, you must create a table:
- Create a table and associate it to the dataset you’ve just created.
- Make sure you save the Table name you choose. That’s the Table ID.
- On Schema, add the structure of the data that will be inserted.
- Click Create table.
Once you create the table, it’s possible to ingest data through the BigQuery API.
Creating a service account keySection titled Creating a service account key
Next, you must create a private key to continue your configuration.
- After creating the service account, access your service account.
- On the left menu, click IAM & Admin > Service Accounts.
- Select the service account you’ve created from the list.
- Click on the KEYS tab on the upper menu > ADD KEY > Create new Key.
- On Key Type, choose the JSON option > CREATE.
- After the confirmation, a JSON file will be automatically downloaded with the credentials.
The file’s content should look similar to this:
Configuring the endpoint in Data StreamingSection titled Configuring the endpoint in Data Streaming
Next, follow these steps to configure the new endpoint you created in Google BigQuery in your Azion Data Streaming.
You can find detailed steps for the entire configuration on the How to use Data Streaming guide.
In the Destination configurations:
- On the Endpoint Type dropdown menu, select Google BigQuery.
- On Project ID, add the ID of your project in Google Cloud.
- On Dataset ID, add the ID of your dataset created on Google BigQuery.
- On Table ID, add the ID of your table that will receive the streamed data.
- On Service Account Key, paste the content of the downloaded
JSONfile that contains your private access key.
The private access key should look similar to this:
- Make sure the Active switch is on.
- Click the Save button.
All authentication with Google Oauth2 and generation of JWTokens will be performed by the Data Streaming backend systems.
After saving the configurations, your data will be streamed to the newly configured endpoint.
You can keep track of the calls made by Data Streaming to Google BigQuery on Real-Time Events. To do so, select Data Source > Data Streaming and choose the filters options as you wish.