ContactSign in

How to use Google BigQuery to receive data from Data Stream

While configuring streams, you need to set up a specific endpoint to stream your Azion data.

Continue reading for a step by step on how to connect a Google BigQuery endpoint to receive data from Data Stream.

Google BigQuery requirements

To get started with Google BigQuery, you must follow a few steps:

  1. Create a Google Cloud Platform account.
  2. Create a project on the Google Cloud Platform. The Project ID will be given while you create it, make sure to save it.
  3. Create a service account on the Google Platform.
  4. The service account must have the BigQuery Admin permissions. Make sure you select that option in the Role dropdown list.

For more details about the standard permissions assigned to the role of data editor of BigQuery, access BigQuery Roles.

You’ll also need to create and configure the following information:

  • Enable the BigQuery API
  • A dataset
  • A service account key

Enabling BigQuery API

Next, you must access the API Manager and enable the BigQuery API.

The BigQuery API supports an endpoint to stream rows into a table. However, this endpoint isn’t supported in the Free Tier version. To use it, it’s necessary to enable the full version of the platform with the billing configuration.

Find more details on this step in the documentation of Billing management on projects. You can consult the fees for this API in the Streaming Inserts section’s price table.

Creating a dataset

After enabling the API, you’ll need to create a dataset. To do so, you must first have created a project in the Google Cloud Console. By default, BigQuery is already enabled in new projects.

After creating the project, follow these steps:

  1. On your Google Cloud console, open the BigQuery page.
  2. Select the project you want to create a dataset on.
  3. Click the Actions option, with the vertical ellipsis > Create dataset.
  4. Fill in the required information. After choosing a Dataset ID, make sure to save it.
  5. Click Create dataset.

After creating a dataset, you must create a table:

  1. Create a table and associate it to the dataset you’ve just created.
  2. Make sure you save the Table name you choose. That’s the Table ID.
  3. On Schema, add the structure of the data that will be inserted.
  4. Click Create table.

Once you create the table, it’s possible to ingest data through the BigQuery API.

Creating a service account key

Next, you must create a private key to continue your configuration.

  1. After creating the service account, access your service account.
  2. On the left menu, click IAM & Admin > Service Accounts.
  3. Select the service account you’ve created from the list.
  4. Click on the KEYS tab on the upper menu > ADD KEY > Create new Key.
  5. On Key Type, choose the JSON option > CREATE.
  6. After the confirmation, a JSON file will be automatically downloaded with the credentials.

The file’s content should look similar to this:

{
  "type": "service_account",
  "project_id": "azion-data",
  "private_key_id": "13e018d99d6ay9e3c9f3e21a7a7e0226a1ae082",
  "private_key": "-----BEGIN PRIVATE KEY-----\\nxxx\\n-----END PRIVATE KEY-----\\n",
  "client_email": "myemail@azion.com",
   ...
}

Configuring the endpoint in Data Stream

Next, follow these steps to configure the new endpoint you created in Google BigQuery in your Azion Data Stream.

You can find detailed steps for the entire configuration on the How to use Data Stream guide.

In the Destination configurations:

  1. On the Connector dropdown menu, select Google BigQuery.
  2. On Project ID, add the ID of your project in Google Cloud.
  3. On Dataset ID, add the ID of your dataset created on Google BigQuery.
  4. On Table ID, add the ID of your table that will receive the streamed data.
  5. On Service Account Key, paste the content of the downloaded JSON file that contains your private access key.

The private access key should look similar to this:

{
"type": "service_account",
"project_id": "[name]",
"private_key_id": "13c73d892hf6e8s04hjkloi6759f1e6df39f9038",
"private_key": "-----BEGIN PRIVATE KEY-----
MIIEvAIpDEryaqLPEuiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCzL+bgfcynhWOx
AKQ6wfsnwl/jEYsu5KxlPTtr11hmHLVtDAC68FVjAL029zfTjCRIG9d2ttm6fySY
Jm7Y1MwpahekDmFhMbISxA5UfN0KAF5Bs/uGU6hm17tq+ZDSA1L9f3UIvAJ5/cqu
9CKhU1Dm1TChL8nxIfAb90G7ga6QJVve3ko/0KHpq7pdm3tp6VsVQ+fgwKNi7L+A
4CvHFT0jX4jRDIFUKePRuxyleZV5p1Y3BHSLCIC1X+oe36a0RMLBCrWVdhHwAqBb
bec3NYTen4Re+BidL0cfJ8IsVhjdWuibQTaT2/V+OzA+JgzXvpYSI0jWUvUiYtRK
jxz7AGaXAgMBAAECggEASA0bua76ElAjTg9ixKFg7u0/4P4cWfAM1cf64+e9zPJ6/
H5NaW8cpWf+7C/MxlOdH/zojHKScMyWhXu0wvpKalGXWr+F5/mVCsu2wqfoIhPh
zeAq72KB5MtBLI4ecPkbCnyGKbt9909TfRrrLBEl58EHNaUwEvRDzsmBpn1JDe75
J2ODNf714DsDtghG5Jy5nZ75Bk6ny5mYp67q6IdCUFJeLgJUwfNdtUJmcQ5x7lw3
ujR0vEyEWXpiSAsIhIi0XgMr5NSbBdH+e+P9gVUZwqtRbshdH6aPalIxh1rhdEtY
JguGzK9nbYQtzm0Mdka3VZtUZIEQAqlg8OZe8xLpa+p392TU64sQlrJxQMZxPNtU
tPuDwtDAgmwGZNGFxgBFIMuzN88QpL5zPFSBbJoHt5xJ3sGNmeuDF9SrBXNrFz
9hmqUtoUa0iNheVNG+Y7smEnJNjuSYldAlBQ5qjqSr1IAJTwoUE0fF1P3SbFK9b2
W6TJ73gqF78EQIJf6t3kOczm/QKB0pRMSuGK2ga45ig2CtMSklUHVjL3A+zcEP9NH
osFRYkxZZShPqKj2j0PAdB2TcUgrl1a+I+6oA1oU/j0fuJiux9pxrz9I8QfTVwJQS
/oCcHsKMrDngi0+DkETHDe9peDPTfO4MAh+G285MDPa3LegEG2iVGsqhp+5v8Jdm0Vl
CyZQJ526IwKBgESw1npFyakE0sMGjlwBRjworH5HjajNPsJjZtspaU7TkCXsS7bt
wFmLmm7205SKM+1N9C4owSn25uxIWbsb/wB6iuK+EyP+K3qnjPI/GsVRpDjXb1Ma
iBe4tZCUUP/lJGj8HvBk+kD/lQoFuFndD6cvwDze+PpUeN2oe7IiiZQBlAoGAcQUp
HT3lCVmxXC049FKa8DyWTJIQJhkJmDADeqlYaCFaUe9YC490Y+BtYZHX0UNDXCnFZ
LIBTtRTPfFU02kUBAcGn0ALc74QwUnJlImvuOeYOlgGwy6QzcRQ6dtfsDWROwKk
NCAAjYBylKF2QcuZC3rwe0qN5EIe/0DoFmWUD7ELCgYBIKy2ojKY2d+IByJakBOXt
ojwlCj+I5GpDtDeVhzw9u+74j7KoLsKE057DnMGgouGdVH2xCKih7E71iDKPx1Li
ar9Dz3LsPzHGYXt0LBa+0RBm8mRVb68AlFuN3XJ7g9H8tXPZl38hwLKMEkDJruapG84nuOcgrp2zGHwYtp9S7DfUg==
-----END PRIVATE KEY-----
",
"client_email": "email@myemailaccount.com",
"client_id": "1835090363570189530221",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/mytest.com"
}
  1. Make sure the Active switch is on.
  2. Click the Save button.

All authentication with Google Oauth2 and generation of JWTokens will be performed by the Data Stream backend systems.

After saving the configurations, your data will be streamed to the newly configured endpoint.

You can keep track of the calls made by Data Stream to Google BigQuery on Real-Time Events. To do so, select Data Source > Data Stream and choose the filters options as you wish.

Learn how to use Google BigQuery to receive data from Data Stream. Watch the video below:

Trademarks

Google BigQuery and Google Cloud Platform are registered trademarks of Google LLC in the US and/or other countries.