--- name: azion-send-logs-to-google-bigquery description: >- Create a stream that writes the logs of your applications as rows in a Google BigQuery table, in Azion Console or with the Azion API, and confirm the delivery. --- # Send logs to Google BigQuery You can send the logs of a stream to a table in [Google BigQuery](https://cloud.google.com/bigquery/) from Azion Console or with the Azion API. [Data Stream](/en/documentation/platform/data-stream/) sends the log lines as rows to a table of a BigQuery dataset. In the stream form, the endpoint is set in the field labeled **Connector**, and BigQuery is its *Google BigQuery* option. For every field and its bounds, refer to [Endpoints](/en/documentation/platform/data-stream/endpoints/#google-bigquery). The stream signs in to Google with a service account key that you provide. Data Stream performs the Google OAuth 2.0 authentication and generates the JSON Web Tokens, so the key is the only credential you set. The example collects the requests of one workload with the *Applications* data source. --- Select your interface once. The prerequisites and every task below show only that path. ## Prerequisites - An Azion account with the **Edit Data Stream** permission. For the permissions, refer to [Stream settings](/en/documentation/platform/data-stream/stream-settings/#permissions). - A [workload](/en/documentation/platform/workloads/) on the account that receives requests. - A Google Cloud project with the BigQuery API enabled, and the ID of the project. - Billing enabled on the project. The free tier does not accept rows streamed into a table. Google Cloud lists the fees for these inserts under **Streaming Inserts** in the BigQuery price table. - A dataset in the project, and its dataset ID. The dataset ID is case sensitive. - A table in the dataset, with a schema for the data the stream sends, and its table ID. The table ID is the name you give the table. - A Google Cloud service account with the **BigQuery Admin** role, and a key of the service account in JSON format. Google Cloud downloads the key as a JSON file when you create it. The key file holds the keys below. Data Stream takes the whole content of the file: ```json { "type": "service_account", "project_id": "", "private_key_id": "", "private_key": "", "client_email": "", "client_id": "", "auth_uri": "", "token_uri": "", "auth_provider_x509_cert_url": "", "client_x509_cert_url": "" } ``` **Console** - Access to Azion Console. To sign in, refer to [How to access Azion Console](/en/documentation/guides/platform/account-and-billing/how-to-access-azion-console/). **API** - A personal token. To create one, refer to [How to manage a personal token](/en/documentation/guides/platform/account-and-billing/personal-tokens/). - The ID of the workload. - `curl`. --- ## Create the stream The stream collects from the workload you choose, through a workload filter. A workload filter keeps your other streams active, which a sampled stream does not. **Console** To create the stream in Azion Console: 1. **Open Data Stream** Access [Azion Console](https://console.azion.com/) > **Data Stream**. 2. **Select + Stream** 3. **Name the stream** In the **General** section, enter a **Name**. For example: `logs-to-bigquery`. 4. **Select the data source** In the **Input** section, select *Applications* in **Data Source**. 5. **Turn off Sampling** In the **Transform** section, turn off **Sampling** while **Option** is still *All Current and Future Workloads*, its starting value. A stream cannot carry sampling and a workload filter together. 6. **Choose the workload** In the **Transform** section, set **Option** to *Filter Workloads*. In **Available Workload**, select your workload and move it to **Chosen Workload** with the arrow. 7. **Select the template** In the **Render Template** section, select *Applications Event Collector* in **Template**. 8. **Select the BigQuery endpoint** In the **Output** section, select *Google BigQuery* in **Connector**. 9. **Enter the table location** Enter the ID of your Google Cloud project in **Project ID**, the dataset ID in **Dataset ID**, and the table ID in **Table ID**. For example: `my-project`, `azion_logs`, and `requests`. 10. **Paste the service account key** In **Service Account Key**, paste the whole content of the JSON key file. The field is a JSON editor. 11. **Keep the stream active** In the **Status** section, keep **Active** turned on. 12. **Select Save** The Console shows `Your data stream has been created`. The stream appears in the **Data Stream** list with `BigQuery` in the **Connector** column and the **Active** status. **API** To create the stream with the API, send a `POST` request to `https://api.azion.com/v4/workspace/stream/streams`. Replace `[TOKEN VALUE]` with your personal token, `` with the ID of your workload, and the table values with your own. Replace `[SERVICE ACCOUNT KEY]` with the content of the key file as one JSON string, with each quotation mark and backslash of the file escaped with a `\`: ```bash curl -X POST 'https://api.azion.com/v4/workspace/stream/streams' \ -H 'Content-Type: application/json' \ -H 'Authorization: Token [TOKEN VALUE]' \ -d '{ "name": "logs-to-bigquery", "active": true, "inputs": [ { "type": "raw_logs", "attributes": { "data_source": "workloads" } } ], "transform": [ { "type": "filter_workloads", "attributes": { "workloads": [] } }, { "type": "render_template", "attributes": { "template": 2 } } ], "outputs": [ { "type": "big_query", "attributes": { "project_id": "my-project", "dataset_id": "azion_logs", "table_id": "requests", "service_account_key": "[SERVICE ACCOUNT KEY]" } } ] }' ``` The `workloads` data source is *Applications* in the Console, and template `2` is *Applications Event Collector*. The API answers `201` with the stored stream: ```json { "state": "executed", "data": { "id": 12350, "name": "logs-to-bigquery", "last_editor": "user@example.com", "created": "2026-01-01T12:10:29.000000Z", "last_modified": "2026-01-01T12:10:29.000000Z", "product_version": "1.0", … "outputs": [ { "type": "big_query", "attributes": { "dataset_id": "azion_logs", "project_id": "my-project", "table_id": "requests", "service_account_key": "[SERVICE ACCOUNT KEY]" } } ] } } ``` Keep the `id`: it identifies the stream in every later request, such as `/v4/workspace/stream/streams/12350`. For every key of the body, refer to [Stream settings](/en/documentation/platform/data-stream/stream-settings/#stream-object). The API and the Console save the stream without contacting Google Cloud. A wrong project, dataset, table, or key surfaces only when the stream sends. An activation takes effect after one to two minutes. --- ## Confirm the delivery [Real-Time Events](/en/documentation/platform/real-time-events/data-sources/#data-stream) records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute: a stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines. **Console** To find the sends in Azion Console: 1. **Open Real-Time Events** Access [Azion Console](https://console.azion.com/) > **Real-Time Events**. 2. **Select the Data Stream data source** 3. **Read the latest sends** Each row is one send. Find the rows with `BIG_QUERY` in **Endpoint Type**, and read their **Status Code**. A **Status Code** of `200` means BigQuery accepted the batch. **Streamed Lines** gives the number of log lines in the batch. **API** To read the sends with the API, query the `dataStreamedEvents` dataset of the Real-Time Events GraphQL API. Replace the dates with a range that covers the activation of the stream: ```bash curl -X POST 'https://api.azion.com/v4/events/graphql' \ -H 'Content-Type: application/json' \ -H 'Authorization: Token [TOKEN VALUE]' \ -d '{"query":"query { dataStreamedEvents(limit: 20, filter: {tsRange: {begin: \"2026-01-01T12:00:00\", end: \"2026-01-01T12:45:00\"}}, orderBy: [ts_DESC]) { ts endpointType statusCode streamedLines dataStreamed } }"}' ``` The API answers `200` with one record for each send, the latest first: ```json { "data": { "dataStreamedEvents": [ { "ts": "2026-01-01T12:02:04Z", "endpointType": "BIG_QUERY", "statusCode": 200, "streamedLines": 2, "dataStreamed": 2797 } ] } } ``` A send to BigQuery carries `BIG_QUERY` in `endpointType`. A `statusCode` of `200` means the endpoint accepted the batch, and an empty `dataStreamedEvents` list means the stream has not sent in the range. For every field, refer to [Real-Time Events GraphQL fields](/en/documentation/devtools/graphql/gql-real-time-events-fields/#datastreamedevents-data-stream). A status other than `200` is the answer of the endpoint, and `503` means Data Stream found the endpoint unavailable. For the causes, refer to [Troubleshoot Data Stream](/en/documentation/platform/data-stream/troubleshooting/). --- ## Next steps - [Endpoints](/en/documentation/platform/data-stream/endpoints.md#google-bigquery): Every field of the BigQuery endpoint, with its type, bounds, and API name. - [Create a custom template](/en/documentation/guides/application-development/frameworks/data-stream-custom-template.md): Choose the variables of each log line, and so the data the table receives. - [Edit, stop, or delete a stream](/en/documentation/guides/platform/observability/delete-data-stream.md): Change the table or the key, pause the stream, or remove it. - [Troubleshoot Data Stream](/en/documentation/platform/data-stream/troubleshooting.md): Find what to change when a send returns a status other than 200.