You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/outputs/bigquery.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,14 +11,14 @@ The implementation doesn't support the following, which would be expected in a f
11
11
12
12
## Google Cloud configuration
13
13
14
-
Fluent Bit streams data into an existing BigQuery table using a service account that you specify. Before using the BigQuery output plugin, you must
14
+
Fluent Bit streams data into an existing BigQuery table using a service account that you specify. Before using the BigQuery output plugin, you must:
15
15
16
16
1. To stream data into BigQuery, you must create a [Google Cloud service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) for Fluent Bit.
17
17
1. Create a BigQuery dataset.
18
18
Fluent Bit doesn't create datasets for your data, so you must [create the dataset]((https://cloud.google.com/bigquery/docs/datasets)) ahead of time. You must also grant the service account `WRITER` permission on the dataset.
19
19
20
-
Within the dataset you will need to create a table for the data to reside in. You can follow the following instructions for creating your table. Pay close attention to the schema. It must match the schema of your output JSON. Unfortunately, since BigQuery doesn't allow dots in field names, you will need to use a filter to change the fields for many of the standard inputs (for example, `mem` or `cpu`).
21
-
1.[Create a BigQuery table]((https://cloud.google.com/bigquery/docs/tables)).
20
+
Within the dataset you must create a table for the data to reside in. Use the following instructions for creating your table. Pay close attention to the schema, as it must match the schema of your output JSON. Unfortunately, because BigQuery doesn't allow dots in field names, you must use a filter to change the fields for many of the standard inputs (for example, `mem` or `cpu`).
21
+
1.[Create a BigQuery table](https://cloud.google.com/bigquery/docs/tables).
22
22
1. Fluent Bit BigQuery output plugin uses a JSON credentials file for authentication credentials. [Authorize the service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) to write to the table.
23
23
1. Provide the service account credentials to Fluent Bit.
24
24
With [workload identity federation](https://cloud.google.com/iam/docs/workload-identity-federation), you can grant on-premises or multi-cloud workloads access to Google Cloud resources, without using a service account key. It can be used as a more secure alternative to service account credentials. Google Cloud's workload identity federation supports several identity providers (see documentation) but Fluent Bit BigQuery plugin currently supports Amazon Web Services (AWS) only.
@@ -32,8 +32,8 @@ Fluent Bit streams data into an existing BigQuery table using a service account
32
32
33
33
| Key | Description | Default |
34
34
| :--- | :--- | :--- |
35
-
|`google_service_credentials`| Absolute path to a Google Cloud credentials JSON file. | Value of the environment variable `$GOOGLE_SERVICE_CREDENTIALS`|
36
-
|`project_id`| The project id containing the BigQuery dataset to stream into. |The value of the `project_id` in the credentials file |
35
+
|`google_service_credentials`| Absolute path to a Google Cloud credentials JSON file. | Value of the environment variable `$GOOGLE_SERVICE_CREDENTIALS`.|
36
+
|`project_id`| The project id containing the BigQuery dataset to stream into. |Value of the `project_id` in the credentials file.|
37
37
|`dataset_id`| The dataset id of the BigQuery dataset to write into. This dataset must exist in your project. |_none_|
38
38
|`table_id`| The table id of the BigQuery table to write into. This table must exist in the specified dataset and the schema must match the output. |_none_|
39
39
|`skip_invalid_rows`| Insert all valid rows of a request, even if invalid rows exist. The default value is false, which causes the entire request to fail if any invalid rows exist. |`Off`|
0 commit comments