Skip to content

Commit 89d76ba

Browse files
authored
DOC-1310 Add google_drive_download processor (#248)
1 parent cdae779 commit 89d76ba

File tree

2 files changed

+153
-0
lines changed

2 files changed

+153
-0
lines changed

modules/ROOT/nav.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,7 @@
147147
*** xref:components:processors/gcp_bigquery_select.adoc[]
148148
*** xref:components:processors/gcp_vertex_ai_chat.adoc[]
149149
*** xref:components:processors/gcp_vertex_ai_embeddings.adoc[]
150+
*** xref:components:processors/google_drive_download.adoc[]
150151
*** xref:components:processors/google_drive_list_labels.adoc[]
151152
*** xref:components:processors/grok.adoc[]
152153
*** xref:components:processors/group_by.adoc[]
Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
= google_drive_download
2+
// tag::single-source[]
3+
:type: processor
4+
:categories: ["AI"]
5+
6+
component_type_dropdown::[]
7+
8+
Downloads files from Google Drive that contain matching file IDs.
9+
10+
Try out the <<example,example pipeline on this page>>, which downloads all files from your Google Drive.
11+
12+
ifndef::env-cloud[]
13+
Introduced in version 4.53.0.
14+
endif::[]
15+
16+
[tabs]
17+
======
18+
Common::
19+
+
20+
--
21+
22+
```yml
23+
# Common configuration fields, showing default values
24+
label: ""
25+
google_drive_download:
26+
credentials_json: "" # No default (optional)
27+
file_id: "" # No default (required)
28+
mime_type: "" # No default (required)
29+
```
30+
31+
--
32+
Advanced::
33+
+
34+
--
35+
36+
```yml
37+
# All configuration fields, showing default values
38+
label: ""
39+
google_drive_download:
40+
credentials_json: "" # No default (optional)
41+
file_id: "" # No default (required)
42+
mime_type: "" # No default (required)
43+
export_mime_types:
44+
application/vnd.google-apps.document: text/markdown
45+
application/vnd.google-apps.drawing: image/png
46+
application/vnd.google-apps.presentation: application/pdf
47+
application/vnd.google-apps.script: application/vnd.google-apps.script+json
48+
application/vnd.google-apps.spreadsheet: text/csv
49+
```
50+
51+
--
52+
======
53+
54+
== Authentication
55+
56+
By default, this processor uses https://cloud.google.com/docs/authentication/application-default-credentials[Google Application Default Credentials (ADC)^] to authenticate with Google APIs.
57+
58+
To set up local ADC authentication, use the following `gcloud` commands:
59+
60+
- Authenticate using Application Default Credentials and grant read-only access to your Google Drive.
61+
+
62+
```bash
63+
gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly'
64+
```
65+
- Assign a quota project to the Application Default Credentials when using a user account.
66+
+
67+
```bash
68+
gcloud auth application-default set-quota-project <project-id>
69+
```
70+
+
71+
Replace the `<project-id>` placeholder with your Google Cloud project ID
72+
73+
To use a service account instead, create a JSON key for the account and add it to the <<credentials_json,`credentials_json`>> field. To access Google Drive files using a service account, either:
74+
75+
- Explicitly share files with the service account's email account
76+
- Use https://support.google.com/a/answer/162106[domain-wide delegation^] to share all files within a Google Workspace
77+
78+
== Fields
79+
80+
=== `credentials_json`
81+
82+
The JSON key for your service account (optional). If left empty, Application Default Credentials are used. For more details, see <<authentication, Authentication>>.
83+
84+
include::components:partial$secret_warning.adoc[]
85+
86+
*Type*: `string`
87+
88+
*Default*: `""`
89+
90+
=== `file_id`
91+
92+
The ID of the file to download from Google Drive. This field supports xref:configuration:interpolation.adoc#bloblang-queries[interpolation functions].
93+
94+
*Type*: `string`
95+
96+
*Default*: `""`
97+
98+
=== `mime_type`
99+
100+
The https://developers.google.com/workspace/drive/api/guides/mime-types[MIME type^] of the file for download. This field supports xref:configuration:interpolation.adoc#bloblang-queries[interpolation functions].
101+
102+
*Type*: `string`
103+
104+
*Default*: `""`
105+
106+
=== `export_mime_types`
107+
108+
Maps Google Drive MIME types to https://developers.google.com/workspace/drive/api/guides/ref-export-formats[supported file export formats^]. The MIME type is the key, and the export format is the value.
109+
110+
*Type*: `object`
111+
112+
*Default*: `{"application/vnd.google-apps.document":"text/markdown","application/vnd.google-apps.drawing":"image/png","application/vnd.google-apps.presentation":"application/pdf","application/vnd.google-apps.script":"application/vnd.google-apps.script+json","application/vnd.google-apps.spreadsheet":"text/csv"}`
113+
114+
```yml
115+
# Examples
116+
117+
export_mime_types:
118+
application/vnd.google-apps.document: application/pdf
119+
application/vnd.google-apps.drawing: application/pdf
120+
application/vnd.google-apps.presentation: application/pdf
121+
application/vnd.google-apps.spreadsheet: application/pdf
122+
123+
export_mime_types:
124+
application/vnd.google-apps.document: application/vnd.openxmlformats-officedocument.wordprocessingml.document
125+
application/vnd.google-apps.drawing: image/svg+xml
126+
application/vnd.google-apps.presentation: application/vnd.openxmlformats-officedocument.presentationml.presentation
127+
application/vnd.google-apps.spreadsheet: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
128+
```
129+
130+
== Example
131+
132+
This example downloads all files from a Google Drive.
133+
134+
```yaml
135+
input:
136+
stdin: {}
137+
pipeline:
138+
processors:
139+
- google_drive_search:
140+
query: "${!content().string()}"
141+
- mutation: 'meta path = this.name'
142+
- google_drive_download:
143+
file_id: "${!this.id}"
144+
mime_type: "${!this.mimeType}"
145+
output:
146+
file:
147+
path: "${!@path}"
148+
codec: all-bytes
149+
```
150+
151+
152+
// end::single-source[]

0 commit comments

Comments
 (0)