Add GoogleFetcher #2074

bartek · 2024-12-04T19:46:56Z

This allows the fetching of items using files.get from Google Drive

tballison

Do you want to merge this into main? Or actually just the tika-grpc-3x-features branch?

If we merge to main, I'll cherrypick back to branch_3x.

main is now at 4.0.0-SNAPSHOT and is the dev branch.

tballison · 2024-12-04T20:07:41Z

...s/tika-fetcher-google/src/main/java/org/apache/tika/pipes/fetchers/google/GoogleFetcher.java

+                }
+
+                if (spoolToTemp) {
+                    File tempFile = Files.createTempFile("spooled-temp", ".dat").toFile();


Something like this will ensure that the temp file gets deleted when the TikaInputStream is closed: https://github.com/apache/tika/blob/main/tika-pipes/tika-fetchers/tika-fetcher-s3/src/main/java/org/apache/tika/pipes/fetcher/s3/S3Fetcher.java#L221

Thanks, used in this fixup: 0823d12

tika-pipes/tika-grpc/example-dockerfile/docker-build.sh

tika-pipes/tika-fetchers/tika-fetcher-google/pom.xml

tballison · 2024-12-04T20:10:41Z

tika-pipes/tika-fetchers/tika-fetcher-google/pom.xml

+    <parent>
+        <artifactId>tika-fetchers</artifactId>
+        <groupId>org.apache.tika</groupId>
+        <version>3.0.0-SNAPSHOT</version>


4.0.0-SNAPSHOT?

3.0.0-SNAPSHOT is referenced in many of the pom.xml files inside tika-pipes. I can update this Google Fetcher to reference 4.0.0-SNAPSHOT, but should there be a separate PR to get everything else on the same version identifier?

So, main is 4.0.0-SNAPSHOT now. If you're targetting main, this needs to be 4.0.0-SNAPSHOT.

Separately for the grpc work, y, you'll want to update everything in that branch to 4.0.0-SNAPSHOT.

I pushed da30ede, hopefully this satisfies what you were expecting?

I can also adjust the merge-to branch to be main. Are all the changes inside tika-grpc-3x-features in main? (excluding this one)

bartek · 2024-12-04T20:15:36Z

@tballison Thanks for review! Honestly, I wasn't expecting one as I'm mostly pushing this to collaborate with @nddipiazza, however, if it makes sense to work as a group, then I will happily do so.

I am pushing to tika-grpc-3x-features as that is the branch we're working off in terms of using the tika-pipes project, but I'm not sure on the overall state of things -- if you have a clearer picture, it'd be great to know.

Happy to hop on a quick call if it's easier. I'll read your other comments in the meantime 👍

tballison · 2024-12-04T20:20:59Z

To the degree we can make small/logical changes in main to achieve the project's goals, all the better? This is definitely a standalone PR that can go straight into main, I think.

bartek · 2024-12-04T20:22:49Z

To the degree we can make small/logical changes in main to achieve the project's goals, all the better? This is definitely a standalone PR that can go straight into main, I think.

Sounds great, I'm glad you think so. Let me sort out your feedback and then we'll go from there. Appreciate the response!

bartek · 2024-12-05T14:47:28Z

...a-fetcher-google/src/main/java/org/apache/tika/pipes/fetchers/google/GoogleDriveFetcher.java

Renamed to GoogleDriveFetcher as the files.get call is part of the Google Drive API

I don't expect this fetcher to service other Google APIs, and the fetchKey/configuration is kept simple by narrowing the fetchers scope

tballison · 2024-12-05T22:38:27Z

Sorry, what I meant was that over on the tika-grpc-3x branch you should change everything to 4.0.0-SNAPSHOT. On this PR, you should only need to do that for the GoogleDriveFetcher, and you should make this PR against main.

bartek · 2024-12-06T00:47:27Z

Sorry, what I meant was that over on the tika-grpc-3x branch you should change everything to 4.0.0-SNAPSHOT. On this PR, you should only need to do that for the GoogleDriveFetcher, and you should make this PR against main.

No worries, made that change here: d439aca

Regarding merging to main, it looks like main and tika-grpc-3x are not in sync. There's an additional ~100 commits if I change the branch I wish to merge to as main.

Not sure how this was previously sorted between yourself/Nick, but I'm happy to figure it out if you offer some suggestions.

…r so that it can be modified

…properly

This allows the fetching of items using files.get from Google Drive

Rename to GoogleDriveFetcher. This name is more appropriate as the files.get call is specific to Google Drive

bartek · 2024-12-06T16:29:35Z

Closing this in favour of #2077, which is based off main and targeting it. The tika-grpc-3x-features branch needs significant cleanup.

tballison requested changes Dec 4, 2024

View reviewed changes

bartek force-pushed the tika-grpc-3x-features branch from c28c558 to e06b9f1 Compare December 5, 2024 01:54

bartek requested a review from tballison December 5, 2024 13:57

bartek commented Dec 5, 2024

View reviewed changes

bartek force-pushed the tika-grpc-3x-features branch from da30ede to d439aca Compare December 6, 2024 00:46

bartek changed the base branch from tika-grpc-3x-features to main December 6, 2024 00:47

bartek changed the base branch from main to tika-grpc-3x-features December 6, 2024 00:47

Nicholas DiPiazza added 18 commits December 6, 2024 12:10

TIKA-4181 - grpc server and client

aa38ee2

latest updates - wip

c59be22

code formatting

f1f3d38

fix the issues with deps and such

c7cae5f

add bidi streaming

d57a557

clean up wording

f80844e

make it as a stale connection expiring store

ce8c887

clean

cc144e1

delete dead code

35f803d

add closeable

818ad66

add closeable

e35102f

add closeable

9ce7aaf

fixed issues related to proto lint

6796079

example docker file

03b6477

example docker file and install docker file

e9db491

fix the rm

adf7964

more info

d5f171e

--platform linux/arm64

9e02b84

Nicholas DiPiazza and others added 26 commits December 6, 2024 12:22

TIKA-4252: log violations to console

bef26d8

TIKA-4252: exclude generated sources

9c63154

TIKA-4252: if tika config is read-only, use a tmp file for tika serve…

09d499d

…r so that it can be modified

TIKA-4252: if tika config is read-only, use a tmp file for tika serve…

9cab0ce

…r so that it can be modified

resolve conflicts

9204f7a

improve build script

15ed7da

TIKA-4166: update commons-collections4

d69976c

add a health check

cf5bc75

fix wrongly named config class

7121da6

add a go package

a473bf6

TIKA-4272: fix issue with headers for all requests

b1f4b70

TIKA-4272: back to additional metadata the old way

7aaa3b9

TIKA-4272: fix an issue where the aad credential was not serializing …

6a47348

…properly

TIKA-4272: fix an issue where the aad credential was not serializing …

17b17f1

…properly

TIKA-4272: add another assertion

b21f1d6

TIKA-4272: add microsoft graph to tika server

9566bc8

TIKA-4272: apply fixes to ms graph connector

678f7b8

TIKA-4272: don't check in fetcher

165c21c

TIKA-4272: don't check in fetcher

b44d7c2

TIKA-4272: push config

cde6e8a

Add GoogleFetcher

79227d7

This allows the fetching of items using files.get from Google Drive

fixup! Add GoogleFetcher

3822c9b

fixup! Add GoogleFetcher

bff8774

fixup! Add GoogleFetcher

5be2d80

Rename to GoogleDriveFetcher. This name is more appropriate as the files.get call is specific to Google Drive

fixup! Add GoogleFetcher

1956f24

Target tika-fetcher-google for 4.0.0-SNAPSHOT

4c254df

bartek force-pushed the tika-grpc-3x-features branch from d439aca to 4c254df Compare December 6, 2024 16:23

bartek closed this Dec 6, 2024

bartek mentioned this pull request Dec 6, 2024

Introduce GoogleDrive Fetcher for tika-pipes #2077

Open

Add GoogleFetcher #2074

Add GoogleFetcher #2074

Uh oh!

Conversation

bartek commented Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tballison left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tballison Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

bartek Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tballison Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

bartek Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

tballison Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

bartek Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

bartek commented Dec 4, 2024

Uh oh!

tballison commented Dec 4, 2024

Uh oh!

bartek commented Dec 4, 2024

Uh oh!

bartek Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

tballison commented Dec 5, 2024

Uh oh!

bartek commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartek commented Dec 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bartek commented Dec 4, 2024 •

edited

Loading

tballison left a comment •

edited

Loading

bartek commented Dec 6, 2024 •

edited

Loading