feat: add GCS support as file provider (#2013) #2117

Suryakantdsa · 2025-08-02T18:44:12Z

Pull Request Template

Description:

Added GCS file provider in pkg/datasource/file/gcs .
Implemented Create , Remove , ReadDir , Open ,Stat and MakeDir using cloud.google.com/go/storage .
Simulates directories using object key prefixes .

Checklist:

I have formatted my code using goimport and golangci-lint.
All new code is covered by unit tests.
This PR does not decrease the overall code coverage.
I have reviewed the code comments and documentation for clarity.

Fixes #2013

Suryakantdsa · 2025-08-03T18:09:13Z

hi @Umang01-hash , @coolwednesday ,

Please have a look at my PR , let me know if you notice anything that needs changes.

thanks !

coolwednesday · 2025-08-04T07:41:15Z

hi @Umang01-hash , @coolwednesday ,

Please have a look at my PR , let me know if you notice anything that needs changes.

thanks !

Hey!

Can you add Setup details for the ease of testing and documentation for users who would want to setup the same by referring documentation (when this feature would be out).

Suryakantdsa · 2025-08-04T13:52:02Z

sure @coolwednesday , I will add the setup guide and documentation to help the user to use this feature.

thanks for the feedback👍

Suryakantdsa · 2025-08-04T17:23:59Z

hey @coolwednesday ,

I have noticed that the setup details for file handling (S3,FTP/SFTP etc .) are already there in gofr/docs/advanced-guide/handling-file/page.md .

Just wanted to confirm , should I add a setup instruction for GCS in the same file or would you prefer this to be in a separate file ?

thanks..!

Suryakantdsa · 2025-08-09T07:09:48Z

Hey @coolwednesday ,

Could you please take a look at this above doubt ?

Suryakantdsa · 2025-08-11T06:56:03Z

hey @ccoVeille @Umang01-hash ,
Could you please take a look at this above doubt ?

coolwednesday

Please update the unified handling-files documentation to include a code snippet demonstrating how to add GCS support.

Additionally, make the necessary changes in the Container package so that it works like any other datasource. For reference, review PR #2076 and replicate the relevant changes.

After that, ensure traces, logs, and metrics are integrated (also refer to #2076 for implementation details).

Once implemented:

Add screenshots showing logs, traces, and metrics from the running cloud store.
Add setup documentation in the docs folder.
Update the navigation.js.
Include any Docker container commands used in contributing.md.

Suryakantdsa · 2025-08-12T15:54:17Z

Ok, got it. I’m working on it and will let you know once it’s completed.
@coolwednesday thanks..!

Suryakantdsa · 2025-08-17T07:44:47Z

hey @Umang01-hash , @coolwednesday ,

I have Updated the existing unified Handling File documentation (handling-file.md) to include GCS setup and usage (no new navigation entry added).
Included Docker container commands in contributing.md.

Here are some screenshots showing logs, traces, and metrics from the running cloud store.

Screenshots (click to expand)

Screenshots (click to expand)

Screenshots (click to expand)

Screenshots (click to expand)

Please let me know if you notice anything that needs changes.
Thank you so much..!

coolwednesday

The whole implementation other than this looks good to me.

pkg/gofr/datasource/file/gcs/fs_dir.go

akshat-kumar-singhal

Overall rewrite of file package should be considered before adding more implementations like GCS etc

docs/advanced-guide/handling-file/page.md

pkg/gofr/datasource/file/gcs/file.go

akshat-kumar-singhal · 2025-08-20T05:41:43Z

pkg/gofr/datasource/file/gcs/file.go

+
+	if err != nil {
+		f.logger.Errorf("failed to write: %v", err)
+		msg = err.Error()


what's the purpose of this line?

it captures the error message in a string , so it can be passed along for logging or response handling

pkg/gofr/datasource/file/gcs/file.go

akshat-kumar-singhal · 2025-08-20T05:43:10Z

pkg/gofr/datasource/file/gcs/file.go

+
+	var msg string
+
+	st := statusErr


Is this being used anywhere?

this variable is used only for logging, not for functional logic.

CONTRIBUTING.md

akshat-kumar-singhal · 2025-08-20T05:46:55Z

pkg/gofr/datasource/file/gcs/logger.go

Should we have a separate logger for each integration we do?

akshat-kumar-singhal · 2025-08-20T05:50:43Z

pkg/gofr/datasource/file/gcs/interface.go

+	bucket *storage.BucketHandle
+}
+
+type gcsClient interface {


what's the advantage of creating this interface? Shouldn't GCSFile be implementing a common CloudStorage interface, if not a generic FileStorage interface? We should typically be having a single FileStorage interface which would allow developers to switch between storage options without changing anything in their code.

@coolwednesday thoughts?

The GCS SDK (cloud.google.com/go/storage) doesn’t provide a common interface. By defining a FileSystem abstraction with methods like Create, Remove, Open, and Rename etc—aligned with the S3 implementation—we ensure consistency across providers. This lets developers use the same API regardless of whether they initialize S3 or GCS, e.g.:

app.AddFileStore(s3.New(&s3.Config{...})) //or app.AddFileStore(gcs.New(&gcs.Config{...})) // now can use the unified FileSystem API file, _ := ctx.File.Create("my_file.txt") ctx.File.Rename("old_name.txt", "new_name.txt") ctx.File.Remove("my_dir") ctx.File.Open()

This ensures developers can use the same API across providers, making it easy to initialize either S3 or GCS and interact with files in a uniform way.

We shouldn't have another interface defined for GCS, it should implement the existing "CloudStorage" interface that S3 is implementing.

You’re right that ideally we should have a common CloudStorage interface which both S3 and GCS implement.

However, currently S3 itself doesn’t implement a common CloudStorage interface — it defines its own s3Client interface around the AWS SDK (github.com/aws/aws-sdk-go-v2/service/s3).

So for now, I referred to the existing S3 interface design and created a similar interface around the GCS SDK (cloud.google.com/go/storage) for GCS to keep consistency.

@coolwednesday @akshat-kumar-singhal

@akshat-kumar-singhal After examining the existing implementations, I agree that a common CloudStorage interface would be ideal architecturally. However, I observe that the S3 implementation currently follows the same pattern - it defines its own s3Client interface rather than implementing a shared abstraction.

For consistency with the existing pattern and to avoid blocking this PR, I suggest we merge the current GCS implementation with a TODO comment indicating future refactoring plans:

// TODO: Future improvement - Refactor both S3 and GCS implementations to use a
// common CloudStorageClient interface for better abstraction. This implementation
// currently follows the same pattern as S3 to maintain consistency with existing code.

This allows us to ship the GCS functionality now while documenting our intention to improve the architecture in a dedicated follow-up PR.

…feedback

Suryakantdsa · 2025-09-03T10:01:54Z

Hi @Umang01-hash,

My branch was not up to date with gofr/development. I’ve rebased it and pushed the updates now.
please have a look.

thank you..!

Umang01-hash

Please remove all the go.mod changes from all the other datasources as they are not related to scope of your PR. Also the changes inside examples/using-add-filestore is also not required. Please revert all of these changes.
There are no tests written for entire newly added code but only some of it. Please make sure to write tests using mocks for the newly added code (feel free to refer other datasources to see how they use mocks for testing).
Only export errors, constants etc when needed, unexport all the ones not used outside the directory.

Umang01-hash · 2025-09-03T11:16:15Z

pkg/gofr/datasource/file/gcs/interface.go

+	bucket *storage.BucketHandle
+}
+
+type gcsClient interface {


@akshat-kumar-singhal After examining the existing implementations, I agree that a common CloudStorage interface would be ideal architecturally. However, I observe that the S3 implementation currently follows the same pattern - it defines its own s3Client interface rather than implementing a shared abstraction.

For consistency with the existing pattern and to avoid blocking this PR, I suggest we merge the current GCS implementation with a TODO comment indicating future refactoring plans:

// TODO: Future improvement - Refactor both S3 and GCS implementations to use a
// common CloudStorageClient interface for better abstraction. This implementation
// currently follows the same pattern as S3 to maintain consistency with existing code.

This allows us to ship the GCS functionality now while documenting our intention to improve the architecture in a dedicated follow-up PR.

Umang01-hash · 2025-09-03T11:17:08Z

pkg/gofr/datasource/file/gcs/go.mod

@@ -0,0 +1,66 @@
+module gofr.dev/pkg/gofr/datasource/file/gcs
+
+go 1.24.0


Please use latest go version i.e. go 1.25

Umang01-hash · 2025-09-03T11:20:59Z

pkg/gofr/datasource/file/gcs/file.go

+}
+
+var (
+	ErrNilGCSFileBody      = errors.New("GCS file body is nil")


Why are these errors exported? If not using them outside of the directory can we please unexport them?

Umang01-hash · 2025-09-03T11:22:05Z

pkg/gofr/datasource/file/gcs/file_parse.go

+	"strings"
+	"time"
+
+	file "gofr.dev/pkg/gofr/datasource/file"


Why we need an alias of same name for this import??

Umang01-hash · 2025-09-03T11:22:27Z

pkg/gofr/datasource/file/gcs/file_parse.go

+var (
+	// errNotPointer is returned when Read method is called with a non-pointer argument.
+	errStringNotPointer = errors.New("input should be a pointer to a string")
+	ErrOutOfRange       = errors.New("out of range")


Why is this error exported?

Suryakantdsa · 2025-09-03T12:28:13Z

Hi @Umang01-hash,
I’ve noted the required changes you mentioned. I’m working on them and will update you once they’re completed.

thanks for your time..

Suryakantdsa · 2025-09-03T15:16:39Z

Hi @Umang01-hash,

I’ve made the required changes. Please have a look when you get a chance.

thank you..!

coolwednesday requested changes Aug 12, 2025

View reviewed changes

Suryakantdsa force-pushed the feat/gcs-file-provider branch from 8c79d6a to 7b1824a Compare August 16, 2025 19:06

Suryakantdsa requested a review from coolwednesday August 17, 2025 07:45

coolwednesday requested changes Aug 19, 2025

View reviewed changes

pkg/gofr/datasource/file/gcs/fs_dir.go Show resolved Hide resolved

coolwednesday added re-review reuired re-review required and removed re-review reuired labels Aug 19, 2025

Suryakantdsa requested a review from coolwednesday August 19, 2025 17:38

akshat-kumar-singhal suggested changes Aug 20, 2025

View reviewed changes

Umang01-hash added the gofr-soc label Aug 20, 2025

Suryakantdsa requested a review from akshat-kumar-singhal August 21, 2025 17:31

Suryakantdsa added 11 commits September 3, 2025 12:40

added gcs filestore and some testcase

fd659be

Added GCS FIlestore

7243056

sync with gofr/development

5a893ba

added testcase for Stat

0ee9c84

Fix some typo error

03e691c

feat: add Google Cloud Storage (GCS) file provider (gofr-dev#2013)

1acaf15

format code using goimports and fix lint issues via golangci-lint

70f0a29

fix lint issue

b192c18

Updated unified handling-files documentation for GCS setup guide

350ecec

added traces and metrics

ff93d53

refactor: update docs, logging, and contribution guidelines based on …

22e4f09

…feedback

Suryakantdsa force-pushed the feat/gcs-file-provider branch from e6dcea0 to 22e4f09 Compare September 3, 2025 09:54

Umang01-hash requested changes Sep 3, 2025

View reviewed changes

feat(file/gcs): add GCS support with Go 1.25 and fix requested changes

690b993

Suryakantdsa force-pushed the feat/gcs-file-provider branch from 22e4f09 to 690b993 Compare September 3, 2025 14:45

Suryakantdsa requested a review from Umang01-hash September 3, 2025 14:54

chore(lint): fix gci import formatting issue

dcd0131

		@@ -0,0 +1,66 @@
		module gofr.dev/pkg/gofr/datasource/file/gcs

		go 1.24.0

feat: add GCS support as file provider (#2013) #2117

Are you sure you want to change the base?

feat: add GCS support as file provider (#2013) #2117

Uh oh!

Conversation

Suryakantdsa commented Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Template

Uh oh!

Suryakantdsa commented Aug 3, 2025

Uh oh!

coolwednesday commented Aug 4, 2025

Uh oh!

Suryakantdsa commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Suryakantdsa commented Aug 4, 2025

Uh oh!

Suryakantdsa commented Aug 9, 2025

Uh oh!

Suryakantdsa commented Aug 11, 2025

Uh oh!

coolwednesday left a comment

Choose a reason for hiding this comment

Uh oh!

Suryakantdsa commented Aug 12, 2025

Uh oh!

Suryakantdsa commented Aug 17, 2025

Uh oh!

coolwednesday left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

akshat-kumar-singhal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Suryakantdsa Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Suryakantdsa Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Suryakantdsa commented Sep 3, 2025

Uh oh!

Umang01-hash left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Suryakantdsa commented Aug 2, 2025 •

edited

Loading

Suryakantdsa commented Aug 4, 2025 •

edited

Loading

Suryakantdsa Aug 22, 2025 •

edited

Loading

Suryakantdsa Aug 22, 2025 •

edited

Loading