Skip to content

Commit 690b993

Browse files
committed
feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
1 parent 87399e9 commit 690b993

File tree

16 files changed

+2451
-55
lines changed

16 files changed

+2451
-55
lines changed

CONTRIBUTING.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -96,13 +96,11 @@ docker run -d --name arangodb \
9696
-e ARANGO_ROOT_PASSWORD=rootpassword \
9797
--pull always \
9898
arangodb:latest
99-
<<<<<<< HEAD
10099
docker run -d --name db -p 8091-8096:8091-8096 -p 11210-11211:11210-11211 couchbase
101-
=======
102100
docker login container-registry.oracle.com
103101
docker pull container-registry.oracle.com/database/free:latest
104102
docker run -d --name oracle-free -p 1521:1521 -e ORACLE_PWD=YourPasswordHere container-registry.oracle.com/database/free:latest
105-
>>>>>>> origin
103+
docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest
106104
```
107105

108106
> [!NOTE]

docs/advanced-guide/handling-file/page.md

Lines changed: 92 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,15 @@ GoFr simplifies the complexity of working with different file stores by offering
66

77
By default, local file-store is initialized and user can access it from the context.
88

9-
GoFr also supports FTP/SFTP file-store. Developers can also connect and use their AWS S3 bucket as a file-store. The file-store can be initialized as follows:
9+
GoFr also supports FTP/SFTP file-store. Developers can also connect and use their cloud storage bucket as a file-store. Following cloud storage options are currently supported:
10+
11+
- **AWS S3**
12+
- **Google Cloud Storage (GCS)**
13+
14+
The file-store can be initialized as follows:
1015

1116
### FTP file-store
17+
1218
```go
1319
package main
1420

@@ -34,6 +40,7 @@ func main() {
3440
```
3541

3642
### SFTP file-store
43+
3744
```go
3845
package main
3946

@@ -60,8 +67,7 @@ func main() {
6067
### AWS S3 Bucket as File-Store
6168

6269
To run S3 File-Store locally we can use localstack,
63-
``docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack``
64-
70+
`docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack`
6571

6672
```go
6773
package main
@@ -90,17 +96,68 @@ func main() {
9096
app.Run()
9197
}
9298
```
93-
> Note: The current implementation supports handling only one bucket at a time,
94-
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.
99+
100+
> Note: The current implementation supports handling only one bucket at a time,
101+
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.
102+
103+
### Google Cloud Storage (GCS) Bucket as File-Store
104+
105+
To run GCS File-Store locally we can use fake-gcs-server:
106+
`docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest`
107+
108+
```go
109+
package main
110+
111+
import (
112+
"gofr.dev/pkg/gofr"
113+
114+
"gofr.dev/pkg/gofr/datasource/file/gcs"
115+
)
116+
117+
func main() {
118+
app := gofr.New()
119+
120+
// Option 1: Using JSON credentials string
121+
app.AddFileStore(gcs.New(&gcs.Config{
122+
BucketName: "my-bucket",
123+
CredentialsJSON: readFile("gcs-credentials.json"),
124+
ProjectID: "my-project-id",
125+
}))
126+
127+
// Option 2: Using default credentials from env
128+
// app.AddFileStore(gcs.New(&gcs.Config{
129+
// BucketName: "my-bucket",
130+
// ProjectID: "my-project-id",
131+
// }))
132+
133+
app.Run()
134+
}
135+
136+
// Helper function to read credentials file
137+
func readFile(filename string) []byte {
138+
data, err := os.ReadFile(filename)
139+
if err != nil {
140+
log.Fatalf("Failed to read credentials file: %v", err)
141+
}
142+
return data
143+
}
144+
145+
```
146+
147+
> **Note:** When connecting to the actual GCS service, authentication can be provided via CredentialsJSON or the GOOGLE_APPLICATION_CREDENTIALS environment variable.
148+
> When using fake-gcs-server, authentication is not required.
149+
> Currently supports one bucket per file-store instance.
95150
96151
### Creating Directory
97152

98153
To create a single directory
154+
99155
```go
100156
err := ctx.File.Mkdir("my_dir",os.ModePerm)
101157
```
102158

103159
To create subdirectories as well
160+
104161
```go
105162
err := ctx.File.MkdirAll("my_dir/sub_dir", os.ModePerm)
106163
```
@@ -114,27 +171,32 @@ currentDir, err := ctx.File.Getwd()
114171
### Change current Directory
115172

116173
To switch to parent directory
174+
117175
```go
118176
currentDir, err := ctx.File.Chdir("..")
119177
```
120178

121179
To switch to another directory in same parent directory
180+
122181
```go
123182
currentDir, err := ctx.File.Chdir("../my_dir2")
124183
```
125184

126185
To switch to a subfolder of the current directory
186+
127187
```go
128188
currentDir, err := ctx.File.Chdir("sub_dir")
129189
```
190+
130191
> Note: This method attempts to change the directory, but S3's flat structure and fixed bucket
131-
> make this operation inapplicable.
192+
> make this operation inapplicable. Similarly, GCS uses a flat structure where directories are simulated through object prefixes.
132193
133194
### Read a Directory
134195

135-
The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.
196+
The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.
136197

137198
If an error occurs during the read operation, ReadDir returns the successfully read entries up to the point of the error along with the error itself. Passing "." as the directory argument returns the entries for the current directory.
199+
138200
```go
139201
entries, err := ctx.File.ReadDir("../testdir")
140202

@@ -143,12 +205,13 @@ for _, entry := range entries {
143205

144206
if entry.IsDir() {
145207
entryType = "Dir"
146-
}
208+
}
147209

148210
fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
149211
}
150212
```
151-
> Note: In S3, directories are represented as prefixes of file keys. This method retrieves file
213+
214+
> Note: In S3 and GCS, directories are represented as prefixes of file keys/object names. This method retrieves file
152215
> entries only from the immediate level within the specified directory.
153216
154217
### Creating and Save a File with Content
@@ -163,14 +226,15 @@ _, _ = file.Write([]byte("Hello World!"))
163226
```
164227

165228
### Reading file as CSV/JSON/TEXT
229+
166230
GoFr support reading CSV/JSON/TEXT files line by line.
167231

168232
```go
169233
reader, err := file.ReadAll()
170234

171235
for reader.Next() {
172236
var b string
173-
237+
174238
// For reading CSV/TEXT files user need to pass pointer to string to SCAN.
175239
// In case of JSON user should pass structs with JSON tags as defined in encoding/json.
176240
err = reader.Scan(&b)
@@ -179,10 +243,12 @@ for reader.Next() {
179243
}
180244
```
181245

182-
183246
### Opening and Reading Content from a File
247+
184248
To open a file with default settings, use the `Open` command, which provides read and seek permissions only. For write permissions, use `OpenFile` with the appropriate file modes.
249+
185250
> Note: In FTP, file permissions are not differentiated; both `Open` and `OpenFile` allow all file operations regardless of specified permissions.
251+
186252
```go
187253
csvFile, _ := ctx.File.Open("my_file.csv")
188254

@@ -205,6 +271,7 @@ if err != nil {
205271
### Getting Information of the file/directory
206272

207273
Stat retrieves details of a file or directory, including its name, size, last modified time, and type (such as whether it is a file or folder)
274+
208275
```go
209276
file, _ := ctx.File.Stat("my_file.text")
210277
entryType := "File"
@@ -215,10 +282,12 @@ if entry.IsDir() {
215282

216283
fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
217284
```
218-
>Note: In S3:
285+
286+
> Note: In S3 and GCS:
287+
>
219288
> - Names without a file extension are treated as directories by default.
220-
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed.
221-
>
289+
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed (S3 specific behavior).
290+
>
222291
> For directories, the method calculates the total size of all contained objects and returns the most recent modification time. For files, it directly returns the file's size and last modified time.
223292
224293
### Rename/Move a File
@@ -234,18 +303,23 @@ err := ctx.File.Rename("old_name.text", "new_name.text")
234303
### Deleting Files
235304

236305
`Remove` deletes a single file
237-
> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method.
306+
307+
> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method. GCS supports deletion of both files and empty directories.
308+
238309
```go
239310
err := ctx.File.Remove("my_dir")
240311
```
241312

242313
The `RemoveAll` command deletes all subdirectories as well. If you delete the current working directory, such as "../currentDir", the working directory will be reset to its parent directory.
243-
> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided.
314+
315+
> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided for S3.
316+
> GCS handles both files and directories.
317+
244318
```go
245319
err := ctx.File.RemoveAll("my_dir/my_text")
246320
```
247321

248-
> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 uses
249-
> a flat file structure, all methods require a full path relative to the S3 bucket.
322+
> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 and GCS use
323+
> a flat file structure, all methods require a full path relative to the bucket.
250324
251325
> Errors have been skipped in the example to focus on the core logic, it is recommended to handle all the errors.

go.work

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ use (
1111
./pkg/gofr/datasource/elasticsearch
1212
./pkg/gofr/datasource/file/ftp
1313
./pkg/gofr/datasource/file/s3
14+
./pkg/gofr/datasource/file/gcs
1415
./pkg/gofr/datasource/file/sftp
1516
./pkg/gofr/datasource/kv-store/badger
1617
./pkg/gofr/datasource/kv-store/nats

0 commit comments

Comments
 (0)