Skip to content

Commit daa6b99

Browse files
lucas-amiaudlucas-amiaud
authored
Merge pull request #28 from Coreoz/feature/gcp-storage-connector
implem GCP Storage connector
2 parents 1e356bd + 680fa2c commit daa6b99

File tree

11 files changed

+348
-1
lines changed

11 files changed

+348
-1
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ The storage modules implements `FileStorageService`:
4343

4444
- [plume-file-storage-database](plume-file-storage-database): store uploaded file in a database
4545
- [plume-file-storage-system](plume-file-storage-system): store uploaded file on a system disk
46+
- [plume-file-storage-gcp](plume-file-storage-gcp): store uploaded file on a GCP Cloud Storage bucket
4647

4748
#### Metadata module
4849

‎lombok.config‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
lombok.log.fieldname = logger

‎plume-file-storage-gcp/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
/target/
2+
/.settings/
3+
/.classpath
4+
/.project
5+
/.factorypath
6+
/.README.md.html

‎plume-file-storage-gcp/README.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Plume File Storage GCP
2+
3+
A Google Cloud Storage backend for the [Plume File](https://github.com/Coreoz/Plume-file) module.
4+
5+
This library provides a `FileStorageService` implementation that stores and retrieves files from
6+
**Google Cloud Storage (GCS)**.
7+
It supports storing files either at the **root of a bucket** or inside a configurable **subfolder (prefix)**.
8+
9+
---
10+
11+
## Features
12+
13+
* ✅ Configurable **base path**: store files at the bucket root or in a “folder”.
14+
* ✅ Works with **Application Default Credentials**, service account JSON key or Workload Identity.
15+
* ✅ Supports injection of a custom `Credentials` object via `GcpCredentialsProvider`.
16+
17+
---
18+
19+
## Installation
20+
21+
To use this module, add the following dependency to your project.
22+
This will
23+
24+
Maven:
25+
26+
```xml
27+
<dependency>
28+
<groupId>com.coreoz</groupId>
29+
<artifactId>plume-file-storage-gcp</artifactId>
30+
</dependency>
31+
```
32+
33+
In the ApplicationModule class, install the following Guice module:
34+
35+
```java
36+
install(new GuiceFileStorageGcpModule());
37+
```
38+
39+
---
40+
41+
## Google Cloud prerequisites
42+
43+
1. **Create or use a GCS bucket**.
44+
45+
2. **Create a service account** and grant it a role that includes the following permissions:
46+
* `storage.objects.create`
47+
* `storage.objects.get`
48+
* `storage.objects.delete`
49+
(For example: `roles/storage.objectAdmin`)
50+
51+
3. **Provide credentials** to your application:
52+
* Recommended: set the configuration parameter `file.storage.gcp.credentials-path` to the path of the
53+
service account JSON key file.
54+
or
55+
* Provide your own implementation of `GcpCredentialsProvider` (see below).
56+
57+
---
58+
59+
## Usage
60+
61+
### 1. Upload / Fetch / Delete a file
62+
63+
See Usage of the [Plume File core module](../plume-file-core/README.md/#usage) to upload files.
64+
65+
### 2. Override credentials with `GcpCredentialsProvider`
66+
67+
You can implement and bind your own `GcpCredentialsProvider` to provide `com.google.auth.Credentials`:
68+
69+
```java
70+
public class GcpCredentialsProvider implements Provider<Credentials> {
71+
@Override
72+
public Credentials get() {
73+
// return your custom Credentials, e.g. from a JSON file or ADC
74+
}
75+
}
76+
```
77+
78+
Then bind it in your Guice module:
79+
80+
```java
81+
import com.google.auth.Credentials;
82+
83+
// your Guice module ...
84+
85+
bind(Credentials.class).toProvider(MyCustomGcpCredentialsProvider.class);
86+
```
87+
88+
---
89+
90+
## Configurations
91+
92+
| Parameter | Description |
93+
|-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
94+
| `file.storage.gcp.bucket-name` | Name of the GCS bucket. |
95+
| `file.storage.gcp.project-id` | GCP project ID. |
96+
| `file.storage.gcp.bucket-base-path` | Optional folder/prefix inside the bucket. Use `""` for root or e.g. `"uploads/"`. |
97+
| `file.storage.gcp.credentials-path` | File system path to the service account JSON key file (only required for [GcpCredentialsProvider.java](src%2Fmain%2Fjava%2Fcom%2Fcoreoz%2Fplume%2Ffile%2Fcredential%2FGcpCredentialsProvider.java)) |
98+

‎plume-file-storage-gcp/pom.xml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<modelVersion>4.0.0</modelVersion>
6+
<parent>
7+
<groupId>com.coreoz</groupId>
8+
<artifactId>plume-file-parent</artifactId>
9+
<version>4.0.2-SNAPSHOT</version>
10+
</parent>
11+
12+
<artifactId>plume-file-storage-gcp</artifactId>
13+
<packaging>jar</packaging>
14+
<name>Plume File Storage GCP</name>
15+
<description>Plume File Core implementation of FileStorageService to store file on GCP Cloud Storage Buckets</description>
16+
17+
<build>
18+
<resources>
19+
<resource>
20+
<directory>src/main/java</directory>
21+
<includes>
22+
<include>**/*.conf</include>
23+
</includes>
24+
</resource>
25+
</resources>
26+
</build>
27+
28+
<dependencies>
29+
<dependency>
30+
<groupId>com.coreoz</groupId>
31+
<artifactId>plume-file-core</artifactId>
32+
</dependency>
33+
34+
<dependency>
35+
<groupId>com.coreoz</groupId>
36+
<artifactId>plume-conf</artifactId>
37+
</dependency>
38+
39+
<dependency>
40+
<groupId>org.projectlombok</groupId>
41+
<artifactId>lombok</artifactId>
42+
<scope>provided</scope>
43+
</dependency>
44+
45+
<dependency>
46+
<groupId>com.google.inject</groupId>
47+
<artifactId>guice</artifactId>
48+
<optional>true</optional>
49+
</dependency>
50+
51+
<dependency>
52+
<groupId>com.google.cloud</groupId>
53+
<artifactId>google-cloud-storage</artifactId>
54+
<version>2.59.0</version>
55+
</dependency>
56+
</dependencies>
57+
58+
<dependencyManagement>
59+
<dependencies>
60+
<dependency>
61+
<groupId>com.coreoz</groupId>
62+
<artifactId>plume-framework-dependencies</artifactId>
63+
<version>${plume-framework-dependencies.version}</version>
64+
<scope>import</scope>
65+
<type>pom</type>
66+
</dependency>
67+
</dependencies>
68+
</dependencyManagement>
69+
70+
</project>

‎plume-file-storage-gcp/src/main/java/com/coreoz/plume/file/configuration/FileStorageGcpConfigurationService.java‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
package com.coreoz.plume.file.configuration;
2+
3+
import com.typesafe.config.Config;
4+
import jakarta.inject.Inject;
5+
6+
public class FileStorageGcpConfigurationService {
7+
private final Config config;
8+
9+
@Inject
10+
public FileStorageGcpConfigurationService(Config config) {
11+
this.config = config;
12+
}
13+
14+
public String gcpProjectId() {
15+
return config.getString("file.storage.gcp.project-id");
16+
}
17+
18+
public String gcpBucketName() {
19+
return config.getString("file.storage.gcp.bucket-name");
20+
}
21+
22+
public String gcpBucketBasePath() {
23+
return config.getString("file.storage.gcp.bucket-base-path");
24+
}
25+
26+
public String gcpCredentialPath() {
27+
return config.getString("file.storage.gcp.credentials-path");
28+
}
29+
}

‎plume-file-storage-gcp/src/main/java/com/coreoz/plume/file/credential/GcpCredentialsProvider.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
package com.coreoz.plume.file.credential;
2+
3+
import com.coreoz.plume.file.configuration.FileStorageGcpConfigurationService;
4+
import com.google.auth.Credentials;
5+
import com.google.auth.oauth2.ServiceAccountCredentials;
6+
import jakarta.inject.Inject;
7+
import jakarta.inject.Provider;
8+
import lombok.SneakyThrows;
9+
10+
import java.io.FileInputStream;
11+
12+
public class GcpCredentialsProvider implements Provider<Credentials> {
13+
private final Credentials credentials;
14+
15+
@SneakyThrows
16+
@Inject
17+
private GcpCredentialsProvider(FileStorageGcpConfigurationService configurationService) {
18+
this.credentials = ServiceAccountCredentials.fromStream(
19+
new FileInputStream(configurationService.gcpCredentialPath())
20+
);
21+
}
22+
23+
@Override
24+
public Credentials get() {
25+
return credentials;
26+
}
27+
}

‎plume-file-storage-gcp/src/main/java/com/coreoz/plume/file/guice/GuiceFileStorageGcpModule.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
package com.coreoz.plume.file.guice;
2+
3+
import com.coreoz.plume.file.credential.GcpCredentialsProvider;
4+
import com.coreoz.plume.file.service.FileStorageGcpService;
5+
import com.coreoz.plume.file.services.storage.FileStorageService;
6+
import com.google.auth.Credentials;
7+
import com.google.inject.AbstractModule;
8+
9+
public class GuiceFileStorageGcpModule extends AbstractModule {
10+
11+
@Override
12+
protected void configure() {
13+
bind(FileStorageService.class).to(FileStorageGcpService.class);
14+
bind(Credentials.class).toProvider(GcpCredentialsProvider.class);
15+
}
16+
}

‎plume-file-storage-gcp/src/main/java/com/coreoz/plume/file/service/FileStorageGcpService.java‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
package com.coreoz.plume.file.service;
2+
3+
import com.coreoz.plume.file.configuration.FileStorageGcpConfigurationService;
4+
import com.coreoz.plume.file.services.storage.FileStorageService;
5+
import com.google.auth.Credentials;
6+
import com.google.cloud.storage.Blob;
7+
import com.google.cloud.storage.BlobId;
8+
import com.google.cloud.storage.BlobInfo;
9+
import com.google.cloud.storage.Storage;
10+
import com.google.cloud.storage.StorageException;
11+
import com.google.cloud.storage.StorageOptions;
12+
import jakarta.inject.Inject;
13+
import jakarta.inject.Singleton;
14+
import lombok.extern.slf4j.Slf4j;
15+
16+
import java.io.IOException;
17+
import java.io.InputStream;
18+
import java.nio.channels.Channels;
19+
import java.util.List;
20+
import java.util.Optional;
21+
22+
@Slf4j
23+
@Singleton
24+
public class FileStorageGcpService implements FileStorageService {
25+
26+
private final Storage storage;
27+
private final String bucketName;
28+
private final String bucketBasePath;
29+
30+
@Inject
31+
public FileStorageGcpService(
32+
FileStorageGcpConfigurationService configurationService,
33+
Credentials credentials
34+
) {
35+
this.storage = StorageOptions.newBuilder()
36+
.setProjectId(configurationService.gcpProjectId())
37+
.setCredentials(credentials)
38+
.build()
39+
.getService();
40+
this.bucketName = configurationService.gcpBucketName();
41+
this.bucketBasePath = normalizeBasePath(configurationService.gcpBucketBasePath());
42+
if (this.storage == null || this.bucketName == null) {
43+
throw new IllegalStateException("GCP Storage or bucket name is not configured properly");
44+
}
45+
}
46+
47+
@Override
48+
public void add(String fileUniqueName, InputStream fileData) throws IOException {
49+
BlobInfo blobInfo = BlobInfo.newBuilder(bucketName, objectName(fileUniqueName)).build();
50+
storage.createFrom(blobInfo, fileData);
51+
}
52+
53+
@Override
54+
public Optional<InputStream> fetch(String fileUniqueName) {
55+
Blob blob = storage.get(BlobId.of(bucketName, objectName(fileUniqueName)));
56+
if (blob == null) {
57+
return Optional.empty();
58+
}
59+
return Optional.of(
60+
Channels.newInputStream(blob.reader())
61+
);
62+
}
63+
64+
@Override
65+
public void deleteAll(List<String> fileUniqueNames) {
66+
for (String fileUniqueName : fileUniqueNames) {
67+
BlobId blobId = BlobId.of(bucketName, objectName(fileUniqueName));
68+
try {
69+
storage.delete(blobId);
70+
} catch (StorageException e) {
71+
logger.warn("Failed to delete file: {}", fileUniqueName, e);
72+
}
73+
}
74+
}
75+
76+
/**
77+
* Construct the object name by combining the base path and the file unique name.
78+
*
79+
* @param fileUniqueName the unique name of the file
80+
* @return the full object name
81+
*/
82+
private String objectName(String fileUniqueName) {
83+
return bucketBasePath + fileUniqueName;
84+
}
85+
86+
/**
87+
* Normalize the base path to ensure it ends with a slash if not empty.
88+
*
89+
* @param basePath the base path to normalize
90+
* @return the normalized base path
91+
*/
92+
private static String normalizeBasePath(String basePath) {
93+
if (basePath == null || basePath.isBlank()) {
94+
return "";
95+
}
96+
return basePath.endsWith("/") ? basePath : basePath + "/";
97+
}
98+
}

‎plume-file-storage-system/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
Plume File Storage System
2-
===========================
2+
=========================
33

44
A [Plume File](../) module to store file data on the system disk through java.io.File class.
55

0 commit comments

Comments
 (0)