Skip to content

Commit 7f498c5

Browse files
groldangroldan
committed
Google Cloud Storage blob store implementation
Add support for storing tiles in Google Cloud Storage buckets, similar to the existing S3 and Azure blob stores. Tiles are organized using the standard TMS key structure. Authentication works through Application Default Credentials (the recommended approach for GCP), API keys, or anonymously for emulators. All configuration parameters support environment variable expansion. Delete operations run asynchronously and use the GCS batch API for efficiency when removing tile ranges or entire layers. This keeps the main request threads responsive during bulk operations. Includes tests using testcontainers with fake-gcs-server, comprehensive javadocs, and user documentation.
1 parent ebc715a commit 7f498c5

19 files changed

+2781
-5
lines changed

‎.github/workflows/gcs-integration.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: GCS BlobStore Integration
2+
3+
on:
4+
push:
5+
tags:
6+
- "**"
7+
pull_request:
8+
paths:
9+
- ".github/workflows/gcs-integration.yml"
10+
- "pom.xml"
11+
- "geowebcache/pom.xml"
12+
- "geowebcache/core/**"
13+
- "geowebcache/gcsblob/**"
14+
15+
concurrency:
16+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
17+
cancel-in-progress: true
18+
19+
jobs:
20+
fake-gcs-server:
21+
name: Fake GCS Server container
22+
runs-on: ubuntu-latest
23+
strategy:
24+
matrix:
25+
java-version: [ 17, 21 ]
26+
steps:
27+
- uses: actions/checkout@v4
28+
- uses: actions/setup-java@v3
29+
with:
30+
distribution: 'temurin'
31+
java-version: ${{ matrix.java-version }}
32+
cache: 'maven'
33+
34+
- name: Tests against Fake GCS Server TestContainers
35+
run: |
36+
mvn verify -f geowebcache/pom.xml -pl :gcs-blob -am \
37+
-Ponline \
38+
-DskipTests=true \
39+
-DskipITs=false -B -ntp
40+
41+
- name: Remove SNAPSHOT jars from repository
42+
run: |
43+
find .m2/repository -name "*SNAPSHOT*" -type d | xargs rm -rf {}

‎documentation/en/user/source/configuration/storage.rst‎

Lines changed: 54 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,13 @@ Storage
1111
Cache
1212
-----
1313

14-
Starting with version 1.8.0, there are two types of persistent storage mechanisms for tiles:
14+
Starting with version 1.8.0, GeoWebCache supports multiple persistent storage mechanisms for tiles:
1515

16-
* File blob store: stores tiles in a directory structure consisting of various image files organized by layer and zoom level.
17-
* S3 blob store: stores tiles in an `Amazon Simple Storage Service <http://aws.amazon.com/s3/>`_ bucket, as individual "objects" following a
16+
* File blob store: stores tiles in a directory structure consisting of various image files organized by layer and zoom level.
17+
* S3 blob store: stores tiles in an `Amazon Simple Storage Service <http://aws.amazon.com/s3/>`_ bucket, as individual "objects" following a
1818
`TMS <http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification>`_-like key structure.
19+
* Google Cloud Storage blob store: stores tiles in a GCS bucket using the same TMS-like structure as S3.
20+
* Azure blob store, MBTiles blob store, Swift blob store: additional storage backends described below.
1921

2022
Zero or more blobstores can be configured in the configuration file to store tiles at different locations and on different storage back-ends.
2123
One of the configured blobstores will be the **default** one. Meaning that it will be used to store the tiles of every layer whose configuration
@@ -287,7 +289,54 @@ GeoServer ``topp:states`` sample layer on a fictitious ``my-geowebcache-bucket``
287289
zoom: 2
288290
})
289291
});
290-
292+
293+
294+
Google Cloud Storage (GCS) Blob Store
295+
+++++++++++++++++++++++++++++++++++++
296+
297+
This blob store allows to configure a cache for layers on a Google Cloud Storage bucket with the same TMS-like key structure as S3:
298+
299+
[prefix]/<layer id>/<gridset id>/<format id>/<parameters hash | "default">/<z>/<x>/<y>.<extension>
300+
301+
Configuration example:
302+
303+
.. code-block:: xml
304+
305+
<GoogleCloudStorageBlobStore default="false">
306+
<id>myGcsCache</id>
307+
<enabled>true</enabled>
308+
<bucket>my-gwc-bucket</bucket>
309+
<prefix>test-cache</prefix>
310+
<projectId>my-gcp-project</projectId>
311+
<useDefaultCredentialsChain>true</useDefaultCredentialsChain>
312+
</GoogleCloudStorageBlobStore>
313+
314+
Properties:
315+
316+
* **bucket**: Mandatory. The name of the GCS bucket where to store tiles.
317+
* **prefix**: Optional. A prefix path to use as the "root folder" to store tiles at.
318+
* **projectId**: Optional. The GCP project ID. Can be omitted if using service account credentials that already specify the project.
319+
* **quotaProjectId**: Optional. Project to bill for quota when using requester-pays buckets.
320+
* **endpointUrl**: Optional. Custom endpoint URL for use with GCS emulators or compatible services.
321+
* **useDefaultCredentialsChain**: Optional. Set to ``true`` to use Application Default Credentials. This will look for credentials in the following order: environment variable GOOGLE_APPLICATION_CREDENTIALS pointing to a service account key file, GCE/GKE metadata service, or gcloud CLI credentials.
322+
* **apiKey**: Optional. API key for authentication. If both apiKey and useDefaultCredentialsChain are provided, apiKey takes precedence.
323+
324+
**Note**: Like S3, all configuration properties support environment variable expansion using the ``${VARIABLE_NAME}`` syntax:
325+
326+
.. code-block:: xml
327+
328+
<bucket>${GCS_BUCKET}</bucket>
329+
<projectId>${GCS_PROJECT_ID}</projectId>
330+
331+
Authentication options:
332+
333+
* **Application Default Credentials** (recommended): Set ``useDefaultCredentialsChain`` to ``true``. This works automatically on GCE/GKE and when GOOGLE_APPLICATION_CREDENTIALS points to a service account key.
334+
* **API Key**: Set the ``apiKey`` property. Less secure, mainly for testing.
335+
* **No auth**: For use with emulators only. Leave both auth options unset.
336+
337+
Implementation notes:
338+
339+
Delete operations run asynchronously in a background thread pool. When deleting tile ranges or layers, tiles are removed in batches using the GCS batch API for efficiency. The thread pool is sized based on available processors and shuts down gracefully on blob store destruction.
291340

292341
Microsoft Azure Blob Store
293342
+++++++++++++++++++++++++++++++++++++++++++++
@@ -861,4 +910,4 @@ Additional Information:
861910

862911
* The package makes use of the open source multi-cloud toolkit `jclouds <https://jclouds.apache.org/>`_
863912
* Jclouds documentation for `getting started with Openstack <https://jclouds.apache.org/guides/openstack/>`_
864-
* Jclouds documentation for `OpenStack Keystone V3 Support <https://jclouds.apache.org/blog/2018/01/16/keystone-v3/>`_ used in config
913+
* Jclouds documentation for `OpenStack Keystone V3 Support <https://jclouds.apache.org/blog/2018/01/16/keystone-v3/>`_ used in config

‎geowebcache/gcsblob/README.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# GCS Blob Store
2+
3+
BlobStore implementation for Google Cloud Storage.
4+
5+
## Overview
6+
7+
This module provides a `BlobStore` that stores tiles in a GCS bucket. Tiles are organized using the standard TMS key structure: `<prefix>/<layer>/<gridset>/<format>/<parameters>/<z>/<x>/<y>.<ext>`
8+
9+
## Components
10+
11+
- `GoogleCloudStorageBlobStore` - Main BlobStore implementation
12+
- `GoogleCloudStorageClient` - Low-level GCS operations, handles batch deletes via background thread pool
13+
- `GoogleCloudStorageBlobStoreInfo` - XStream-serializable configuration
14+
- `GoogleCloudStorageConfigProvider` - Spring integration for config management
15+
16+
## Building
17+
18+
```bash
19+
mvn clean install
20+
```
21+
22+
## Testing
23+
24+
Unit tests run against a fake GCS server via testcontainers:
25+
26+
```bash
27+
mvn test
28+
```
29+
30+
Integration tests use the same fake GCS server but run through the failsafe plugin:
31+
32+
```bash
33+
mvn verify -Ponline
34+
```
35+
36+
## Configuration
37+
38+
Supports environment variable expansion in all config parameters. Example:
39+
40+
```xml
41+
<GoogleCloudStorageBlobStore>
42+
<id>gcs-store</id>
43+
<enabled>true</enabled>
44+
<bucket>${GCS_BUCKET}</bucket>
45+
<prefix>gwc</prefix>
46+
<projectId>${GCS_PROJECT_ID}</projectId>
47+
<useDefaultCredentialsChain>true</useDefaultCredentialsChain>
48+
</GoogleCloudStorageBlobStore>
49+
```
50+
51+
### Parameters
52+
53+
- `bucket` (required) - GCS bucket name
54+
- `prefix` (optional) - Path prefix within the bucket. If not set, operates at bucket root
55+
- `projectId` (optional) - GCP project ID
56+
- `quotaProjectId` (optional) - Project to bill for quota (for requester-pays buckets)
57+
- `endpointUrl` (optional) - Custom endpoint URL for emulators or GCS-compatible services
58+
- `useDefaultCredentialsChain` (optional) - Set to `true` to use Application Default Credentials
59+
- `apiKey` (optional) - API key for authentication
60+
61+
Authentication options (pick one):
62+
- `useDefaultCredentialsChain` - Uses Application Default Credentials
63+
- `apiKey` - API key for simple auth
64+
- No auth specified - Anonymous access (useful for emulators)
65+
66+
## Notes
67+
68+
Delete operations run asynchronously on a background thread pool sized to available processors. The pool shuts down gracefully on blob store destruction with a 60s timeout.

‎geowebcache/gcsblob/pom.xml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<parent>
5+
<groupId>org.geowebcache</groupId>
6+
<artifactId>geowebcache</artifactId>
7+
<version>2.0-SNAPSHOT</version>
8+
</parent>
9+
<artifactId>gcs-blob</artifactId>
10+
<name>Google Cloud Storage blob store</name>
11+
12+
<properties>
13+
<google-cloud-storage.version>2.55.0</google-cloud-storage.version>
14+
<testcontainers-fake-gcs-server.version>0.2.0</testcontainers-fake-gcs-server.version>
15+
</properties>
16+
17+
<dependencyManagement>
18+
<dependencies>
19+
<dependency>
20+
<groupId>io.aiven</groupId>
21+
<artifactId>testcontainers-fake-gcs-server</artifactId>
22+
<version>${testcontainers-fake-gcs-server.version}</version>
23+
<scope>test</scope>
24+
</dependency>
25+
</dependencies>
26+
</dependencyManagement>
27+
<dependencies>
28+
<dependency>
29+
<groupId>org.geowebcache</groupId>
30+
<artifactId>gwc-core</artifactId>
31+
<version>${project.version}</version>
32+
</dependency>
33+
<dependency>
34+
<groupId>com.google.cloud</groupId>
35+
<artifactId>google-cloud-storage</artifactId>
36+
<version>${google-cloud-storage.version}</version>
37+
</dependency>
38+
39+
<dependency>
40+
<groupId>org.mockito</groupId>
41+
<artifactId>mockito-core</artifactId>
42+
<scope>test</scope>
43+
</dependency>
44+
<dependency>
45+
<groupId>org.easymock</groupId>
46+
<artifactId>easymock</artifactId>
47+
<scope>test</scope>
48+
</dependency>
49+
<dependency>
50+
<groupId>org.geowebcache</groupId>
51+
<artifactId>gwc-core</artifactId>
52+
<classifier>tests</classifier>
53+
<scope>test</scope>
54+
</dependency>
55+
<dependency>
56+
<groupId>javax.servlet</groupId>
57+
<artifactId>javax.servlet-api</artifactId>
58+
<scope>provided</scope>
59+
</dependency>
60+
<dependency>
61+
<groupId>io.aiven</groupId>
62+
<artifactId>testcontainers-fake-gcs-server</artifactId>
63+
<scope>test</scope>
64+
</dependency>
65+
<dependency>
66+
<groupId>org.awaitility</groupId>
67+
<artifactId>awaitility</artifactId>
68+
<scope>test</scope>
69+
</dependency>
70+
</dependencies>
71+
<profiles>
72+
<profile>
73+
<id>online</id>
74+
<activation>
75+
<activeByDefault>false</activeByDefault>
76+
</activation>
77+
<build>
78+
<plugins>
79+
<plugin>
80+
<artifactId>maven-failsafe-plugin</artifactId>
81+
<configuration>
82+
<forkCount>1</forkCount>
83+
<reuseForks>false</reuseForks>
84+
</configuration>
85+
</plugin>
86+
</plugins>
87+
</build>
88+
</profile>
89+
</profiles>
90+
</project>

‎geowebcache/gcsblob/src/main/java/org/geowebcache/storage/blobstore/gcs/CacheId.java‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
/**
2+
* This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General
3+
* Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
4+
* later version.
5+
*
6+
* <p>This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
7+
* warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
8+
*
9+
* <p>You should have received a copy of the GNU Lesser General Public License along with this program. If not, see
10+
* <http://www.gnu.org/licenses/>.
11+
*
12+
* @author Gabriel Roldan, Camptocamp, Copyright 2025
13+
*/
14+
package org.geowebcache.storage.blobstore.gcs;
15+
16+
import org.geowebcache.mime.MimeType;
17+
18+
/**
19+
* A record to hold the components of a tile's cache identity.
20+
*
21+
* <p>Note {@code layerName} is used to provide the layer name to callbacks (see
22+
* {@link GoogleCloudStorageBlobStore#sendTileDeleted(TileLocation, long)}, may the layer id be different than the layer
23+
* name like in GeoServer tile layers. For all other purposes, {@code layerId} uniquely identifies the layer (e.g. for
24+
* layer cache prefixes)
25+
*
26+
* @param layerId The unique identifier of the layer.
27+
* @param layerName The name of the layer.
28+
* @param gridsetId The identifier of the gridset.
29+
* @param format The MIME type of the tile.
30+
* @param parametersId The identifier for the tile's parameter set, can be {@code null}.
31+
* @since 1.28
32+
*/
33+
record CacheId(String layerId, String layerName, String gridsetId, MimeType format, String parametersId) {}

0 commit comments

Comments
 (0)