Skip to content

Commit 68c9fd0

Browse files
ankitaluthra1ankitaluthra1
authored
Zonal documentation (#781)
1 parent 803652e commit 68c9fd0

File tree

2 files changed

+218
-0
lines changed

2 files changed

+218
-0
lines changed

‎docs/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,7 @@ Contents
238238
api
239239
developer
240240
hns_buckets
241+
rapid_storage_support
241242
fuse
242243
changelog
243244
code-of-conduct

‎docs/source/rapid_storage_support.rst‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
=============================
2+
Rapid Storage (Zonal Buckets)
3+
=============================
4+
5+
To accelerate data-intensive workloads such as AI/ML training, model checkpointing, and analytics, Google Cloud Storage (GCS) offers **Rapid Storage** through **Zonal Buckets**.
6+
7+
``gcsfs`` provides full support for accessing, reading, and writing to Rapid Storage buckets.
8+
9+
What is Rapid Storage?
10+
----------------------
11+
12+
Rapid Storage is a storage class designed for **high-performance data access** in Cloud Storage.
13+
Unlike standard buckets that span an entire region or multi-region, Rapid buckets are **Zonal**—they are located within a specific Google Cloud zone. This allows you to co-locate your storage with your compute resources (such as GPUs or TPUs),
14+
resulting in significantly lower latency and higher throughput.
15+
16+
Key capabilities include:
17+
18+
* **Low latency and high throughput:** Ideal for data-intensive AI/ML, evaluating models, and logging.
19+
* **Native Appends:** You can natively append data to objects.
20+
* **Immediate Visibility:** Appendable objects appear in the bucket namespace as soon as you start writing to them and can be read concurrently.
21+
22+
You can find detailed documentation on zonal bucket here: https://docs.cloud.google.com/storage/docs/rapid/rapid-bucket.
23+
24+
Using Rapid Storage with ``gcsfs``
25+
----------------------------------
26+
Rapid Storage is fully supported by gcsfs without any code changes needed .To interact with Rapid Storage,
27+
the underlying filesystem operations will automatically route through
28+
the newly added ``ExtendedFileSystem`` designed to support multiple storage types like HNS and Rapid.
29+
You can interact with these buckets just like any other filesystem.
30+
31+
**Code Example**
32+
33+
.. code-block:: python
34+
35+
import gcsfs
36+
37+
# Initialize the filesystem
38+
fs = gcsfs.GCSFileSystem()
39+
40+
# Writing to a Rapid bucket
41+
with fs.open('my-zonal-rapid-bucket/data/checkpoint.pt', 'wb') as f:
42+
f.write(b"model data...")
43+
44+
# Appending to an existing object (Native Rapid feature)
45+
with fs.open('my-zonal-rapid-bucket/data/checkpoint.pt', 'ab') as f:
46+
f.write(b"appended data...")
47+
48+
Under the Hood: The ``ExtendedFileSystem`` and ``ZonalFile``
49+
------------------------------------------------------------
50+
51+
`gcsfs` enables Rapid Storage support through the ``ExtendedFileSystem`` and a specialized ``ZonalFile`` file handler. Both ``ExtendedFileSystem`` and ``ZonalFile`` inherits same semantics as existing ``GCSFileSystem`` and ``GCSFile``
52+
making Rapid support fully backward compatible for all operations.
53+
54+
At initialization, ``ExtendedFileSystem`` evaluates the underlying bucket's storage layout. If it detects Rapid storage, file-level operations are dynamically routed to the ``ZonalFile`` class instead of the standard ``GCSFile``.
55+
56+
Unlike standard operations which use HTTP endpoints, ``ZonalFile`` utilizes the Google Cloud Storage gRPC API—specifically the ``AsyncMultiRangeDownloader`` (MRD) for reads and ``AsyncAppendableObjectWriter`` (AAOW) for writes.
57+
58+
Operation Semantics: Standard vs. Rapid Storage
59+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
60+
61+
The table below highlights how core filesystem and file-level operations change when interacting with a Rapid Zonal bucket compared to a standard GCS bucket.
62+
63+
.. list-table::
64+
:widths: 15 40 45
65+
:header-rows: 1
66+
67+
* - Class / Method
68+
- Standard Storage (``GCSFileSystem`` / ``GCSFile``)
69+
- Rapid Storage (``ExtendedFileSystem`` / ``ZonalFile``)
70+
* - **open(mode='a')**
71+
- **Not supported.** Overwritten to ``w`` mode with a warning.
72+
- **Supported.** Natively opens an append stream to the object via gRPC.
73+
* - **ExtendedFileSystem._open**
74+
- Returns a standard ``GCSFile`` instance.
75+
- Returns a ``ZonalFile`` instance, initializing gRPC streams.
76+
* - **cat_file`` / ``_fetch_range**
77+
- Uses standard HTTP GET range requests.
78+
- Uses gRPC `AsyncMultiRangeDownloader <https://github.com/googleapis/python-storage/blob/8b7fbde10c80337c4b4a2f6c8a860e28371a770b/google/cloud/storage/asyncio/async_multi_range_downloader.py#L92>`_ (MRD) for parallel byte-range fetching.
79+
* - **get_file**
80+
- Downloads using standard HTTP GET requests.
81+
- Downloads via gRPC MRD in configurable chunks.
82+
* - **put_file / pipe_file**
83+
- Uses HTTP multipart or resumable uploads.
84+
- Uses Bidirectional RPC (`AsyncAppendableObjectWriter <https://github.com/googleapis/python-storage/blob/8b7fbde10c80337c4b4a2f6c8a860e28371a770b/google/cloud/storage/asyncio/async_appendable_object_writer.py#L102>`_) for direct, high-performance writes.
85+
* - **cp_file (Copy)**
86+
- Server-side rewrite (``rewriteTo`` API).
87+
- **Not supported.** Raises ``NotImplementedError`` as zonal objects do not support rewrites.
88+
* - **write / flush**
89+
- Buffers locally and uploads chunks via HTTP POST when flushed.
90+
- Streams chunks directly to the AAOW stream for immediate persistence.
91+
* - **discard**
92+
- Cancels an in-progress HTTP multi-upload and cleans up.
93+
- **Not applicable.** Logs a warning since streaming data cannot be canceled.
94+
* - **close**
95+
- Finalizes the file upload to GCS.
96+
- Closes streams but leaves the object unfinalized (appendable) by default, unless ``finalize_on_close=True`` or `.commit()` is done.
97+
* - **mv**
98+
- Object-level copy-and-delete logic.
99+
- Uses native, atomic ``rename_folder`` API for folders. All directory semantics described in the :doc:`HNS documentation <hns_buckets>` also apply For Rapid.
100+
101+
Performance Benchmarks
102+
----------------------
103+
104+
Rapid Storage via gRPC significantly improves read and write performance compared to standard HTTP regional buckets.
105+
Here are the microbenchmarks
106+
Rapid drastically outperform standard buckets across different read patterns, including both sequential and random reads, as well as for writes.
107+
To reproduce using more combinations, please see `gcsfs/perf/microbenchmarks`
108+
109+
.. list-table:: **Sequential Reads**
110+
:header-rows: 1
111+
112+
* - IO Size
113+
- Processes
114+
- Rapid (MB/s)
115+
- Standard (MB/s)
116+
* - 1 MB
117+
- Single Process
118+
- 469.09
119+
- 37.76
120+
* - 16 MB
121+
- Single Process
122+
- 628.59
123+
- 64.50
124+
* - 1 MB
125+
- 48 Processes
126+
- 16932
127+
- 2202
128+
* - 16 MB
129+
- 48 Processes
130+
- 19213.27
131+
- 4010.50
132+
133+
.. list-table:: **Random Reads**
134+
:header-rows: 1
135+
136+
* - IO Size
137+
- Processes
138+
- Rapid Throughput (MB/s)
139+
- Standard (MB/s)
140+
* - 64 KB
141+
- Single Process
142+
- 39
143+
- 0.77
144+
* - 16 MB
145+
- Single Process
146+
- 602.12
147+
- 66.92
148+
* - 64 KB
149+
- 48 Processes
150+
- 2081
151+
- 51
152+
* - 16 MB
153+
- 48 Processes
154+
- 21448
155+
- 4504
156+
157+
.. list-table:: **Writes**
158+
:widths: 20 20 30 30
159+
:header-rows: 1
160+
161+
* - IO Size
162+
- Processes
163+
- Zonal Bucket
164+
- Regional Bucket
165+
* - 16 MB
166+
- Single Process
167+
- 326
168+
- 100
169+
* - 16 MB
170+
- 48 Processes
171+
- 13418
172+
- 4722
173+
174+
Multiprocessing and gRPC
175+
----------------------------------------------
176+
177+
Because `gcsfs` relies on gRPC to interact with Rapid storage, developers must be careful when using multiprocessing. Users use libraries such as multiprocessing, subprocess, concurrent.futures.ProcessPoolExecutor, etc, to work around the GIL. These modules call `fork()` underneath the hood.
178+
179+
However, gRPC Python wraps gRPC core, which uses internal multithreading for performance, and hence doesn't support `fork()`.
180+
Using `fork()` for multi-processing can lead to hangs or segmentation faults when child processes attempt to use the network layer
181+
where the application creates gRPC Python objects (e.g., client channel)before invoking `fork()`. However, if the application only
182+
instantiate gRPC Python objects after calling `fork()`, then `fork()` will work normally, since there is no C extension binding at this point.
183+
184+
**Alternative: Use `forkserver` or `spawn` instead of `fork`**
185+
186+
To resolve `fork` issue, you can use `forkserver` or `spawn` instead of `fork` where the child process will create their own grpc connection.
187+
You can configure Python's `multiprocessing` module to override the start method as shown in the snippet below.
188+
For example while using data loaders in frameworks like PyTorch
189+
(e.g., `torch.utils.data.DataLoader` with `num_workers > 0`) alongside `gcsfs` with Rapid storage:
190+
191+
.. code-block:: python
192+
193+
# Use forkserver
194+
import torch.multiprocessing
195+
# This must be done before other imports or initialization
196+
try:
197+
torch.multiprocessing.set_start_method('forkserver', force=True)
198+
# or use torch.multiprocessing.set_start_method('forkserver', force=True)
199+
except RuntimeError:
200+
pass # Context already set
201+
202+
* **forkserver (Recommended for Performance):** Starts a single, clean server process at the beginning of your program. Subsequent workers are forked from this clean state. This is much faster than `spawn` because it avoids re-initializing the Python interpreter and re-importing libraries for every worker. Note that `forkserver` is only available on Unix platforms.
203+
* **spawn (Recommended for Maximum Safety/Compatibility):** Starts a completely fresh Python interpreter for each worker. While this incurs a high startup latency and memory overhead ("import tax"), it is 100% immune to inheriting locked mutexes from background threads. It is also fully cross-platform (works on Windows).
204+
205+
Important Differences to Keep in Mind
206+
-------------------------------------
207+
208+
When working with Rapid Storage in ``gcsfs``, keep the following GCS limitations and behaviors in mind:
209+
210+
1. **HNS Requirement:** You cannot use Rapid Storage without a Hierarchical Namespace. All directory semantics described in the :doc:`HNS documentation <hns_buckets>` apply here (e.g., real folder resources, strict ``mkdir`` behavior).
211+
2. **Append Semantics:** In a standard flat GCS bucket, appending to a file is typically a costly operation requiring a full object download and rewrite. Rapid Storage supports **native appends**. When you open a file in append mode (``ab``), ``gcsfs`` natively appends to the object and the object's size grows in real-time.
212+
3. **Single Writer:** Appendable objects can only have one active writer at a time. If a new write stream is established for an object, the original stream is interrupted and will return an error from Cloud Storage.
213+
4. **Finalization:** Once an object is finalized (e.g., the write stream is closed), you can no longer append to it. To take advantage of `native appends`, gcsfs keeps the object unfinalized by default.
214+
5. **Incompatibilities:** Zonal buckets currently do not support certain standard GCS features, full list is here: https://docs.cloud.google.com/storage/docs/rapid/rapid-bucket#incompatibilities
215+
216+
217+
For more details on managing, pricing, and optimizing these buckets, refer to the official documentation for `Rapid Bucket <https://cloud.google.com/storage/docs/rapid/rapid-bucket>`_.

0 commit comments

Comments
 (0)