Feat: Introduce ExtendedGcsFileSystem for Zonal Bucket gRPC Read Path #707

ankitaluthra1 · 2025-11-02T06:29:19Z

This pull request introduces base skeleton for adding different types of storages and would be behind experimental flag until full support for new storage types is added.

Key Changes:

GCSFileSystemAdapter: Added gcsfs/gcsfs_adapter.py to subclass GCSFileSystem. This adapter dynamically handles different bucket types using gcs storage layout api, directing operations to the appropriate file implementation.
gRPC Integration: Integrated the asynchronous gRPC client within the adapter and ZonalFile.
ZonalFile: Created gcsfs/zonal_file.py as a subclass of GCSFile. This class is specifically designed for interacting with GCS Zonal Buckets. The existing GCSFile from gcsfs.core continues to be used for all non-zonal bucket operations within the adapter, ensuring backward compatibility for standard bucket types.
Experimental Flag: The new Zonal Bucket/HNS features will only be enabled via an experimental_zb_hns_support flag passed to the GCSFileSystem constructor. The existing gcsfs.core.GCSFileSystem remains the foundation. When the experimental_zb_hns_support flag is not passed or is False, an instance of GCSFileSystem is created as usual, retaining all original functionality and read/write methods.
Utilities: Added gcsfs/zb_hns_utils.py for helper functions for gcsfs_adapter and zonal file.
Testing: Unit tests are added corresponding to newly added files.

…pecial type

…onal

…s new tests

…e developed - experimental_zb_hns_support

Extend gcsfs to create new filesystem. async download

…onously

Merged GCSFileSystemAdapter and ZonalFile POC

Rename gcshnsfilesystem to GCSFileSystemAdapter

Feat: Implement gRPC Read Path for Zonal Bucket

…t being used. Updated zonal tests to work with both real and fake gcs Added more tests for zonal path reads

Feat: Zonal Read optimizations and test suite

Internal main to main

* Add mrd closing logic in ZonalFile and GCSFSAdapter Add exception handling in GCSFile fetch_range * Fix gcs_adapter tests to use zonal mocks add test to validate mrd stream is closed with GCSFile closing * FIx: use patch for auth in gcs_adapter fixture for fake gcs only

martindurant

Preliminary comments, not yet having looked through the main code of "adapter"

gcsfs/core.py

gcsfs/zonal_file.py

gcsfs/tests/conftest.py

gcsfs/tests/test_core.py

gcsfs/tests/test_extended_gcsfs.py

martindurant · 2025-11-06T21:43:07Z

gcsfs/tests/test_extended_gcsfs.py

+def zonal_mocks():
+    """A factory fixture for mocking Zonal bucket functionality."""
+
+    @contextlib.contextmanager


Why make a context in a fixture, which already acts like a context?

The inner function allows us to pass different file_data parameters to the mock for each specific call which fetches the correct rvalue for assertion. Also, Some tests has write operations. Since Zonal currently supports reads only, the context manager helps to apply the zonal mocks only during the read portion of the test. If the mocks were active for the entire test duration, the write operations would fail. This seemed the best way to achieve both of the requirements without duplicating mock setup code across every test.

Hi, following up on this. The context manager lets us cleanly scope the mocks to the read phase of the test. Does this approach seem reasonable, or should we switch to adding separate mocks for the tests with write?

gcsfs/tests/test_extended_gcsfs.py

gcsfs/zb_hns_utils.py

…est fixtures and pytest treats this as error on ci

martindurant · 2025-11-10T13:05:12Z

What I am suggesting, is that import of the adapter modure should call register_implementation with the new class, replacing GCSFileSystem in the registry.

The environment variable would be used like:

if os.getenv(..):
    import gcsfs.adapter

in __init__.

For the sake of tests, you could either make a fixture that ensures the right class is registered for each test function/module, or make a separate CI run with the environment variable set.

…Test Mocking (#8) * Add mrd closing logic in ZonalFile and GCSFSAdapter Add exception handling in GCSFile fetch_range * Fix gcs_adapter tests to use zonal mocks add test to validate mrd stream is closed with GCSFile closing * FIx: use patch for auth in gcs_adapter fixture for fake gcs only * Change imports to absolute imports * Rename GcsFileSystemAdapter to ExtendedGcsFileSystem * Mock _sync_get_bucket_type call to return UNKNOWN while opening a file in write mode * Updated ExtendedGcsFileSystem class description Separated cleanup logic from extended_gcsfs fixture for better readability

* registers ExtendedGCSFileSystem when experimental variable is set instead of __new__ hack * fixes registry race condition * removes unnecessary registry * adds separate ci tests * adds test_init.py * fixes pre-commit

gcsfs/__init__.py

removed redundant environment variable setting removed try block for creating TEST_BUCKET to expose errors in test setup

martindurant

Final review before merging this. Most question - you can mark comments in the code, if you intend to come back and change things in a future PR.

One question: can bucket types change? Do we need an explicit way to clear the bucket type cache?

martindurant · 2025-11-13T14:59:14Z

gcsfs/__init__.py

+        )
+        # Fallback to core GCSFileSystem, do not register here
+
 __all__ = ["GCSFileSystem", "GCSMap"]


It may not be important, but GCSMap will still refer to the original GCSFileSystem regardless of the branch taken above.

Currently zonal feature is defaulted to false, so this shouldn't cause any issue. I've added a todo to handle this use case in future PRs once we have more clarity on this

gcsfs/zb_hns_utils.py

martindurant · 2025-11-13T16:28:39Z

gcsfs/zonal_file.py

+            self.mrd = asyn.sync(
+                self.gcsfs.loop, self._init_mrd, self.bucket, self.key, self.generation
+            )


Is it possible to defer this until we're actually writing something?

mrd is short for Multi Range Downloader which uses gRPC to download objects and it cannot be used for writing data. mrd is initialized (underlying stream is created and opened) during ZonalFile.open() so that the mrd is ready to use for reading after opening ZonalFile.

martindurant · 2025-11-13T16:30:50Z

gcsfs/zonal_file.py

+        if self.mrd:
+            asyn.sync(self.gcsfs.loop, self.mrd.close)
+        super().close()


Doesn't super().close() potentially cause a write? So the .mrd should be closed second, not first. Maybe this is not important when we are only implementing read mode.

As mentioned in the previous comment, mrd is used for reads/ downloads only and not for writing. Hence, mrd can be closed first without any issue.

gcsfs/tests/test_zb_hns_utils.py

gcsfs/extended_gcsfs.py

* remove redundant create mrd method from zb_hns_utils * Added mrd to _cat_file method signature and added description * Optimized _process_limits_to_offset_and_length method to only call info when size is None. * Add todo to update GCSMap to use correct implementation

martindurant · 2025-11-18T17:29:28Z

Please let me know when you are done with my comments, and I shall merge.

ankitaluthra1 · 2025-11-18T18:39:41Z

Please let me know when you are done with my comments, and I shall merge.

Done with comments, PTAL

martindurant · 2025-11-19T16:00:39Z

@ankitaluthra1 - this PR is causing the CI in fsspec to fail ( fsspec/filesystem_spec#1944 ). It seems extended (mrd) tests are being run in combination with the original GCSFileSystem. Can you please investigate?

ankitaluthra1 · 2025-11-20T17:40:36Z

@ankitaluthra1 - this PR is causing the CI in fsspec to fail ( fsspec/filesystem_spec#1944 ). It seems extended (mrd) tests are being run in combination with the original GCSFileSystem. Can you please investigate?

Added the fix, PTAL - #712

ankitaluthra1 and others added 30 commits October 1, 2025 06:27

adds logic for handling requests from new adapter is bucket type os s…

ecbec0e

…pecial type

adds logic for hnadling requests from new adapter is bucket type is z…

d506a60

…onal

refactor bucket type enum to include value for HNS bucket type | Fixe…

9a70933

…s new tests

adds feature toggle behind which experimental feature support would b…

eb209f5

…e developed - experimental_zb_hns_support

updates tests to include experimental feature toggle

0b606b1

minor fix

409c86e

moves mrd creation inside core.py

8a4ca85

moves mrd creation to separate GCSFile

c5a9eae

minor comment

e95495f

Extend gcsfs to create new filesystem

95440d5

Extend gcsfs to create new filesystem. async download

Extend gcsfs and gcsfile to override methods. download happens synchr…

dfcbb7f

…onously

Merge pull request from ankitaluthra1/lankita-poc-zonal

f3d1031

Merged GCSFileSystemAdapter and ZonalFile POC

fixes new test in test_core.py

20b36fc

refactors/renames GCS Filesystem Adapter

c9f569a

Override _cat_file to handle other read methods of gcsfs

e5fc935

creates grpc client inside GCSHNSFilesystem

50d30b3

refactors to reuse grpc client in cat_ranges

cbf00b3

Move classmethods in ZonalFile to a util file

a202579

Implement logic to process limits as offset and length

5064a97

Add fallback logic for non zonal buckets

ac276f6

Rename gcshnsfilesystem to GCSFileSystemAdapter

Add unit tests for zb_hns_utils

dbecb12

Add test for GCSFSAdapter with read_block

37a495b

Update _get_storage_layout to use a single return statement

cba7d27

Updated gcs_adapter.open to pass on correct default values to GCSFile

4ff9fe4

Move logic for handing 0 length in MRD to zb_hns_utils

389a4b0

Add comments for clarity

133b4fa

Merge pull request #4 from ankitaluthra1/zb-features

31b2a2f

Feat: Implement gRPC Read Path for Zonal Bucket

Updated zonal file to only create mrd for read mode.

e36b7ed

Updated gcs_adapter fixture to not setup bucket when real gcs endpoin…

25cd0ef

…t being used. Updated zonal tests to work with both real and fake gcs Added more tests for zonal path reads

Updated gcs_adapter.open to pass on correct default values to GCSFile

8425464

ankitaluthra1 and others added 11 commits November 2, 2025 11:54

Merge pull request #5 from ankitaluthra1/zb-features

7983528

Feat: Zonal Read optimizations and test suite

Merge pull request from ankitaluthra1/internal-main

cdf574a

Internal main to main

Merge branch 'fsspec:main' into main

b411bef

fixes lint errors

c47b53f

fixes lint errors

e37d0fb

fixes conda install error

11af000

mocks fake test credentials for grpc client

c4cf777

fixes conflicting lint rules black and isort

5714012

adds missing pytest package in conda install

fccd43a

refactor get bucket type

11dd722

martindurant reviewed Nov 7, 2025

View reviewed changes

ankitaluthra1 added 2 commits November 9, 2025 10:14

adds GCSFS_EXPERIMENTAL_ZB_HNS_SUPPORT as env variable instead of kwargs

0c3df8e

removes timeout from pytest fixture coming as mark has no effect on t…

0e9c2ab

…est fixtures and pytest treats this as error on ci

ankitaluthra1 changed the title ~~Feat: Introduce GCSFileSystemAdapter for Zonal Bucket gRPC Read Path~~ Feat: Introduce ExtendedGcsFileSystem for Zonal Bucket gRPC Read Path Nov 12, 2025

Replaces __new__ with conditional import in init

a8945c2

* registers ExtendedGCSFileSystem when experimental variable is set instead of __new__ hack * fixes registry race condition * removes unnecessary registry * adds separate ci tests * adds test_init.py * fixes pre-commit

martindurant reviewed Nov 12, 2025

View reviewed changes

gcsfs/__init__.py Show resolved Hide resolved

ankitaluthra1 and others added 3 commits November 14, 2025 16:51

Merge branch 'main' into main

774b53d

fixes lint errors

90d0cf4

simplified logic in cleanup_gcs for unit tests

8672c28

removed redundant environment variable setting removed try block for creating TEST_BUCKET to expose errors in test setup

martindurant reviewed Nov 17, 2025

View reviewed changes

ankitaluthra1 and others added 2 commits November 18, 2025 12:08

refactored get_bucket_type

5398cfe

martindurant merged commit 47c2d76 into fsspec:main Nov 18, 2025
6 checks passed

Feat: Introduce ExtendedGcsFileSystem for Zonal Bucket gRPC Read Path #707

Feat: Introduce ExtendedGcsFileSystem for Zonal Bucket gRPC Read Path #707

Uh oh!

Conversation

ankitaluthra1 commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martindurant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suni72 Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

martindurant commented Nov 10, 2025

Uh oh!

Uh oh!

martindurant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martindurant commented Nov 18, 2025

Uh oh!

ankitaluthra1 commented Nov 18, 2025

Uh oh!

Uh oh!

martindurant commented Nov 19, 2025

Uh oh!

ankitaluthra1 commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ankitaluthra1 commented Nov 2, 2025 •

edited

Loading

suni72 Nov 10, 2025 •

edited

Loading

ankitaluthra1 commented Nov 20, 2025 •

edited

Loading