feat(zb-write): Support write mode in Zonal File #16

suni72 · 2025-11-21T12:37:51Z

Key Changes

_ init_: Updated to initialize AsyncAppendableObjectWriter (AAOW) when the file mode is write-enabled ('w' mode).
_init_aaow: Added a helper method to initialize the AAOW.
write: Overriden to write data to gcs using AAOW.
flush: Overridden to flush data from AAOW instead of initiating a Multi-Part Upload (MPU).
close: Overridden to ensure the AAOW is closed properly.
- If autocommit=True (default), the object is finalized on close.
- If autocommit=False, the object remains appendable.
discard: Reimplemented to log a warning ("Discard is unavailable for Zonal Buckets..."), as Bidi streams cannot be discarded/cancelled once written.
Test: Added unit tests for new methods in zb_hns_utils and zonal_file
Fix: Fixed auth issue with grpc client when using anon token

… generation=None to overwrite

added unit tests for zonal file methods changed zonal_mocks to mock _get_bucket_type instead of mocking two methods

This reverts commit d51c8c3.

ankitaluthra1

Adding initial comments to get started on resolution, yet to look at zonal_file and zb_hns_utils.

ankitaluthra1 · 2025-11-21T17:04:40Z

.github/workflows/ci.yml

          --log-date-format="%H:%M:%S" \
          gcsfs/ \
-          --ignore=gcsfs/tests/test_extended_gcsfs.py
+          --ignore=gcsfs/tests/test_extended_gcsfs.py \


Instead of adding ignore on CI, please do the same in tests itself as added in main
reason: fsspec also run all gcsfs tests instead of adding --ignore in all ci rules, should be contained in test itself

Done. Added check for experimental env variable in the test file

ankitaluthra1 · 2025-11-21T17:06:40Z

gcsfs/core.py

+    from .extended_gcsfs import ExtendedGcsFileSystem
+    from .extended_gcsfs import upload_chunk as ext_upload_chunk
+
+    if isinstance(fs, ExtendedGcsFileSystem) and isinstance(


Lets add clear documentation here whats the new behaviour, condition is added instead of overriding functionality in ExtendedFileSystem to avoid change in imports etc. Same for similar methods

Added.
Conditional logic is used here since these methods don't belong to a class, so override was not possible.

ankitaluthra1 · 2025-11-21T17:08:19Z

gcsfs/extended_gcsfs.py

        self.grpc_client = None
        self.storage_control_client = None
+        self.credential = self.credentials.credentials
+        if self.credentials.token == "anon":


lets add documentation so reader is clear why only anon has to be added differently, something like anon is used tests to bypass credentials

since this is separate from writes, does it make sense to seggregate the PRs so this can be merged easily. Also verify if all existing also tests run with experimental true, AFAIK creds was the only issue ?

ankitaluthra1 · 2025-11-21T17:26:15Z

gcsfs/extended_gcsfs.py

+
+async def upload_chunk(fs, location, data, offset, size, content_type):
+    raise NotImplementedError(
+        "upload_chunk is not implemented yet for ExtendedGcsFileSystem. Please use write() instead."


upload_chunk is not implemented yet for ExtendedGcsFileSystem. Please use write() instead.
rephrase to ? upload_chunk is not implemented yet for zonal experimental feature. Please use write() instead.

ankitaluthra1 · 2025-11-22T08:26:55Z

gcsfs/zonal_file.py

+
+    def _initiate_upload(self):
+        """Initiates the upload for Zonal buckets using gRPC."""
+        from gcsfs.extended_gcsfs import initiate_upload


Is this import inline to avoid cyclic import error ? If yes, is it better to move ZonalFile class inside extended_filesystem.py similar to exisiting core.py ? or seggregating it is more readable ?

Yes, it is to avoid cyclic import error. I think it is better to keep the zonal features separated for readability since extended_filesystem will have HNS implementation as well.

ankitaluthra1 · 2025-11-22T08:28:50Z

gcsfs/zonal_file.py


+    def write(self, data):
+        """
+        Writes data using AsyncAppendableObjectWriter.


Lets add link for AsyncAppendableObjectWriter

ankitaluthra1 · 2025-11-22T08:29:27Z

gcsfs/zonal_file.py

+    def discard(self):
+        """Discard is not applicable for Zonal Buckets. Log a warning instead."""
+        logger.warning(
+            "Discard is unavailable for Zonal Buckets. \


unavailable change this to not applicable

ankitaluthra1 · 2025-11-22T08:36:20Z

gcsfs/core.py

+    from .extended_gcsfs import ExtendedGcsFileSystem
+    from .extended_gcsfs import initiate_upload as ext_initiate_upload
+
+    if isinstance(fs, ExtendedGcsFileSystem) and await fs._is_zonal_bucket(bucket):


why isinstance needed here? isnt is_zonal check sufficient to trigger new functionality

is_zonal_bucket method belongs to ExtendedGcsFileSystem. So the first check is there to make sure is_zonal_bucket method is found in the class.

ankitaluthra1

/gcbrun

… the bucket

suni72 added 10 commits November 21, 2025 12:14

Support write mode in ZonalFile and override related methods

cce7e99

Add unit test for init_aaow method in zb_hns_utils

1a855e0

Implement zonal write methods

c38f1ee

Remove overwrite parameter since it is not added in AAOW. instead use…

c104f3d

… generation=None to overwrite

Remove redundant init_aaow method from zonal_file

8427749

update statement for NotImplementedErrors

64f2513

set autocommit=false as default for ZonalFile

feea4d3

add logic to route upload methods to zonal implementation

0c719f5

Fixed auth issue when using token="anon"

ca976d6

added unit tests for zonal file methods changed zonal_mocks to mock _get_bucket_type instead of mocking two methods

Revert "set autocommit=false as default for ZonalFile"

866d61e

This reverts commit d51c8c3.

suni72 force-pushed the zb-write-copy branch from 42a8eac to 3bf77d9 Compare November 21, 2025 12:40

Update ci pipeline to run test_zonal_file with Extended tests only

9fdcbe0

suni72 force-pushed the zb-write-copy branch from 3bf77d9 to 9fdcbe0 Compare November 21, 2025 12:43

suni72 requested review from ankitaluthra1 and jasha26 November 21, 2025 12:45

ankitaluthra1 requested changes Nov 21, 2025

View reviewed changes

ankitaluthra1 requested changes Nov 22, 2025

View reviewed changes

ankitaluthra1 reviewed Nov 25, 2025

View reviewed changes

Mahalaxmibejugam and others added 5 commits November 26, 2025 12:49

Separate versioned and non-versioned tests to use different bucket

81fb615

Update cleanup logic in tests to empty the bucket instead of deleting…

5112276

… the bucket

Merge remote-tracking branch 'origin/main' into zb-write-copy

0c7ae3f

add logic to skip test in test_file itself

83581b2

Added comments for clarity

bed5bc1

suni72 force-pushed the zb-write-copy branch 2 times, most recently from 816e73b to 95dd97b Compare December 4, 2025 16:35

Merge branch 'main' into zb-write-copy

57ca20a

suni72 force-pushed the zb-write-copy branch from 95dd97b to 57ca20a Compare December 4, 2025 16:49

feat(zb-write): Support write mode in Zonal File #16

Are you sure you want to change the base?

feat(zb-write): Support write mode in Zonal File #16

Uh oh!

Conversation

suni72 commented Nov 21, 2025

Uh oh!

ankitaluthra1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankitaluthra1 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants