Skip to content

Refactor file range processing and update logging in mrd#790

Merged
ankitaluthra1 merged 13 commits intofsspec:mainfrom
ankitaluthra1:process-limit-fix
Mar 26, 2026
Merged

Refactor file range processing and update logging in mrd#790
ankitaluthra1 merged 13 commits intofsspec:mainfrom
ankitaluthra1:process-limit-fix

Conversation

@suni72
Copy link
Copy Markdown
Contributor

@suni72 suni72 commented Mar 26, 2026

Key changes:

  • Enhanced File Range Processing:
    • The _cat_file method now explicitly returns empty bytes when start >= end >= 0 to prevent invalid HTTP range requests.
    • The logic for start and end parameters in _cat_file and _process_limits_to_offset_and_length has been refined to correctly handle 0 values and negative indices.
    • Invalid ranges (e.g., start > end) are now gracefully handled by returning an empty range instead of raising ValueError in _process_limits_to_offset_and_length.
    • Negative start offsets are clamped to 0 if they would result in an offset before the beginning of the file in _process_limits_to_offset_and_length.
  • Multi-Range Download Refactoring:
    • The multi-range download logic has been extracted into a new download_ranges function in gcsfs/zb_hns_utils.py.
    • This new function includes improved logging for short reads and validates the number of requested ranges against MRD_MAX_RANGES (1000).
  • Zonal Bucket Merge Restrictions:
    • A new _merge method has been added to extended_gcsfs.py which explicitly raises a NotImplementedError if an attempt is made to merge objects in a Zonal Bucket, as server-side compose is not supported for them. For standard buckets, it delegates to the parent class's _merge implementation.
  • Improved Logging:
    • Warning messages in _fetch_range_split, _cat_file, and _get_file now include the relevant file path for better debugging.
  • Test Updates:
    • Unit tests for _process_limits_to_offset_and_length have been updated to reflect the new behavior, and new tests for _merge and download_ranges have been added.

@ankitaluthra1
Copy link
Copy Markdown
Collaborator

/gcbrun

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 26, 2026

Codecov Report

❌ Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.30%. Comparing base (fee3945) to head (97e0423).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
gcsfs/core.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #790      +/-   ##
==========================================
+ Coverage   75.84%   76.30%   +0.46%     
==========================================
  Files          14       14              
  Lines        2645     2663      +18     
==========================================
+ Hits         2006     2032      +26     
+ Misses        639      631       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@suni72 suni72 marked this pull request as ready for review March 26, 2026 09:15
@ankitaluthra1
Copy link
Copy Markdown
Collaborator

/gcbrun

@ankitaluthra1 ankitaluthra1 merged commit fe2072a into fsspec:main Mar 26, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants