Skip to content

Conversation

aabmass
Copy link
Contributor

@aabmass aabmass commented Sep 24, 2025

Calling put with lists like put([a], [b]) avoids the expensive path expansion in AbstractFileSystem/AsyncFileSystem

filesystem_spec/fsspec/spec.py

Lines 1049 to 1051 in 7aa982a

if isinstance(lpath, list) and isinstance(rpath, list):
# No need to expand paths when both source and destination
# are provided as lists

In particular this avoids calling isdir() which can be really expensive in object stores like gcsfs. Concretely it saves 3 GET requests in each commit through a simplecache chained to gcsfs. See fsspec/gcsfs#702 for additional context.

@aabmass aabmass marked this pull request as ready for review September 24, 2025 04:09
@martindurant
Copy link
Member

Perfect, thank you.

@martindurant martindurant merged commit 6943420 into fsspec:master Sep 24, 2025
10 checks passed
@aabmass aabmass deleted the put-list branch September 24, 2025 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants