Skip to content

Conversation

@kylebarron
Copy link
Member

@kylebarron kylebarron commented Oct 25, 2024

Change list

  • Add support for sync iterators and async iterators in obstore.put.
    • Improved PutInput concept, which is an enum over pull-based sources, sync push-based sources, and async push-based sources.
    • Previous accepted input into PutInput is now contained in PullSource, including files on disk, file-like objects, and buffers.
    • Now there's both SyncPushSource, which covers sync iterators emitting buffer protocol objects, and AsyncPushSource, which covers async iterators emitting buffer protocol objects.
  • Add tests.
  • More detailed exceptions by using Debug instead of Display on the rust side.
  • Add note in BytesStream docstring about making the timeout parameter longer.
  • Rename Buffer to Bytes in type hints (from Use pyo3-bytes from obstore #128)
  • Add BytesStream to website docs
  • Update documentation

Closes #47

@kylebarron kylebarron changed the title wip: async stream upload Async stream upload Nov 22, 2024
@kylebarron kylebarron mentioned this pull request Jan 14, 2025
@kylebarron
Copy link
Member Author

It turns out that the error I was previously hitting was just the request timeout, since I was downloading a large file.

@kylebarron kylebarron merged commit 78860c6 into main Jan 14, 2025
4 checks passed
@kylebarron kylebarron deleted the kyle/async-stream-upload branch January 14, 2025 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming uploads

2 participants