Skip to content

Conversation

@wbbradley
Copy link
Contributor

Description

Implement streaming blobs by slivers via a new endpoint /v1/blobs/{blob_id}/stream.

Test plan

Manually tested. CI Pipeline.


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that
a user might notice and any actions they must take to implement updates. (Add release notes after the colon for each item)

  • Storage node:
  • Aggregator: There is a new alpha endpoint /v1alpha/blobs/{blob_id}/stream which is intended to reduce latency to first byte, and is intended to be used to stream blob data.
  • Publisher:
  • CLI:

@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from 9a67517 to a18a119 Compare December 19, 2025 20:52
@github-actions
Copy link
Contributor

Warning: This PR modifies one of the example config files. Please consider the
following:

  • Make sure the changes are backwards compatible with the current configuration.
  • Make sure any added parameters follow the conventions of the existing parameters; in
    particular, durations should take seconds or milliseconds using the naming convention
    _secs or _millis, respectively.
  • If there are added optional parameter sections, it should be possible to specify them
    partially. A useful pattern there is to implement Default for the struct and derive
    #[serde(default)] on it, see BlobRecoveryConfig as an example.
  • You may need to update the documentation to reflect the changes.

@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from 3d6668f to 4de2dc4 Compare December 20, 2025 17:06
@wbbradley wbbradley force-pushed the streaming-slivers branch 5 times, most recently from fc5857a to 7ad8b46 Compare December 28, 2025 22:40
@wbbradley wbbradley force-pushed the streaming-slivers branch 4 times, most recently from 7b33c91 to 2598d63 Compare January 2, 2026 23:34
@jpcsmith jpcsmith removed their request for review January 5, 2026 11:59
@jpcsmith
Copy link
Contributor

jpcsmith commented Jan 5, 2026

Removed myself since Markus is having a look. Let me know if there something that I should look at regardless.

@mlegner mlegner removed their request for review January 6, 2026 16:14
@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from 9529114 to 7b9ff3d Compare January 7, 2026 04:38
Copy link
Contributor

@halfprice halfprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wbbradley for implementing this feature! This looks great and I only have some minor comments.

Another thing we discussed was to use streaming blob as default and replace the existing blocking get_blob. Can you create a issue tracking this?

@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from fc99dc3 to 83136c6 Compare January 13, 2026 19:07
@wbbradley wbbradley requested a review from halfprice January 14, 2026 04:31
@wbbradley
Copy link
Contributor Author

@halfprice I believe I've addressed your comments. I've also implemented a new e2e test called test_streaming_blob.

@wbbradley
Copy link
Contributor Author

I've added WAL-1107 to track using this functionality for get_blob.

Copy link
Contributor

@halfprice halfprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks @wbbradley for adding this.

@wbbradley wbbradley merged commit e3a5ab6 into main Jan 15, 2026
24 checks passed
@wbbradley wbbradley deleted the streaming-slivers branch January 15, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants