Skip to content

Add streaming throughput monitor #494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 16 commits into
base: master
Choose a base branch
from
Draft

Add streaming throughput monitor #494

wants to merge 16 commits into from

Conversation

jpsamaroo
Copy link
Member

@jpsamaroo jpsamaroo commented Mar 29, 2024 •
edited
Loading

In collaboration with @davidizzle.

Implements a throughput monitor for streaming tasks, which monitors a pipeline and prints throughput estimates periodically.

Todo:

  • Automatically allocate ctr
  • Make some things configurable (printing delay, whether to clear line, bytes/second or samples/second)

jpsamaroo and others added 16 commits December 18, 2023 19:11
@jpsamaroo
Add metadata to EagerThunk
af692b5
@jpsamaroo
Sch: Allow occupancy key to be Any
7204709
@jpsamaroo
Add streaming API
b400189
@jpsamaroo
fixup! Add streaming API
78146e6
@jpsamaroo
fixup! fixup! Add streaming API
d43a262
@jpsamaroo
TEMP: Add stream migration support
8eb1a6a
@jpsamaroo
fixup! Add metadata to EagerThunk
b22567c
@jpsamaroo
fixup! fixup! fixup! Add streaming API
bc79b84
@jpsamaroo
fixup! fixup! fixup! fixup! Add streaming API
1eea436
@jpsamaroo
fixup! fixup! fixup! fixup! fixup! Add streaming API
6b9a1af
@JamesWrigley
Reference Dagger.EAGER_THUNK_STREAMS explicitly
329e8be
@JamesWrigley
Use Base.promote_op() instead of Base._return_type()
73c62dc 
return_type() is kinda broken in v1.10, see:
JuliaLang/julia#52385

In any case Base.promote_op() is the official public API for this operation so
we should use it anyway.
@jpsamaroo
Merge pull request #484 from JamesWrigley/undefined-variable
b4f41c8 
Reference Dagger.EAGER_THUNK_STREAMS explicitly
@JamesWrigley
Special-case StreamingFunction in EagerThunkMetadata() constructor
09aedee 
This always us to handle all the other kinds of task specs.
@jpsamaroo
Merge pull request #486 from JamesWrigley/test-fixes
29d790f 
Special-case StreamingFunction in EagerThunkMetadata() constructor
@davidizzle @jpsamaroo
Add streaming throughput monitor
366cb37 
Co-authored-by: Julian Samaroo <[email protected]>
@jpsamaroo jpsamaroo mentioned this pull request Mar 31, 2024
Merged
18 tasks
@jpsamaroo jpsamaroo force-pushed the jps/stream2 branch 2 times, most recently from 0f747ed to fa01410 Compare April 5, 2024 21:52
@JamesWrigley JamesWrigley force-pushed the jps/stream2 branch 2 times, most recently from ed89a7f to 3274093 Compare August 3, 2024 16:57
@jpsamaroo jpsamaroo force-pushed the jps/stream2 branch 4 times, most recently from 16827af to a765cbe Compare December 9, 2024 16:43
Base automatically changed from jps/stream2 to master December 9, 2024 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement streaming
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jpsamaroo @JamesWrigley @davidizzle