refactor(hydro_lang)!: migrate Hydro IR to have Flo semantics at all levels #2136
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Deploying hydro with
|
| Latest commit: |
62094c0
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f3c7c47f.hydroflow.pages.dev |
| Branch Preview URL: | https://pr2136.hydroflow.pages.dev |
Member
Author
|
Draft until I add unit tests for regressions I caught during development. |
shadaj
force-pushed
the
pr2136
branch
4 times, most recently
from
862275b to
0b8922d
Compare
…levels Previously, the Hydro -> Hydro IR layer did a bunch of work to translate between Flo semantics and DFIR semantics, so that Hydro IR -> DFIR was generally 1:1. In preparation for the simulator and DFIR's migration to Flo semantics, we now use Flo semantics in the Hydro IR (top level operators do not reset their state on batch boundaries), and shift the Flo -> DFIR semantics translation to the IR -> DFIR layer. This also comes with a breaking API change to clean up naming and avoid function overloading. In particular, `batch` / `snapshot` are renamed to `batch_atomic` and `snapshot_atomic` for the case where the batched result is supposed to be in-sync with the atomic execution context. There were a couple of bugs during the transition to the new IR semantics that were not caught by existing unit tests, so I added additional tests that cover each of them. Also makes some minor improvements / bug fixes the way: - Restricts `KeyedSingleton::snapshot` to unbounded-value only, since bounded value keyed singletons do not replay their elements - Groups together definitions for `resolve_futures` and `resolve_futures_ordered` - Removes various unnecessary `NoAtomic` trait bounds
davidchuyaya
approved these changes
Sep 23, 2025
MingweiSamuel
approved these changes
Sep 23, 2025
| metadata: HydroIrMetadata, | ||
| }, | ||
|
|
||
| YieldConcat { |
Member
Author
There was a problem hiding this comment.
In Flo-terminology, yields the bounded collection inside the tick to outside the tick by using the "concat" operator (++) for that collection type. The concat is the "natural" combination method, appending elements for stream, replacing the value for singleton, etc.
Comment on lines
+2030
to
+2032
| // if both inputs are root, the output is expected to have streamy semantics, so we need | ||
| // a multiset_delta() to negate the replay behavior | ||
| parse_quote! { |
Member
There was a problem hiding this comment.
Do we have an issue to fix this at the DFIR level?
| #source_ident = source_iter(#expr); | ||
| } | ||
| } else { | ||
| // TODO(shadaj): a more natural semantics would be to to re-evaluate the expression on each tick |
luckyworkama
approved these changes
Sep 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, the Hydro -> Hydro IR layer did a bunch of work to translate between Flo semantics and DFIR semantics, so that Hydro IR -> DFIR was generally 1:1. In preparation for the simulator and DFIR's migration to Flo semantics, we now use Flo semantics in the Hydro IR (top level operators do not reset their state on batch boundaries), and shift the Flo -> DFIR semantics translation to the IR -> DFIR layer.
This also comes with a breaking API change to clean up naming and avoid function overloading. In particular,
batch/snapshotare renamed tobatch_atomicandsnapshot_atomicfor the case where the batched result is supposed to be in-sync with the atomic execution context.There were a couple of bugs during the transition to the new IR semantics that were not caught by existing unit tests, so I added additional tests that cover each of them.
Also makes some minor improvements / bug fixes the way:
KeyedSingleton::snapshotto unbounded-value only, since bounded value keyed singletons do not replay their elementsresolve_futuresandresolve_futures_orderedNoAtomictrait boundsStack created with Sapling. Best reviewed with ReviewStack.