state in Hydroflow

[jhellerstein]

Hydroflow state today

external

• streams (as referenced in source_stream and dest_sink)
• context object

internal

• stream-typed variables, e.g.

strm = source_stream(...) -> map(|i| ...);

• singleton-typed variables, e.g.

v = source_stream(...) -> fold(|| ...);
source_stream(...) -> map(|i| i + #v);

Suggestion: Should context be referenced (or at least referenceable) as #context like any singleton?
Question: What system state should be in context, and what system state should come in a stream? For example, should the dynamic global state in the context (e.g. current_tick, current_tick_start and current_stratum, subgraph_last_tick...) instead or additionally be available as a stream? (Not that current_subgraph is not transducer-global).
Possible Rubrics:

• First axis: what is context? local runtime state of the transducer.
• Second axis: what is a singleton for? It's a singleton within a tick, and a stream is a collection (which has unbounded size) within a tick.
• If we believe both of the above, should we have a stream-typed context_stream for data that is of collection type per tick? That's an open design question. Not necessarily -- maybe we "flatten" such data out of the singleton context??

Singleton Example: A Hybrid Logical Clock, which is a function of the local tick and an input packet.
Takes in a stream of packets, computes the HLC as the "max" HLC of all packets and local clock, and then uses
it to create HLC-stamped output packets.

cur_hlc = source_stream(packets) -> map(|p| hlc(p, context.current_tick()) -> lattice_fold::<HLC>();
source_stream(packets) -> map(|p| hlc_stamp(p, #cur_hlc)) -> dest_sink(out_stream);

Coming

Async invocation

Ryan A's futures?

• feat(hydroflow): Added poll_futures and poll_futures_async operators. #1143
• Async code invoked in map
• poll_futures operator handles the result
• can turn futures into source_stream

Joe/Rohit handwritten req/resp code:

• Outstanding requests go into a join hashtable before going across "the network"
• responses and timeouts stream in and join with the outstanding requests to produce outputs

Patterns

• external events show up in streams
• "subscribing" to an event is the static side of a join:

events -> [0]my_subscriptions
subscribe -> [1]my_subscriptions;
my_subscriptions::<'tick, 'static>() ->  ... // handle subscription

• "handling events" is just dataflow

events -> [0]my_subscriptions -> ... // handle subscription