Skip to content

Conversation

@frankmcsherry
Copy link
Member

@frankmcsherry frankmcsherry commented Nov 29, 2024

The SizableContainer trait spoke about capacities for containers, meant to allow them to express their capacity, a preference for a capacity, and how to acquire more capacity. It seems like this logic was used to implement some right-sizing of containers, but always in a way that the container could have chosen for itself anyhow.

With that in mind, this PR tries out a new trait that instead looks like

/// A container that can be sized and reveals its capacity.
pub trait SizableContainer: Container {
    /// Indicates that the container is "full" and should be shipped.
    fn at_capacity(&self) -> bool;
    /// Restores `self` to its desired capacity, if it has one.
    ///
    /// The `stash` argument is available, and may have the intended capacity.
    /// However, it may be non-empty, and may be of the wrong capacity. The
    /// method should guard against these cases.
    fn ensure_capacity(&mut self, stash: &mut Option<Self>);
}

Here the container can indicate if it is "at capacity" or in Vec terms if its length equals its capacity, and also it can restore its capacity. Each implementation had seemingly straight-forward implementations, and the place the logic was used seem to work great with these functions.


The commits continue with one that removes PushPartitioned and allows Exchange to be parameterized by CB: ContainerBuilder, rather than by containers.


One VERY IMPORTANT thing to sort out: if folks are allowed to create Exchange edges with their own container builders, they must preserve the counts correctly, otherwise progress tracking breaks. This might be a moment to revisit that, and see if there is a reasonable decoupling of .len() from "number of items in the container". At least, there is a very specific thing that needs to be maintained (count of records accepted), and perhaps it shouldn't be up to the container builder to maintain this?

@frankmcsherry frankmcsherry changed the title Reorganize SizableContainer Reorganize Container traits Nov 30, 2024
@frankmcsherry frankmcsherry merged commit f201abd into TimelyDataflow:master Nov 30, 2024
7 checks passed
@github-actions github-actions bot mentioned this pull request Nov 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant