Skip to content

decomposing foldxt #19

@ExpandingMan

Description

@ExpandingMan

One may write pseudocode for foldxt and foldxd as something like:

function foldxt(step, xf, xs)
    parts = split_into_parts(xs)
    output = init_output()
    @threads for part  parts
        insert!(output, foldxl(step, xf, part))
    end
    combine(xf, output)
end

This is fairly intuitive, however currently Transducers.jl doesn't really expose any of these intermediate steps, and the internals of it are such that it might not be all that obvious what's going on until close inspection (at least it wasn't to me).

I propose that we create a public API for each of these pieces such that a user could easily write its own foldxt. As I see it, some benefits of this would be:

  • The semantics of foldxt might be somewhat more obvious. This is important because the combine step can cause it to behave quite differently from foldxl.
  • Users could create custom combine. Currently users have to rely on Transducers to provide combine functionality.
  • Users could more easily create custom splitting strategies.

On the other hand, exposing these details could make it significantly easier to write unsafe code, though note that a custom combine needn't rely on parallelism at all.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions