Missing distributed array creation methods

[stevenrbrandt]

• array_d(array) : split up an existing array across localities (we have tile(), see below)

• fromfunction(func, shape) : like numpy's fromfunction (see Adding indices and fromfunction primitives #1289)

• file_read_hdf5_d(filename) : like file_read_csv_d except it reads from hdf5 files

Related methods that would be helpful...

• file_write_hdf5_d(filename) : like file_read_hdf5_d except it writes files

• file_write_csv_d(filename) : like file_write_hdf5_d except it writes csv files

[ct-clmsn]

@stevenrbrandt,

• Would array_d accept an array shape and raw data as an input argument?

• If the input is a shape, would the shape be generated across multiple machines symmetrically? by that, I mean, an input of 4x4 would create 4x4 tiles across N-machines.

• Could a shape argument be capable of defining the size, how many machines to tile across, and the memory layout of the tile (row/column)?

example:

array_d([[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]]) -> each machine creates a 4x4 matrix tile of 0's.

array_d(shape=((4,4), row)) -> all machines create a 4x4 row order tile

array_d(shape=( (4,4), [ (0, row) , (1, col) ] ) ) -> each machine makes a 4x4 tile, machine 0 creates row order tile, machine 1 creates a column order tile; if a machine isn't listed, it creates a 4x4 using a default layout.

[hkaiser]

array_d( array ) : split up an existing array across localities

We have tile(array, ...) that is doing exactly that (see: https://github.com/STEllAR-GROUP/phylanx/blob/master/src/plugins/matrixops/tile_operation.cpp#L32-L45)

[hkaiser]

@stevenrbrandt How should the distributed array be initialized? We have available the equivalents for numpy.zeros, numpy.ones, numpy.empty, etc. (all of those are mapped onto our constant primitive). We should however make sure that our distributed constant_d primitive supports all variations.

[stevenrbrandt]

@hkaiser array_d(arg) should not take a shape arg. It should work kind of like numpy.array(arg) does. It should convert a non-distributed array into a distributed one. The value arg should keep both its data and its shape, but that shape should get split up.

The 'tile()' primitive creates a bigger array by replicating an input array. I assume you meant retile_d()? As discussed in the meeting, retile_d() does not create a distributed array, it retiles an existing distributed array.

array_d() could take an optional tiling argument and default to sym.

[stevenrbrandt]

@ct-clmsn see above about what I think array_d() ought to do.

[hkaiser]

@hkaiser array_d(arg) should not take a shape arg. It should work kind of like numpy.array(arg) does. It should convert a non-distributed array into a distributed one. The value arg should keep both its data and its shape, but that shape should get split up.

The 'tile()' primitive creates a bigger array by replicating an input array. I assume you meant retile_d()? As discussed in the meeting, retile_d() does not create a distributed array, it retiles an existing distributed array.

array_d() could take an optional tiling argument and default to sym.

In all of this you are assuming a single locality loading a file (or otherwise generating the array) which should then be distributed across other localities. The execution model we target at this point however is pure SPMD, i.e. every locality executes the same code. How should the array_d primitive proposed operate when executed on the localities that do not have the data? In the end the (re-)tile primitive would do the right thing if one of the localities has all of the data and all the other localities just have an empty array.

[stevenrbrandt]

@hkaiser -- if we assume pure SPMD, then any non-distributed array must have an identical copy on each locality (unless it's a random array). I guess that means array_d() would throw away all but it's part of the data and provide a unifying id for the distinct pieces.

[hkaiser]

@hkaiser -- if we assume pure SPMD, then any non-distributed array must have an identical copy on each locality (unless it's a random array). I guess that means array_d() would throw away all but it's part of the data and provide a unifying id for the distinct pieces.

Ok. You can do that using slice_d already, I think. If that doesn't work (yet), using slice/annotate should do the trick as well.

[hkaiser]

array_d([[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]]) -> each machine creates a 4x4 matrix tile of 0's.

array_d(shape=((4,4), row)) -> all machines create a 4x4 row order tile

array_d(shape=( (4,4), [ (0, row) , (1, col) ] ) ) -> each machine makes a 4x4 tile, machine 0 creates row order tile, machine 1 creates a column order tile; if a machine isn't listed, it creates a 4x4 using a default layout.

@ct-clmsn I like this approach as it provides a concise way to specify tiles. We currently have some means for that, but it's difficult to use.

[ct-clmsn]

@hkaiser - in reference to tile - cool! i recalled ya'll had some mechanism defined earlier, couldn't remember the name. As for the creation routines - thanks, we've collectively talked about doing these sorts of things, not sure if we've ever placed it into text before; hope it captures the discussion lines.

@stevenrbrandt - I like the concept of registering local data into a distributed AGAS container (matrix/array/vector). did you have any comment on the examples in the response I posted earlier? Do those fit into your model or are they separate conceptually and build on what you are thinking?

[stevenrbrandt]

@hkaiser I don't see how slice_column_d() or slice_row_d() provide this functionality. As for annotate_d(), sure, that works. However, it means the user has to figure out exactly which piece of which array they want on which processor rather than just getting a standard tiling. I think annotate_d() is too low level for most users.

[stevenrbrandt]

@ct-clmsn I'm not sure I understand your idea. Normally, when we think of a distributed array, I think of a large array with some definite shape that gets chopped into pieces and distributed. You seem to want to have a distributed array that exists as an identical copy on each processor, but with different tilings. Is that correct? I'm a little unclear about how such an array is kept in sync during a calculation.