Removing the 'ak.singletons' and 'ak.firsts' functions: any complaints?

[aminnj]

Hi,

I'm using Python 3.7.3 and awkward 1.0.2 and I see that ak.singletons() only adds a dimension when there is a missing element. E.g.,

ak.singletons([2,0,1])
# <Array [2, 0, 1] type='3 * int64'>
ak.singletons([2,None,1])
# <Array [[2], [], [1]] type='3 * var * int64'>

which means I have two cases to deal with when using one array as an index for another.

The first case (which doesn't give me my desired output) where the first array has inner arrays of size at least 1 (and thus each value in the second index array is defined):

vals = ak.Array([[43, 15, 10.5], [11.5], [50, 5]])
idx = ak.Array([2, 0, 1])
vals[ak.singletons(idx)]
# <Array [[50, 5], [43, 15, 10.5], [11.5]] type='3 * var * float64
# but I was expecting to get [[10.5],[11.5],[5]]

And another case (which does give me my desired output) where the first array has at least one empty inner array (and thus at least one missing element in the indexing array):

vals = ak.Array([[43, 15, 10.5], [], [50, 5]])
idx = ak.Array([2, None, 1])
vals[ak.singletons(idx)]
# <Array [[10.5], [], [5]] type='3 * var * float64'>

Is this intended? If so, is there a more awkward-like way of doing vals[idx]?

Thanks!

[jpivarski]

The semantics of this—what it's supposed to do—could be better thought-out. It was intended for cases in which missing values are present because it was for handling the output of ak.argmin and ak.argmax. (Here are some early thoughts about the problem: #203.) The problem is basically this:

>>> # We want to find the max of one_quantity and look at the corresponding quantity in "another."
>>> one_quantity = ak.Array([[1, 2, 3], [], [5, 4]])
>>> another = ak.Array([[1.1, 2.2, 3.3], [], [5.5, 4.4]])

>>> # As of Awkward 1, argmax gives one dimensional output with Nones, not singletons/empties.
>>> # (This is required for conformance with NumPy.)
>>> ak.argmax(one_quantity, axis=1)
<Array [2, None, 0] type='3 * ?int64'>

>>> # Applying that to "another" is not what we want, since it applies to the first dimension.
>>> # It rearranges the lists, rather than picking out the corresponding element from each list.
>>> another[ak.argmax(one_quantity, axis=1)]
<Array [[5.5, 4.4], None, [1.1, 2.2, 3.3]] type='3 * option[var * float64]'>

>>> # ak.singletons gets us to that other format.
>>> ak.singletons(ak.argmax(one_quantity, axis=1))
<Array [[2], [], [0]] type='3 * var * int64'>

>>> # And we get what we want: the corresponding value from each list.
>>> another[ak.singletons(ak.argmax(one_quantity, axis=1))]
<Array [[3.3], [], [5.5]] type='3 * var * float64'>

>>> # Which we then have to pair with ak.firsts to turn it back into flat-with-missing-values.
>>> ak.firsts(another[ak.singletons(ak.argmax(one_quantity, axis=1))])
<Array [3.3, None, 5.5] type='3 * ?float64'>

However, that's non-obvious and verbose. I don't remember where I first saw it, but some users of Awkward found a much simpler solution:

>>> # The 'keepdims' argument was included for NumPy compatibility...
>>> ak.argmax(one_quantity, axis=1, keepdims=True)
<Array [[2], [None], [0]] type='3 * var * ?int64'>

>>> # But it gives us nearly what ak.singletons does, which simplifies the max-by-another problem.
>>> another[ak.argmax(one_quantity, axis=1, keepdims=True)]
<Array [[3.3], [None], [5.5]] type='3 * var * ?float64'>

>>> # And then we can do the "ak.firsts" thing with a simple slice.
>>> another[ak.argmax(one_quantity, axis=1, keepdims=True)][:, 0]
<Array [3.3, None, 5.5] type='3 * ?float64'>

So in truth, ak.singletons and ak.firsts aren't really needed to solve the problem they were invented for, and their semantics weren't carefully thought through for other problems.

For instance, I wonder if you can get what you want by turning a regular axis created with np.newaxis into an irregular one with ak.from_regular:

>>> vals = ak.Array([[43, 15, 10.5], [11.5], [50, 5]])
>>> idx = ak.Array([2, 0, 1])

>>> # np.newaxis makes a new axis with length 1.
>>> idx[:, np.newaxis]
<Array [[2], [0], [1]] type='3 * 1 * int64'>

>>> # But we're about to do a jagged slice, so turn the length-1 dimension into a variable-length one.
>>> ak.from_regular(idx[:, np.newaxis])
<Array [[2], [0], [1]] type='3 * var * int64'>

>>> # Use this to slice "vals". It picks out the nth item from each list.
>>> vals[ak.from_regular(idx[:, np.newaxis])]
<Array [[10.5], [11.5], [5]] type='3 * var * float64'>

>>> # It works exactly the same way if you have any missing data.
>>> idx = ak.Array([2, None, 1])
>>> vals[ak.from_regular(idx[:, np.newaxis])]
<Array [[10.5], [None], [5]] type='3 * var * ?float64'>

>>> # And that shape is exactly what you want if you're going to be removing this dimension.
>>> vals[ak.from_regular(idx[:, np.newaxis])][:, 0]
<Array [10.5, None, 5] type='3 * ?float64'>

So maybe ak.singletons and ak.firsts should be deprecated, and these slicing tricks should be advertised in tutorials on the https://awkward-array.org site. (I really need to write more of that documentation.) If you agree, I'll convert this Issue into a Discussion and then it would remain visible for others to comment on the proposed deprecation and benefit from the ad-hoc documentation above.

[nsmith-]

I think here is where I pointed out that keepdims needed to return an irregular axis to be used as a slicer: #434 and its nice to see now there is a utility to convert between regular and irregular with ak.from_regular

[drahnreb]

Slicing can be hard. Ragged slicing can be harder. This might be a bit off-topic regarding ak.singletons and ak.firsts deprecation but I think a slicing abstraction layer e.g. an awkward-style np.take that extends to unique challenges in the ragged space, like #370, could combine and simplify a few common or interesting slicing applications.

Apart from the discussed scenarios, I see useful extensions like e.g. multidimensional index slicing

idxs = [0, 1, 0, 2, 1]
ak.singletons(ak.pad_none(a, ak.num(a))[np.arange(len(a)), idxs])

or "ragged-to-ragged slicing" instead of semi-internal layout fiddling (where I mutate starts and stops indices).

[jpivarski]

We're heading a little in that direction—the ak.mask syntax deliberately makes it look like a slice because it's almost a slice:

>>> array = ak.Array(range(10))
>>> array
<Array [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] type='10 * int64'>
>>> array[array % 2 == 0]
<Array [0, 2, 4, 6, 8] type='5 * int64'>
>>> array.mask[array % 2 == 0]
<Array [0, None, 2, None, ... 6, None, 8, None] type='10 * ?int64'>

and I was thinking about a pick that also acts as an easier slice: #203 (comment)

I'd like to avoid the word take, though, because it already has a meaning. @nsmith- mentioned below the usefulness of some shorthands. If they can be expressed entirely in terms of existing operations, maybe they could go into a new operations submodule:

src/awkward
└── operations
    ├── convert.py
    ├── derived.py     <---- ?
    ├── describe.py
    ├── reducers.py
    └── structure.py

[aminnj]

Thanks for the detailed answer! I need to remember to use keepdims=True when the index array comes from other awkward operations. The np.newaxis trick works for when I have a handmade index array (that might use -1 instead of None for when the collection is empty).

Maybe ak.singletons/ak.firsts is still useful to keep around, given that they can convert the output of argmax/argmin after they have been computed (at least, with the above alternatives advertised somewhere)? I've also found ak.firsts to be a nicer alternative to ak.pad_none(a, 1)[:,0] since it turns inner [] into None)

[nsmith-]

I want to +1 the use of ak.firsts as shorthand for ak.pad_none(a, 1)[:, 0] which is somewhat common. In fact, it would be nice to be able to optionally ignore bounds in slices like this, returning None when out of bounds.

[jpivarski]

But you did raise an important point, that ak.singletons and ak.firsts are not well thought-through. They don't do what you expected for non-option-type arrays, so they ought to be fixed (given a simple, easily describable rule for what they should do in all circumstances) or removed. Removing is definitely less work!

I'll convert this into a Discussion to let others weigh in on it. Now would be a good time to schedule them for removal in 1.2.0 or 1.3.0 (see Roadmap).

[nsmith-]

Nick shows clearly an inconsistency for ak.singletons but is there one also for ak.firsts?

[agoose77]

I currently use ak.firsts to handle the case where I have a bool array on which I wish to evaluate something like ak.where, except yielding None where no items are True, and yielding only one value for the inner axis.

local_nonzero = ak.local_index(is_pid_i)[is_pid_i]
j = ak.firsts(local_nonzero)

I wonder if there's a better way to do this that doesn't require local_index, but I suspect not because somewhere the local indices need to be computed. Clearly, without firsts, I'd need to pad_none and then slice.

[jpivarski]

Conclusion: ak.singletons and ak.firsts will be kept, but their implementation in Awkward version 2.0 should probably be a rewrite without guarantee of backward compatibility, so that they can be better suited to the cases they were applied to, as opposed to the ak.argmin/argmax case.

See #1189 for an example of something that wasn't well thought-through.

ak.firsts has an axis but ak.singletons doesn't. Should it? (Can it?)