Add operator.getitemtuple & operator.itemtuplegetter

[dg-pb]

Proposal:

from itertools import repeat
from operator import getitemtuple, itemtuplegetter

obj = ['a', 'b']
indices = [0, 1, 2, 3]
defaults = repeat(None)

itg = itemtuplegetter(indices, defaults=defaults)
print(itg(obj))
# ('a', 'b', None, None)

print(getitemtuple(obj, indices, defaults))
# ('a', 'b', None, None)

Other form was considered: itemitergetter()() -> Iterator.
But its performance has turned out to be non-competitive (see discourse discusssion).
Thus, it is best to follow tuple return same as existing itemtgetter.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

https://discuss.python.org/t/itemgetter-split-into-2-objects/69020/33

[picnixz]

Ignore this if it has this been already decided on DPO (there are too many messages for a fast overview).

---

Bikeshedding here but can we align the default with whatever next does? namely a positional-only argument passed to itg(object, default) instead of passing the default at construction time? Or allow both maybe (and reject if defaults have already been specified). My rationale is that you might want to only construct itemtuplegetter and, when you want to use defaults, you would then do itg(x, d) insteada of itg(x).

But otherwise, I'm fine with specifying the defaults at construction time, but without a keyword argument (for instance, "defaults" could mean the default values to return and not the filling values). So, we could also use fill=... or fillevalues=... (we have fillvalue for zip_longest).

[dg-pb]

Bikeshedding here but can we align the default with whatever next does?

The case when one wants to do itg(obj, defaults) is covered by getitemtuple(obj, items, defaults).
itg(obj, defaults) would have no advantages over the former as it would need to do large part of the input parsing that is done in __init__ at __call__-time.

For the convenience of binding, one can use partial(getitemtuple, Placeholder, items). Would be marginally slower than specialisation in itemtuplegetter, but I think it is a happy middle.

But otherwise, I'm fine with specifying the defaults at construction time, but without a keyword argument (for instance, "defaults" could mean the default values to return and not the filling values). So, we could also use fill=... or fillevalues=... (we have fillvalue for zip_longest).

Could you clarify? defaults are "default values to return" if item not found in object.
I.e.

items = ['c', 'b']
defaults = [0]


getitemtuple(obj, items, defaults)

# Equiv

result = list()
for i in range(len(items)):
    idx = items[i]
    if i < len(defaults):
        try:
            result.append(obj[idx])
        except (KeyError, IndexError):
            result.append(defaults[i])
    else:
        result.append(obj[idx])

[picnixz]

Ok, my bad for this one. Because of repeat(None), I was under the impression that it was used in a "streaming" fashion instead but that's not the case. So you can ignore my question.

[dg-pb]

In this initial version, there is a restriction on the fact that defaults can only be provided for "leading" items.
And there is no way to have a default for 2nd, but not the 1st item.

To overcome this, in many cases one can adjust the order of items.

While this is a good solution that can address many cases, there is an option for NoDefault sentinel extension in the future, which would allow having both "desired order" and "sparse-hand-picked defaults".

However, I think time will tell whether there is a need for such and this initial concept already covers many cases.

[picnixz]

If you want defaults to skip something, use a dict with keys being the indices (I think this should work):

if isinstance(defaults, Mapping):
	for i in range(len(items)):
	    idx = items[i]
	    try:
	        result.append(obj[idx])
	    except LookupError:
	        result.append(defaults[i])

Sparse defaults should assume that you don't expect other indices to fail. Non-sparse defaults would use your above example.

[dg-pb]

Thought about this and unfortunately it imposes undesirable restriction - dict only supports hashable keys.

Also, dict access is very slow compared to sequence and while it is nothing major, but this specific component seems to be largely liked due to its performance, thus this might be a dealbreaker for many.

Of course, as you suggested one can have an option to use dict or not, but this just leads to further restriction and still leaves user with needing to pick some of the features without the ability to have "all".

Thus, this leads to more complexity and specialisation.
While, NoDefault placeholder is not as elegant as dict, but it would provide a definite "have it all" solution.

Personally, I like sentinel solutions. I think part of the discomfort with these can be addressed to lack of convenient implementation toolkit (currently one needs to write whole new object in C and deal with many nuances of it).

I suspect that this might change to some degree given there was a place in CPython, where one can just add 1-line to create new singleton-sentinel.

[picnixz]

Oh right. I tend to forget that itemgetter is not restricted to hashable keys but can be done for any type of keys. Forget the dict suggestion.

[rhettinger]

Sorry, I don't support this being added to the operator module. It is well beyond the module's intended scope, but its biggest issue is that it creates confusion from too many similar ways to do it. Tell someone that there is an itemgetter, getitemtuple, and itemtuplegetter and see if they can deduce in the intended distinctions and know which to choose.

Also, the performance aspect of the newsgroup discussion is a red herring. It is reasonably possible that the JIT efforts will make plain Python code nearly as fast as what this would offer. There is no longer sufficient motivation to invent more exotic ways to avoid a simple lambda or def.

Recall that the reason Guido allowed itemgetter and attrgetter was to support writing nice looking key-functions. It was never intended to evolve beyond that.

[picnixz]

Tell someone that there is an itemgetter, getitemtuple, and itemtuplegetter and see if they can deduce in the intended distinctions and know which to choose.

(I personally couldn't at first; I don't use a lot of itemgetter or attrgetter personally, so I'm also -0.)

[dg-pb]

Gauging sentiment in discussions I didn't think there is going to be a lot of resistance.

All good points above, I hope I can convey a bit more complete rationale for this by answering them.

---

its biggest issue is that it creates confusion from too many similar ways to do it. Tell someone that there is an itemgetter, getitemtuple, and itemtuplegetter and see if they can deduce in the intended distinctions and know which to choose.

I don't think it is that confusing.
It isn't 3 objects.
It is 2 + 2:

1. getitem (and its itemgetter)
2. getitemtuple (and its itemtuplegetter)

The mental model is that itemgetter should be used for 1 item and itemtuplegetter for many items.

itemgetter should never have allowed multiple items.
I don't think unpredictability in its return value was a good decision.

itemgetter does 2 things, but doesn't do the second one well. At the same it has blocked the way for other reliable tool addition for multi-item cases such as this.

It currently is prone to bugs when constructing it with an unknown number of items without a good alternative solution: https://github.com/python/cpython/pull/124788/files#diff-8657d1a12c81ea9da8c75d3de3827fc6f1e033339453e9587f1ec2f5dfc5111dR314

---

Another motivation for this was also to cover the multi-item part of: https://discuss.python.org/t/indexable-get-method-1-2-3-get-4-none/53430/3
So https://discuss.python.org/t/allowing-missing-item-or-attributes-in-operators-itemgetter-and-attrgetter/66256/4 would cover the single-item case and this would cover the multi-item.

---

It is well beyond the module's intended scope

I think it could be fairly reasonable to include "abstract object layer" as a possible secondary scope.

There are some functionality/features desirable by community and there are occasional requests to extend various collections with new methods.

Naturally, it is extremely unlikely that something will be worth the effort to add a new method to say list.

However, adding small utilities to operator when there is a good case would seem reasonable to me. Such do not have any impact on object protocols and can often cover more than 1 object. E.g. list.get only applies to list while getitem(..., default=) would apply to most of the objects that support __getitem__.

Having said that, I don't think this should be abused and only allow for additions that cover a sufficiently large scope of needs with minimal additions.

Also, the performance aspect of the newsgroup discussion is a red herring. It is reasonably possible that the JIT efforts will make plain Python code nearly as fast as what this would offer.

Although I see that theoretically it could come close, I doubt it will ever match.
Performance is only a fraction of the rationale for this, but knowing that it will always remain the most performant solution, even if by a small percentage, is a positive.

Some benchmarks:

lst = [None] * N
idxs = list(range(N))
noidxs = [N] * N
# N = 2 (ns)
%timeit getitemtuple(lst, idxs)           # C: 190, Py: 370
%timeit getitemtuple(lst, noidxs, idxs)   # C: 550, Py: 1000
# N = 100K (ms)
%timeit getitemtuple(lst, idxs)           # C: 1.7, Py: 2.9
%timeit getitemtuple(lst, noidxs, idxs)   # C: 18,  Py: 31

So currently it is 40-50% faster.
So even if, say, JIT makes Python version only 10% slower, this would still be a 10% faster solution that can be used to simplify code in variety of cases.

E.g. This would be a good option for partial defaults.
In this case, it solves a fairly common problem, where otherwise there is no convenient solution.

sys_argv = [0, 1]

b, a = getitemtuple(sys_argv, [2, 1], ['b_dflt'])
(a, b)    # 0, 'b_dflt'

getitemtuple(sys_argv, [], ['b_dflt'])
# Error, first argument does not have a default

---

To sum up, I think this proposal is quite well balanced addition in several aspects:

1. Straightens out design flaws of itemgetter
2. Covers list.get multi-item case. This came up in the past and I predict will come up over and over again from time to time.
3. Provides convenience by abstracting "multiple-item-case" try-except-KeyError/IndexError-default for a wider object scope, which is a fairly common and quite verbose pattern. And this is useful in a variety of different cases - both mappings and sequences.
4. Although I can see that JIT might come close, I doubt that it would ever fully match the performance of this addition.

[dg-pb]

Recall that the reason Guido allowed itemgetter and attrgetter was to support writing nice looking key-functions. It was never intended to evolve beyond that.

I appreciate the absence of initial intent, but I think this has already been affected by https://www.hyrumslaw.com to the extent that it might make sense to improve the situation by solving some unintended consequences.