ENH: Allow where/mask/Indexers to accept callable

[sinhrks]

• closes Allow .where to accept callable as condition #12533
• closes API: chained reshaping ops #11485
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

Current impl is very simple like assign. Another idea is applying some restriction to return value from callable (for example, only allow aligned DataFrame or scalar).

[kawochen]

One potential problem is when you subclass DataFrame and make it callable.

[sinhrks]

Do you know such a package? The logic is not specific here and I don't think we have to care ATM.

[kawochen]

I don't know anyone that uses it that way. But since the order of consideration is relevant, I think it should be documented that the args are first tested for callability.

[jreback]

let's simultaneously do all of the chained reshaping ops #11485 in 0.18.1, making them consistent.

[sinhrks]

Sure, once done and changed the title.

[jreback]

need to have some tests if people try to pass something like:

df.loc[lambda x: ...., list_of_columns] which is a bit odd, but I suppose ok, otherwise we need to interpret this

[sinhrks]

OK, added some tests for mixture of lamda and scalar/list.

[jreback]

The following methods / indexers now acccept a callable

[codecov-io]

Current coverage is 83.91%

Merging #12539 into master will increase coverage by +<.01%

@@             master     #12539   diff @@
==========================================
  Files           136        136          
  Lines         49908      49933    +25   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41876      41901    +25   
  Misses         8032       8032          
  Partials          0          0          

Powered by Codecov. Last updated by 3921b32

[TomAugspurger]

What do you mean by this?

[jreback]

ok, how about for .query() we than raise if the input is NOT a string with a nice mesage (to use .loc)? or could show a warning and just do it anyhow.

[jorisvandenbossche]

You already get an error when passing a callable to query, but I agree a nicer message can be raised.

[shoyer]

I would definitely raise if the input to .query() is not a string. Adding warnings and then doing something anyways is a bad idea.

[jreback]

ok @sinhrks if you can make those mods would be great. (you might need to adjust docs/example as I think you use .query, but can just change to .loc)

[sinhrks]

Thanks for all the feedbacks. Updated. Lmk if I missed something.

[jreback]

I would make this a separate file: indexing/test_callables.py

[jreback]

just minor change otherwise lgtm.

@jorisvandenbossche @shoyer

[jorisvandenbossche]

I would add this example somewhere in the where docs as well (http://pandas-docs.github.io/pandas-docs-travis/indexing.html#the-where-method-and-masking)

[jorisvandenbossche]

@sinhrks Can you also update the docstrings for loc/iloc/ix (eg for loc https://github.com/pydata/pandas/blob/65ed3afd092908130905e7d86ff1b6b9c2002d00/pandas/core/indexing.py#L1314), these are used in the API docs.

For the rest, +1!

[jreback]

add a versionadded tag here (and other doc-strings)

[sinhrks]

assign impl is unchanged.

[jreback]

@sinhrks some very minor doc-changes. ping when pushed.

[jreback]

thanks @sinhrks !

awesome enhancement!