ENH: implement shift(method='head|tail')

[jreback]

http://stackoverflow.com/questions/22670904/python-pandas-turn-absolute-periods-into-relative-periods/22671904?noredirect=1#comment34537319_22671904

this works, but I think could be more efficient on a whole frame basis

In [37]: def f(x):
   ....:     y = x.dropna()
   ....:     return Series(y.values,x.index[len(x)-len(y):])
   ....: 

In [40]: roller = pd.rolling_sum(df,3).reset_index(drop=True)

In [41]: roller
Out[41]: 
    1   2   3
0 NaN NaN NaN
1 NaN NaN NaN
2  61  56  56
3   9   9   9
4  12  12  12
5 NaN  15  15
6 NaN NaN  11
7 NaN NaN NaN

[8 rows x 3 columns]

In [43]: roller.apply(f).reindex_like(roller)
Out[43]: 
    1   2   3
0 NaN NaN NaN
1 NaN NaN NaN
2 NaN NaN NaN
3 NaN NaN  56
4 NaN  56   9
5  61   9  12
6   9  12  15
7  12  15  11

[8 rows x 3 columns]

[dsm054]

I think we'd want a call pattern which would support pulling the non-nan values up to the head -- which is actually what I usually need -- as well as pushing them down to the tail, so we'd need both an axis and a direction (a way to position "down").

[dsm054]

I'm also not sure that dropna is the right method to use, because this will never actually return a smaller frame. It's more like a shift + fillna.

[jreback]

yep.....its more like movena; not sure that really want another method name though.....

[dsm054]

If we don't want to add a new method name, then maybe we should use fillna. It already accepts a method argument, and we could simply add broll/froll or something (bfall/ffall)? Because fillna can already return series/frames which still have nans, it's really like a "do something with some nans, maybe" in practice anyway.

[jreback]

good idea! and I like those method names. you could use the limit argument to put in spacers (e.g. offsets from the end) as well.

[jreback]

maybe df.shift(method='broll') or df.shift(3,method='froll')

[dsm054]

Either would work for me: it's sort of a shift (because it repositions) and sort of a fillna (because it changes non-nan values based on the nan values -- or maybe I should say that the other way around.)

I'm not sure what df.shift(3, method='froll') would do, though. Group all the non-nan values and assign them to .iloc[3:]? To .iloc[:-3]?

[jreback]

how about head/tail instead of froll/broll then it's clear which end u r talking about

so df.shift(3,method='head') would give u 3 rows of nan followed by squashed data

this might be tricky as what if u can't fit the data in (after squashing) will it expand the frame?

[dsm054]

If we're viewing falling as a special shift, I don't think it should expand the frame because shift wouldn't. But I wonder if we should make it explicit that it's different from a normal shift in that it's value-dependent, not merely index-dependent, and call it valid_to_head or valid_to_tail or notnull_to_head or notnull_to_tail.

[Aside: I don't care much about the behaviour of the df.shift(3, method='somename') case because when I need this, I only ever need the non-nan values pushed to one of the edges.]