Preserve Alignment Between Index and Values for Non-Monotonic Stack

[WillAyd]

• closes Data is mismatched with labels after stack with MultiIndex columns #20945
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Not overly familiar with this code so submitting for review as there's probably a better way of going about it. The root cause of the referenced issue IIUC is that the index labels of the caller are non-monotonic. stack essentially takes values and labels from the level that is getting pushed down into the rows with an implicit assumption that both are monotonic, hence the index/values get misaligned.

This breaks at least one other test so not ready to merge, but looking for feedback on:

• If there's a better way to align the values with the labels in this function AND/OR
• If we make any guarantees about the order of the labels for the level(s) being moved in this function

[pep8speaks]

Hello @WillAyd! Thanks for updating the PR.

• In the file pandas/tests/frame/test_reshape.py, following are the PEP8 issues :

Line 427:80: E501 line too long (81 > 79 characters)

Comment last updated on May 14, 2018 at 21:28 Hours UTC

[jreback]

we have a sort_monotonic function in a MI to do this

[WillAyd]

Good to know, though I must have explained the issue incorrectly. Within the call to _stack_multi_columns on master here's how the column index of the failing example looks:

MultiIndex(levels=[['A', 'B'], ['a', 'b', 'c', 'd']],
           labels=[[0, 0, 0, 0, 1, 1, 1, 1], [2, 1, 0, 3, 2, 1, 0, 3]],
           names=['dim2', 'foo'])

I built another DataFrame manually which was equivalent (at least according to tm.assert_frame_equal) and had a column index that look as follows:

MultiIndex(levels=[['A', 'B'], ['c', 'b', 'a', 'd']],
           labels=[[0, 0, 0, 0, 1, 1, 1, 1], [0, 1, 2, 3, 0, 1, 2, 3]],
           names=['dim2', 'foo'])

The former yielded incorrect results at the end of the stack operation but the latter was fine, even though they were coming from two frames that look exactly the same.

I believe the problem is that when iterating over the groups, _stack_multi_columns gets a slice of the frames' values. With the latter, getting a slice of the 4 values at a time keeps it aligned with the second level labels since they are sequential. That does not apply to the former example hence the issue.

Is the fact that the values of the DataFrame do not align with the labels of the column index by design?

[jreback]

eg.

In [2]: mi = pd.MultiIndex(levels=[['A', 'B'], ['c', 'b', 'a', 'd']],
   ...:            labels=[[0, 0, 0, 0, 1, 1, 1, 1], [0, 1, 2, 3, 0, 1, 2, 3]],
   ...:            names=['dim2', 'foo'])
   ...:            

In [3]: mi._sort_levels_monotonic()
Out[3]: 
MultiIndex(levels=[['A', 'B'], ['a', 'b', 'c', 'd']],
           labels=[[0, 0, 0, 0, 1, 1, 1, 1], [2, 1, 0, 3, 2, 1, 0, 3]],
           names=['dim2', 'foo'])

[WillAyd]

solved via #21043