Commit 29fc559
[SPARK-55296][PS][FOLLOW-UP] Fix CoW mode not to break groupby
### What changes were proposed in this pull request?
This is a follow-up of #54375.
Fixes CoW mode not to break `groupby`.
Delays to disconnect the anchor to when actually being updated.
### Why are the changes needed?
The CoW mode was supported at #54375, but it disconnected the anchor too early, causing to break `groupby`.
```py
>>> import pandas as pd
>>> import pyspark.pandas as ps
>>>
>>> pdf1 = pd.DataFrame({"C": [0.362, 0.227, 1.267, -0.562], "B": [1, 2, 3, 4]})
>>> pdf2 = pd.DataFrame({"A": [1, 1, 2, 2]})
>>>
>>> psdf1 = ps.from_pandas(pdf1)
>>> psdf2 = ps.from_pandas(pdf2)
>>>
>>> pdf1.groupby([pdf1.C, pdf2.A]).agg("sum").sort_index()
B
C A
-0.562 2 4
0.227 1 2
0.362 1 1
1.267 2 3
>>> psdf1.groupby([psdf1.C, psdf2.A]).agg("sum").sort_index()
C B
C A
-0.562 2 -0.562 4
0.227 1 0.227 2
0.362 1 0.362 1
1.267 2 1.267 3
```
### Does this PR introduce _any_ user-facing change?
Yes, it will behave more like pandas 3.
### How was this patch tested?
The existing tests should pass.
### Was this patch authored or co-authored using generative AI tooling?
Codex (GPT-5.3-Codex)
Closes #54392 from ueshin/issues/SPARK-55296/fix_groupby.
Authored-by: Takuya Ueshin <ueshin@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>1 parent 434aa7a commit 29fc559
2 files changed
+16
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
587 | 587 | | |
588 | 588 | | |
589 | 589 | | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
590 | 600 | | |
591 | 601 | | |
592 | 602 | | |
| |||
811 | 821 | | |
812 | 822 | | |
813 | 823 | | |
814 | | - | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
815 | 829 | | |
816 | 830 | | |
817 | 831 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
430 | 430 | | |
431 | 431 | | |
432 | 432 | | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
| 433 | + | |
437 | 434 | | |
438 | 435 | | |
439 | 436 | | |
| |||
0 commit comments