BUG: Raise MergeError when suffixes result in duplicate column names … #61422

Farsidetfs · 2025-05-09T23:21:41Z

All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Farsidetfs · 2025-05-10T02:51:41Z

pre-commit.ci autofix

chilin0525 · 2025-05-10T05:57:34Z

Thanks for your contribution!

Just a quick note: you don't need to write GH#61402 in the PR description — simply using #61402 in PR description, is enough, GitHub will automatically link it 😀.
Also, since this PR addresses a bug, please make sure to:

Add a unit test that covers this case
Include an entry in the doc/source/whatsnew/vx.y.z.rst file to document your fix

For reference, you can check the contributing guidelines here: https://pandas.pydata.org/docs/development/contributing_codebase.html#documenting-your-code

Farsidetfs · 2025-05-10T07:14:13Z

Thanks for the pointers. I'll get those added in here soon. Trying to track down why the Unit Tests / Linux-32-bit(pull_request) is failing. I didn't change anything that should have effected Series, so it's kinda weird.

I also can't get the pytest to run normally on my dev yet either, so I haven't been able to fully replicate the failure locally yet. So, still a little more work to do here.

chilin0525 · 2025-05-10T07:19:47Z

@Farsidetfs I believe the CI failure is not related to your changes. It appears to be caused by the cython version — pandas unit tests fail with cython==3.1.0. You may notice that the same test failures have occurred in several recent PRs as well. I already address the issue in #61423.

nikaltipar

Added a few comments from a bird's eye view. Thanks for the change.

nikaltipar · 2025-05-10T08:15:16Z

pandas/core/reshape/merge.py

+    if len(left_collisions) > 0:
+        raise MergeError(
+            "Passing 'suffixes' which cause duplicate columns "
+            f"{set(left_collisions)} is not allowed"
+        )
+    if len(right_collisions) > 0:
+        raise MergeError(
+            "Passing 'suffixes' which cause duplicate columns "
+            f"{set(right_collisions)} is not allowed"
+        )


I would recommend you combine this into a common error to reduce repetition (bonus points for combining it with the pre-existing error just a few lines below)

nikaltipar · 2025-05-10T08:26:58Z

pandas/core/reshape/merge.py

+    # Check for duplicates created by suffixes
+    left_collisions = llabels.intersection(right.difference(to_rename))
+    right_collisions = rlabels.intersection(left.difference(to_rename))
+    if len(left_collisions) > 0:


Doesn't just if not left_collisions.empty: work? Same for a similar check below.

nikaltipar · 2025-05-10T08:36:02Z

pandas/core/reshape/merge.py

    llabels = left._transform_index(lrenamer)
    rlabels = right._transform_index(rrenamer)

+    # Check for duplicates created by suffixes


For new readers of this code, the comment might not be descriptive enough. While your code is supposed to find suffixes that are caused by duplicated would-be created columns across dataframes, there is an extra section that does a duplicate checking just below your new code (but would-be created columns due to duplicity within the same dataframe).

Farsidetfs · 2025-05-12T21:23:26Z

pre-commit.ci autofix

Farsidetfs · 2025-05-12T22:48:33Z

pre-commit.ci autofix

Farsidetfs · 2025-05-13T00:06:45Z

pre-commit.ci autofix

Farsidetfs · 2025-05-13T01:15:21Z

pre-commit.ci autofix

Farsidetfs · 2025-05-13T02:45:26Z

@nikaltipar I think this should be ready now. Please let me know if I've missed anything. I took your advice and combined the two with slight modifications to improve efficiency using sets throughout rather than just convert at the end.

nikaltipar · 2025-05-14T11:13:03Z

@nikaltipar I think this should be ready now. Please let me know if I've missed anything. I took your advice and combined the two with slight modifications to improve efficiency using sets throughout rather than just convert at the end.

Thanks for taking care of that, @Farsidetfs ! It looks good to me, no other comments from my side. Thanks for adding the unit-tests, too!

chilin0525 · 2025-05-14T20:50:15Z

@nikaltipar Could you rebase main branch to trigger CI again?

nikaltipar · 2025-05-14T21:46:20Z

@nikaltipar Could you rebase main branch to trigger CI again?

I am not able to, I'll have to wait for @Farsidetfs

Farsidetfs · 2025-05-15T05:11:24Z

Rebase complete. Thanks. Let me know if there's anything else needed.

nikaltipar · 2025-05-15T10:40:09Z

#61402

rhshadrach

Looking good!

rhshadrach · 2025-05-19T02:24:11Z

doc/source/whatsnew/v2.3.0.rst

 Reshaping
 ^^^^^^^^^
-
+- Bug in :meth:`DataFrame.merge` where user-provided suffixes could result in duplicate column names if the resulting names matched existing columns. Now raises a :class:`MergeError` in such cases. (:issue:`61402`)


Can you move this note to v3.0.0

rhshadrach · 2025-05-19T02:31:59Z

pandas/tests/reshape/merge/test_merge.py

+    df1 = DataFrame({"col1": [1], "col2": [2]})
+    df2 = DataFrame({"col1": [1], "col2": [2], "col2_dup": [3]})
+    with pytest.raises(MergeError, match="duplicate columns"):
+        merge(df1, df2, on="col1", suffixes=("_dup", ""))


Can you parametrize this test with

@pytest.mark.parametrize("suffixes", [("_dup", ""), ("", "_dup")])

@rhshadrach Done. Please let me know if I missed anything.

pandas/core/reshape/merge.py

…(GH#61402)

Farsidetfs · 2025-05-24T17:20:40Z

@rhshadrach All requested changes are complete, so it should be ready for your review to unblock merge. Thanks

rhshadrach

lgtm

rhshadrach · 2025-05-30T18:31:31Z

@Farsidetfs - just a conflict that needs resolved in the whatsnew.

datapythonista · 2025-06-02T18:59:30Z

pre-commit.ci autofix

Farsidetfs · 2025-06-05T23:18:54Z

pre-commit.ci autofix

…61402

Farsidetfs · 2025-06-06T00:36:18Z

@rhshadrach Merge conflicts resolved, ready for merge

datapythonista · 2025-06-06T14:12:40Z

Thanks @Farsidetfs very nice

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch 2 times, most recently from fe71e1a to c9bfc5a Compare May 10, 2025 02:48

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch from dc1bb47 to ad744df Compare May 10, 2025 05:54

chilin0525 mentioned this pull request May 10, 2025

BUG: Duplicate columns allowed on merge if originating from separate dataframes #61402

Closed

3 tasks

nikaltipar reviewed May 10, 2025

View reviewed changes

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch from ad744df to 9a40cd0 Compare May 12, 2025 20:45

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch 3 times, most recently from e9efd0d to 1476957 Compare May 12, 2025 22:35

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch from cb23817 to 6b82c85 Compare May 13, 2025 00:02

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch from 5aff565 to e62c70f Compare May 13, 2025 01:11

nikaltipar approved these changes May 14, 2025

View reviewed changes

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch from 7d3837a to c377804 Compare May 15, 2025 04:03

nikaltipar approved these changes May 15, 2025

View reviewed changes

rhshadrach requested changes May 19, 2025

View reviewed changes

rhshadrach added the Bug label May 19, 2025

rhshadrach added the Reshaping Concat, Merge/Join, Stack/Unstack, Explode label May 19, 2025

rhshadrach added this to the 3.0 milestone May 19, 2025

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch 2 times, most recently from b7b3a95 to bc62afe Compare May 22, 2025 03:49

Farsidetfs requested a review from rhshadrach May 22, 2025 05:26

mroeschke reviewed May 22, 2025

View reviewed changes

pandas/core/reshape/merge.py Outdated Show resolved Hide resolved

BUG: Raise MergeError when suffixes result in duplicate column names …

9bc2b66

…(GH#61402)

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch from 5b816ce to 9bc2b66 Compare May 23, 2025 19:20

mroeschke approved these changes May 23, 2025

View reviewed changes

rhshadrach approved these changes May 30, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into fix-merge-suffixes-…

fdb5861

…61402

Farsidetfs force-pushed the fix-merge-suffixes-61402 branch from 12ce6ce to fdb5861 Compare June 5, 2025 23:33

datapythonista merged commit 297bec4 into pandas-dev:main Jun 6, 2025
44 checks passed

Uh oh!

BUG: Raise MergeError when suffixes result in duplicate column names … #61422

BUG: Raise MergeError when suffixes result in duplicate column names … #61422

Uh oh!

Conversation

Farsidetfs commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Farsidetfs commented May 10, 2025

Uh oh!

chilin0525 commented May 10, 2025

Uh oh!

Farsidetfs commented May 10, 2025

Uh oh!

chilin0525 commented May 10, 2025

Uh oh!

nikaltipar left a comment

Choose a reason for hiding this comment

Uh oh!

nikaltipar May 10, 2025

Choose a reason for hiding this comment

Uh oh!

nikaltipar May 10, 2025

Choose a reason for hiding this comment

Uh oh!

nikaltipar May 10, 2025

Choose a reason for hiding this comment

Uh oh!

Farsidetfs commented May 12, 2025

Uh oh!

Farsidetfs commented May 12, 2025

Uh oh!

Farsidetfs commented May 13, 2025

Uh oh!

Farsidetfs commented May 13, 2025

Uh oh!

Farsidetfs commented May 13, 2025

Uh oh!

nikaltipar commented May 14, 2025

Uh oh!

chilin0525 commented May 14, 2025

Uh oh!

nikaltipar commented May 14, 2025

Uh oh!

Farsidetfs commented May 15, 2025

Uh oh!

nikaltipar commented May 15, 2025

Uh oh!

rhshadrach left a comment

Choose a reason for hiding this comment

Uh oh!

rhshadrach May 19, 2025

Choose a reason for hiding this comment

Uh oh!

rhshadrach May 19, 2025

Choose a reason for hiding this comment

Uh oh!

Farsidetfs May 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Farsidetfs commented May 24, 2025

Uh oh!

rhshadrach left a comment

Choose a reason for hiding this comment

Uh oh!

rhshadrach commented May 30, 2025

Uh oh!

datapythonista commented Jun 2, 2025

Uh oh!

Farsidetfs commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Farsidetfs commented Jun 6, 2025

Uh oh!

Uh oh!

datapythonista commented Jun 6, 2025

Uh oh!

Reviewers

Assignees

Farsidetfs commented May 9, 2025 •

edited

Loading

Farsidetfs commented Jun 5, 2025 •

edited

Loading