BUG: duplicated() is reporting rows as duplicates when they aren't upon visual inspection.

[nvd2291]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

I got the creditcard.csv from Kaggle: https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud 

import pandas as pd
import numpy as np

credit_card_df = pd.read_csv('creditcard.csv')

duplicated_df = credit_card_df[credit_card_df.duplicated()]

duplicated_df

Issue Description

If you look at the output of the duplicated_df you can see rows 33 and 35 reported as duplicates when they aren't. The values are close but not exact duplicates

Expected Behavior

Would expect these rows to not be reported as duplicates because the values in the columns that aren't named 'Time' are not identical to each other.

Installed Versions

[asishm]

Thanks for the report. I think you are misunderstanding what df.duplicated() / df.duplicated(keep='first') does.

If rows 2, 3 and 4 are identical, then df.duplicated() will return False for row 2 and True for rows 3 and 4. So using a boolean index on the original dataframe would only contain rows 3 and 4. You could use df.duplicated(keep=False) which will return True for all duplicates including the first occurence.

You can see in the kaggle csv that rows 32,33 are identical and 34,35 are identical, therefore the duplicated method returns True for rows 33 and 35.

[nvd2291]

Thanks for the clairifcation. I didn't realize this