BUG: incompatible dtype when creating new columns with loc

[zbaibg]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Example1:

import pandas as pd
df = pd.DataFrame({"x": [1]})
df.loc[df["x"] == 1, ["y"]] = "1"

Output:
<stdin>:1: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '1' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.

Example2
import pandas as pd
df = pd.DataFrame()
df.loc[0,['x','y']]=[1,'1']

Output:
<stdin>:1: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '1' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.

Issue Description

The example 1 is very similar to the issue at #55025 . I only add a square bracket for the column name "y" in df.loc[] and it gives warning.

Example 2 is similar, but deal with multiple columns at once. This is what I need in my code.

Expected Behavior

Run the following codes and it gives no warning:

import pandas as pd
df = pd.DataFrame()
df.loc[0,['x','y']]=[1,'1']

Installed Versions

/home/bg/soft/miniconda3/lib/python3.11/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : fd3f571
python : 3.11.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.133.1-microsoft-standard-WSL2
Version : #1 SMP Thu Oct 5 21:02:42 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.0
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.2
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.12.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : 0.19.0
tzdata : 2023.4
qtpy : None
pyqt5 : None

[phofl]

cc @MarcoGorelli

[MarcoGorelli]

thanks @zbaibg for the report and @phofl for the ping

neither of these raise any warning on main, so they've been addressed

closing then

[MichalRIcar]

Hi,

I am writing to hear opinions on the best pratice regarding the recent change in silent dtype casting.
Mainly, the recommended approach with dtypes when using them for basic numerical operations/transformation (inside pipeline).

Below, I've outlined three examples to illustrate the issue:

(1) OLD CODE (behavior) will cause WARNING

D = pd.DataFrame({'C0':['A','B'], 'C1':[10,20]})
D.loc[:,'C1'] = D.loc[:,'C1']/100

FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[0.1 0.2]' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.

(2) SOLUTION A

D = pd.DataFrame({'C0':['A','B'],'C1':[10,20]})
D['C1'] = D['C1']/100

(3) SOLUTION B

D = pd.DataFrame({'C0':['A','B'], 'C1':[10,20]})
D = D.astype({'C1':float})
D.loc[:,'C1'] = D.loc[:,'C1']/100

From the proper pandas perspective, I believe Solution B (3) is the recommended approach after pandas>=v2.1.

However, when comparing code (1) to Solution (2), where code (1) warns and Solution (2) doesn't, I wonder if Solution (2) is optimal, especially when dealing with hundreds of columns.

I believe the issue (1) vs. (2) is that pandas has Performance Warning issue as D['C1'] treats transformation of the column C1 in the DataFrame differently (creates a new block at the dataframe backend structure) than D.loc[:,'C1'] (which changes the current block without creating one at the dataframe backend structure).

Obviously, the less extra lines of code, the better - thus I tend to choose (2) instead of (3). Historically (1) is in the code design as it is explicit way to communicate that we intentially transforming the whole column ( D.loc[:,...] ) and want to stay optimal to not create a new structure block at the dataframe backend (i.e. if we used D[...].)

Is my understanding of the best new pratice correct?

[leakyH]

Hi @MichalRIcar , I’ve also encountered this warning. It seems that this issue: #57229 (comment) discussed the same topic. And in this issue, it is recommended to use df.isetitem to replace the full column, which is

In cases where frame.columns is unique, this is equivalent to frame[frame.columns[i]] = value.
according to https://pandas.pydata.org/pandas-docs/version/2.1/reference/api/pandas.DataFrame.isetitem.html.

So maybe your solution A is recommended, even though it's not quite 'pandas-style' to me.

[MichalRIcar]

Hello @leakyH, thank you very much for providing the context, that makes life easier! I actually converged to it too as haven't seen any issues with it, happy to hear that there are two of us now.. :-)

[piotor87]

Hi all. Just to chime in because I"ve run into the same issue (with pandas 2.2.1). The weird thing is that it happens apparently when the data frame has only one value.

    err_df.loc[:,'ERR_VALUE'] = err_df.loc[:,col]

I was running the code on test data in small chunks and it would randomly appear, so i tried to check the composition of err_df and the warning would only appear in such case. If i run the same data in larger test chunks the warning does not appear instead.

python3 ~/Dropbox/Projects/kanta_lab_preprocessing/finngen_qc/main.py --raw-data /mnt/disks/data/kanta/test/tmp.txt --out /mnt/disks/data/kanta/test/ --chunk-size 200

Percent: [------------------->] 100%

Done.

vs

python3 ~/Dropbox/Projects/kanta_lab_preprocessing/finngen_qc/main.py --raw-data /mnt/disks/data/kanta/test/tmp.txt --out /mnt/disks/data/kanta/test/ --chunk-size 100

Percent: [--------->          ] 50%/home/pete/Dropbox/Projects/kanta_lab_preprocessing/finngen_qc/filters/filter_minimal.py:63: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '['K']' has dtype incompatible with string, please explicitly cast to a compatible dtype first.

  err_df.loc[:,'ERR_VALUE'] = err_df.loc[:,col]

Percent: [------------------->] 100%

Done.

And this is confirmed because if i add an exception:

if len(err_df)>1:
       err_df.loc[:,'ERR_VALUE'] = err_df.loc[:,col]

The same code runs without warning

[jacktang]

My panda is v2.2.2 and still the warning comes up. So how to fix it?