TypeError: dtype '<class 'datetime.datetime'>' not understood

[dickreuter]

df
  valuation_date
0     2018-06-08
1     2018-06-11
2     2018-06-12
3     2018-06-13
4     2018-06-14
5     2018-06-15

type(valuation_dates_df)
Out[16]: pandas.core.frame.DataFrame


type(valuation_dates_df['valuation_date'])
Out[15]: pandas.core.series.Series

To reproduce the error:

dict= {'valuation_date': {0: Timestamp('2018-06-08 00:00:00'),
  1: Timestamp('2018-06-11 00:00:00'),
  2: Timestamp('2018-06-12 00:00:00'),
  3: Timestamp('2018-06-13 00:00:00'),
  4: Timestamp('2018-06-14 00:00:00'),
  5: Timestamp('2018-06-15 00:00:00')}}

pd.DataFrame.from_records(dict)['valuation_date'].astype(datetime.datetime)

valuation_dates_df['valuation_date'].astype(datetime.datetime)
  File "C:\git\mre_x\_tcp\work\win-na-x64-release\py3\mre_venv3\lib\site-packages\pandas\core\dtypes\common.py", line 2029, in pandas_dtype
    raise TypeError("dtype '{}' not understood".format(dtype))
TypeError: dtype '<class 'datetime.datetime'>' not understood

The issue occurs with pandas 0.24.1 and did not occur in 0.23.4.

[TomAugspurger]

I'm not sure this should be expected to work. The dtype argument should be a dtype, not a type. I'd recommend .dt.to_pydatetime().

[mroeschke]

The docstring technically says a Python type should work (like int), though I'm not a fan of it in this case since .dt.to_pydatetime() is more idiomatic.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html

[TomAugspurger]

May be good to note that python types are eventually coerced to a dtype with np.dtype (or pandas_dtype) and so they should be types that NumPy knows how to handle natively.

…

On Thu, Mar 14, 2019 at 1:34 PM Matthew Roeschke ***@***.***> wrote: The docstring technically says a Python type should work (like int), though I'm not a fan of it in this case since .dt.to_pydatetime() is more idiomatic. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#25730 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIrX7dRdzFSKvv-BtX3bzHt6qBx1Rks5vWpY3gaJpZM4b0UeO> .

[MarcelBeining]

@TomAugspurger I have a very similar problem and the solution .dt.to_pydatetime(). does not work for me because I need to keep it in the data frame so that I can hand it all over to SQLAlchemy, and when I try df.date = df.date.dt.to_pydatetime() it simply transforms the datetime objects back to datetime64 which then does not work with SQLAlchemy :-(

[TomAugspurger]

@MarcelBeining I'd recommend applying date time.datetime elementwise to every value in the column, and overwriting the column with the (object dtype) output .

What remains to be done in this issue? Is it just changing the NDFrame.astype docs for dtype to update or Python type to or Python type, which is cast to a dtype with pandas_dtype,?

[brezniczky]

Sorry - might be a stupid question, but what would you suggest as the best approach to convert a datetime64 index column to native datetime? I don't know how I should reach for the .dt.to_pydatetime() option (at least in a straightforward manner, say without casting to a pandas.Series, which apparently I am falling back to) - is there an idiomatic way? I feel like I am missing a better option.

(I have some vague hope that it might be related to the design considerations with the original problem on account of consistency/learnability/intuitive use - apologies about the off otherwise.)

[WillAyd]

Depending on what you want tz_localize(None) or tz_convert(None). You can see these documented in the user guide:

https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#working-with-time-zones

[brezniczky]

Thank you in retrospect!

[abimael-dominguez]

I changed from this:

import pandas as pd
date.astype(datetime)

to this:

import pandas as pd
pd.to_datetime(date)

where 'date' is a pandas series of dates in string type before the casting.

[kenorb]

Same here happening in Colab.

TypeError: dtype '<class 'datetime.date'>' not understood

Code:

from matplotlib import dates as mdates
import datetime as dt
dfc = df.copy()
dfc = dfc.reset_index()
dfc['Date'] = mdates.date2num(dfc['Date'].astype(dt.date))
dfc.head()

Changed the line to (as per above suggestion):

dfc['Date'] = mdates.date2num(pd.to_datetime(dfc['Date']))

and it works.

Colab link: https://colab.research.google.com/github/LastAncientOne/Stock_Analysis_For_Quant/blob/master/Python_Stock/Technical_Indicators/RSI.ipynb

[matthewgdv]

Just bumping this issue. The docstring does imply that python types can be used as the first argument to Series.astype.

And it does work with other python types like int and float. Yes, it's possible to use pd.to_datetime, but for simple cases (for example, converting python dates to timestamps) it's annoying to have to break the symmetry between how you would coerce other types and how you coerce datetimes.

Personally, I'd much rather use a built-in type as an argument than a magic string like 'datetime64[ns]' where it's really easy to make a typo (or just forget what it was meant to be and be forced to context-switch to your browser) since your IDE can't help you out. I'd also prefer to use the Series.astype method on the Series object I'm working on rather than having to import a specialized function just for this.

Is there any reason we can't just special-case it like by adding a line like this at the top of the astype method:

if isinstance(dtype, type) and issubclass(dtype, dt.datetime):
    dtype = 'datetime64[ns]'

[jreback]

datetime.datetime is not a valid numpy type as it's under specified (no units);

-1 on any expansion here

[matthewgdv]

@jreback

By that logic neither should python int or float be allowed, but they currently are (int seems to default to int32).

So I don't see why we can't put practicality over purity here and let users have their shorthand conveniences.

I'd also like to point out that the alternative way to perform this operation:

df['some_col'] = pd.to_datetime(df['some_col'])

also doesn't specify any units. So I'm not sure that's a strong argument here.

EDIT:

On re-reading this comment I'm realizing it maybe comes off as a bit combative, particularly since text doesn't convey tone. I'm sorry if so. I really appreciate what you and the rest of the pandas maintainer team do for so many people. I just figured I'd raise something that has been a repeated annoyance to me in the past, but I respect that not everyone may agree and perhaps with good reason. There might be implementation details I'm unaware of that could make a change like this messier than it at first appears. In terms of the actual reasoning for why I think datetime.datetime should be a legal argument for astype, I hope I've made a decent case :p