Skip to content

Conversation

ollz272
Copy link
Contributor

@ollz272 ollz272 commented Sep 23, 2024

Change Summary

Related to an issue i've opened in pydantic, adds a new config option to customise how pydantic serializes datetimes.

We now from this PR have 3 options:

  1. The default iso8601, which retains existing behaviour
  2. seconds_int, which returns the timestamp of the datetime in seconds
  3. milliseconds_int, which returns the timestamp of the datetime in milliseconds

This would be a worthwhile feature for us to have, so hope this is okay to implement!

Related issue number

pydantic/pydantic#10454

Checklist

  • Unit tests for the changes exist
  • Documentation reflects the changes where applicable
  • Pydantic tests pass with this pydantic-core (except for expected changes)
  • My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @sydney-runkle

Copy link

codecov bot commented Sep 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.26%. Comparing base (ab503cb) to head (f12edec).
Report is 184 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1465      +/-   ##
==========================================
- Coverage   90.21%   89.26%   -0.95%     
==========================================
  Files         106      112       +6     
  Lines       16339    17896    +1557     
  Branches       36       41       +5     
==========================================
+ Hits        14740    15975    +1235     
- Misses       1592     1901     +309     
- Partials        7       20      +13     
Files with missing lines Coverage Δ
python/pydantic_core/core_schema.py 94.78% <100.00%> (+0.01%) ⬆️
src/errors/validation_exception.rs 92.95% <100.00%> (+0.08%) ⬆️
src/serializers/config.rs 95.83% <100.00%> (+1.38%) ⬆️
src/serializers/extra.rs 96.41% <100.00%> (-3.12%) ⬇️
src/serializers/infer.rs 90.96% <100.00%> (-4.16%) ⬇️
src/serializers/mod.rs 100.00% <100.00%> (ø)
src/serializers/shared.rs 79.25% <ø> (+0.04%) ⬆️
src/serializers/type_serializers/datetime_etc.rs 100.00% <100.00%> (ø)

... and 46 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4aa52a8...f12edec. Read the comment docs.

Copy link

codspeed-hq bot commented Sep 23, 2024

CodSpeed Performance Report

Merging #1465 will not alter performance

Comparing ollz272:add-datetime-serializer (6226f2b) with main (4aa52a8)

Summary

✅ 155 untouched benchmarks

@ollz272 ollz272 marked this pull request as ready for review September 23, 2024 17:51
@ollz272
Copy link
Contributor Author

ollz272 commented Sep 24, 2024

please review

}

pub(crate) fn datetime_to_seconds(py_dt: &Bound<'_, PyDateTime>) -> PyResult<i64> {
pydatetime_as_datetime(py_dt).map(|dt| dt.timestamp())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, seems to be an issue here :(

Looks like dt.timestamp() here doesn't take into account the number of microseconds in the datetime, so anything under a second resolution will get ignored.

Looks like its a speed date thing:
https://github.com/pydantic/speedate/blob/d15ac62b7e98d6e4b8a769b6f5ffab2ad1db5ca5/src/datetime.rs#L580-L582
https://github.com/pydantic/speedate/blob/main/src/time.rs#L388-L393

So i guess a few options:

  1. We don't care, ignore (i dont think this is right)
  2. We care and can calculate the timestamp ourselves here
  3. We care and need to add something upstream to speeddate to allow for returning the timestamp as a f64 with microsecond precision

🤷🏻

'milliseconds_int',
),
(
datetime(2024, 1, 1, 0, 0, 0, 100, ),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this situation fails :(

Comment on lines 382 to +383
timedelta_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_float'` or `'milliseconds_float'`.
datetime_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_int'` or `'milliseconds_int'`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
timedelta_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_float'` or `'milliseconds_float'`.
datetime_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_int'` or `'milliseconds_int'`.
timedelta_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_float'` or `'milliseconds_float'`.
datetime_mode: How to serialize `datetime` objects, either `'iso8601'`, `'seconds_int'` or `'milliseconds_int'`.

exclude_none: Whether to exclude fields that have a value of `None`.
round_trip: Whether to enable serialization and validation round-trip support.
timedelta_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_float'`, or`'milliseconds_float'`.
datetime_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_int'`, or`'milliseconds_int'`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
datetime_mode: How to serialize `timedelta` objects, either `'iso8601'`, `'seconds_int'`, or`'milliseconds_int'`.
datetime_mode: How to serialize `datetime` objects, either `'iso8601'`, `'seconds_int'`, or`'milliseconds_int'`.

@sydney-runkle
Copy link
Contributor

Closing for now, but we're going to implement this in v2.11 :).

Copy link
Contributor

@sydney-runkle sydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why these didn't register earlier, but going to go ahead and publish these for my reference later :)

Str: super::type_serializers::string::StrSerializer;
Bytes: super::type_serializers::bytes::BytesSerializer;
Datetime: super::type_serializers::datetime_etc::DatetimeSerializer;
Datetime: super::type_serializers::datetime_etc::DateTimeSerializer;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's stick with DatetimeSerializer, given that datetime is one word.

Comment on lines +216 to +218
Self::Iso8601 => Ok(datetime_to_string(datetime)?.into_py(py)),
Self::SecondsInt => Ok(datetime_to_seconds(datetime)?.into_py(py)),
Self::MillisecondsInt => Ok(datetime_to_milliseconds(datetime)?.into_py(py)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it feels a bit odd to me that we're pulling in serialization functions into our config logic, though I understand the utility here.

@davidhewitt, any idea what the best approach for a refactor would be here so that we get the same behavior? I don't have any great ideas given that this generally follows the pattern that the other modes have.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any chance to get a SecondsStr, which would be equivalent to doing dt.isoformat(timespec="seconds"), or a Iso8601 with microseconds being stripped?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants