Support for dates older than Sep 21, 1677

[iarizex]

Problem Description

When I have dates older than Sep 21, 1677 in my dataset and try to process them as datetime, SDV complains with an error such as:

InvalidDataError: The provided data does not match the metadata:
Invalid values found for datetime column 'dateOfBirth': [15550217, 16560130].

Expected behavior

I would like SDV to support dates older than Sep 21, 1677.

Additional context

I know that SDV is ultimately trying to read datetime using a 64-bit format, being Sep 21, 1677 the earliest time that can be represented in 64-bit format. But it would be nice to be able to handle older dates.

[npatki]

Hi @iarizex thanks for filing this issue. Adding some more context from our Slack thread:

It seems like these old dates exist in your database due to user-input errors, as the actual column is dateOfBirth. These old dates (invalid data) could definitely be cleaned out manually, but you mentioned that it would be a good thing if the synthetic data actually reflected the messiness of the real data.

Possible Solutions

From SDV's end, there are actually several different ways we could achieve this:

1. We could try reading the datetimes using a longer format than 64-bit, allowing support for much older dates. This would be more useful if the dates were actually valid inputs, and you needed to learn patterns from such older dates.
2. We could keep the date format as 64-bit, which means we'd keep the minimum at Sep 21, 1677. However instead of erroring out on older dates, the system could be more resilient to out-of-bounds dates (remembering them separately and re-creating them in the synthetic data). Seems like this would be sufficient for your needs?

We will keep this feature request open as we track and find a solution to this problem.

Workaround

In case any other user comes across this, I am copy-pasting the workaround I had mentioned in the Slack thread:

In the meantime, as a workaround, would recommend perhaps adding a constant # of years to each of your datetimes so they are in the modern range. If the same constant is added to each datetime, it should not affect the overall synthetic data quality (you can subtract it back later).