Skip to content

Commit 2436c56

Browse files
ArjunJagdalesevero
andauthored
Add support for Date features in Croissant schema (#3223)
* Update croissant_utils.py * Update croissant_utils.py * Update croissant_utils.py ImportError: cannot import name 'Date' from 'datasets' * Changes as per the spec and `mlcroissant` removed `sc:Time` and now only use `sc:Date` and `sc:DateTime`. Also added a fallback import to avoid CI issues on older `datasets` versions. * Update croissant_utils.py * Update croissant_utils.py * Update croissant_utils.py * Update croissant_utils.py --------- Co-authored-by: Sylvain Lesage <[email protected]>
1 parent cf79aee commit 2436c56

File tree

1 file changed

+9
-3
lines changed

1 file changed

+9
-3
lines changed

libs/libcommon/src/libcommon/croissant_utils.py

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -77,9 +77,14 @@ def truncate_features_from_croissant_crumbs_response(content: Mapping[str, Any])
7777
"int64": "cr:Int64",
7878
"large_string": "sc:Text",
7979
"string": "sc:Text",
80-
"time32": "sc:Date",
81-
"time64": "sc:Date",
82-
"timestamp[ns]": "sc:Date",
80+
"time32[s]": "sc:Time",
81+
"time32[ms]": "sc:Time",
82+
"time64[us]": "sc:Time",
83+
"time64[ns]": "sc:Time",
84+
"timestamp[s]": "sc:DateTime",
85+
"timestamp[ms]": "sc:DateTime",
86+
"timestamp[us]": "sc:DateTime",
87+
"timestamp[ns]": "sc:DateTime",
8388
"uint8": "cr:UInt8",
8489
"uint16": "cr:UInt16",
8590
"uint32": "cr:UInt32",
@@ -202,4 +207,5 @@ def feature_to_croissant_field(
202207
field["isArray"] = True
203208
field["arrayShape"] = ",".join([str(shape) if shape else "-1" for shape in array_shape])
204209
return field
210+
205211
return None

0 commit comments

Comments
 (0)