ENH: improve support for datetime columns (#486)

Co-authored-by: Joris Van den Bossche <[email protected]>

Author: theroggy

CHANGES.md

@@ -10,6 +10,9 @@
 
 ### Improvements
 
+-   Add `datetime_as_string` and `mixed_offsets_as_utc` parameters to `read_dataframe`
+    to choose the way datetime columns are returned + several fixes when reading and
+    writing datetimes (#486).
 -   Add listing of GDAL data types and subtypes to `read_info` (#556).
 -   Add support to read list fields without arrow (#558, #597).
 
@@ -183,7 +186,7 @@
 
 ### Improvements
 
--   Support reading and writing datetimes with timezones (#253).
+-   Support reading and writing datetimes with time zones (#253).
 -   Support writing dataframes without geometry column (#267).
 -   Calculate feature count by iterating over features if GDAL returns an
     unknown count for a data layer (e.g., OSM driver); this may have signficant

docs/source/introduction.md

@@ -481,13 +481,17 @@ Not all file formats have dedicated support to store datetime data, like ESRI
 Shapefile. For such formats, or if you require precision > ms, a workaround is to
 convert the datetimes to string.
 
-Timezone information is preserved where possible, however GDAL only represents
-time zones as UTC offsets, whilst pandas uses IANA time zones (via `pytz` or
-`zoneinfo`). This means that dataframes with columns containing multiple offsets
-(e.g. when switching from standard time to summer time) will be written correctly,
-but when read via `pyogrio.read_dataframe()` will be returned as a UTC datetime
-column, as there is no way to reconstruct the original timezone from the individual
-offsets present.
+When you have datetime columns with time zone information, it is important to
+note that GDAL only represents time zones as UTC offsets, whilst pandas uses
+IANA time zones (via `pytz` or `zoneinfo`). As a result, even if a column in a
+DataFrame contains datetimes in a single time zone, this will often still result
+in mixed time zone offsets being written for time zones where daylight saving
+time is used (e.g. +01:00 and +02:00 offsets for time zone Europe/Brussels).
+When roundtripping through GDAL, the information about the original time zone
+is lost, only the offsets can be preserved. By default,
+{func}`pyogrio.read_dataframe()` will convert columns with mixed offsets to UTC
+to return a datetime64 column. If you want to preserve the original offsets,
+you can use `datetime_as_string=True` or `mixed_offsets_as_utc=False`.
 
 ## Dataset and layer creation options
 

pyogrio/_io.pyx

@@ -980,10 +980,16 @@ cdef process_fields(
 
             if datetime_as_string:
                 # defer datetime parsing to user/ pandas layer
-                # Update to OGR_F_GetFieldAsISO8601DateTime when GDAL 3.7+ only
-                data[i] = get_string(
-                    OGR_F_GetFieldAsString(ogr_feature, field_index), encoding=encoding
-                )
+                IF CTE_GDAL_VERSION >= (3, 7, 0):
+                    data[i] = get_string(
+                        OGR_F_GetFieldAsISO8601DateTime(ogr_feature, field_index, NULL),
+                        encoding=encoding,
+                    )
+                ELSE:
+                    data[i] = get_string(
+                        OGR_F_GetFieldAsString(ogr_feature, field_index),
+                        encoding=encoding,
+                    )
             else:
                 success = OGR_F_GetFieldAsDateTimeEx(
                     ogr_feature,
@@ -1602,6 +1608,7 @@ def ogr_open_arrow(
     int return_fids=False,
     int batch_size=0,
     use_pyarrow=False,
+    datetime_as_string=False,
 ):
 
     cdef int err = 0
@@ -1819,6 +1826,12 @@ def ogr_open_arrow(
                 "GEOARROW".encode("UTF-8")
             )
 
+        # Read DateTime fields as strings, as the Arrow DateTime column type is
+        # quite limited regarding support for mixed time zones,...
+        IF CTE_GDAL_VERSION >= (3, 11, 0):
+            if datetime_as_string:
+                options = CSLSetNameValue(options, "DATETIME_AS_STRING", "YES")
+
         # make sure layer is read from beginning
         OGR_L_ResetReading(ogr_layer)
 

pyogrio/_ogr.pxd

@@ -423,6 +423,14 @@ cdef extern from "ogr_api.h":
         OGRLayerH hLayer, ArrowArrayStream *out_stream, char** papszOptions
     )
 
+IF CTE_GDAL_VERSION >= (3, 7, 0):
+
+    cdef extern from "ogr_api.h":
+        const char* OGR_F_GetFieldAsISO8601DateTime(
+            OGRFeatureH feature, int n, char** papszOptions
+        )
+
+
 IF CTE_GDAL_VERSION >= (3, 8, 0):
 
     cdef extern from "ogr_api.h":