Date and Timestamp

[pinakigit]

Background

We have DB2 tables where fields are declared as Date and Timestamp. When files are downloaded, the Copybook defines these fields as alphanumeric:
Date: X(10)
Timestamp: X(26)

When these files are moved to a distributed environment and read through Cobrix, the fields are treated as String, as the Cobol Copybook defines them as alphanumeric.

Feature

For these date fields, can we modify the Copybook definition to something other than X(10) or X(26) , in order to explicitly indicate that they represent Date and Timestamp fields? This change would allow Cobrix to correctly interpret them as Date and Timestamp data types when creating the DataFrame.

Example [Optional]

A simple example if applicable.

Proposed Solution [Optional]

Solution Ideas
1.
2.
3.

[yruslan]

Hi @pinakigit ,

Thanks for such a detailed feature request. However, there is not standard date and timestamp representation across all mainframe systems, so we can't implement this in Cobrix for a general use case.

In our use cases we cast fields after loading the dataframe, e.g.

val df = spark.read.format("cobol")...load("file.dat")
  .withColumn("DATE_FILED", to_date(col("DATE_FILED"), "yyyy-MM-dd")) 

It does not have performance impact because operations are executed lazily.