How do the extraction with only one field without conversion?

[jmiguelsousa]

Hi,

First of all, thank you very much for this very useful library as this Cobol parser.

I am currently using it and I have a question. I am changing the way I read a cobol file. I have replaced a script we had by using this library.

One of the fields we want to extract without conversion, so I have used the debug option to true, because we have an ad hoc function to handle it.

This solution works fine, but it is generating all the columns without conversion, when it would be enough to generate only that column without conversion.

The question is if there is a way to do this, without having to use the debug option.

Thanks in advance.

Best regards,

Miguel

[yruslan]

Hi @jmiguelsousa,

Thanks a lot for the question!

Actually we have a similar requirements in one of our use cases. This is not supported yet, but eventually we can support an option like this:

.option("raw_fields", "GROUP1.FIELD1, GROUP2.FIELD2")

Do you also want unconverted fields to be of the type 'raw' (aka BinaryType) rather than 'hex', or you need both?

[jmiguelsousa]

Hi @yruslan,

Thank you very much for the quick reply.

In our case we need the field in "hex" format, but I think it may be useful to be able to choose the type of field format.

I suspected that this option was not yet supported in Cobrix. If it´s implemented in the future, I will use it.

Best regards

[yruslan]

Do you also need to keep both the converted field and the raw field. Or in your case you just need to have the raw field?

Btw, while thinking about this I've also come up with an idea for a workaround that would work even now.

If a field in a copybook is redefined with another field that has PIC X(10) COMP., e.g. X combined with COMP., the field should be converted to Spark as binary copy of the original field. E.g.

       05 FIELD_1                        PIC X(10).
       05 FIELD_1_RAW REDEFINES FIELD_1  PIC X(10) COMP.

The Spark schema will look like this:

root
 |-- FIELD_1: string (nullable = true)
 |-- FIELD_1_RAW: binary (nullable = true)

The binary field can be converted to HEX like this:

df.withColumn("FIELD_1_RAW", hex(col("FIELD_1_RAW")))