Parse Unblocked EBCIDIC File for different cobal copybooks

[divakardatapro]

Background [Optional]

I have a EBCDIC file which will have three types of records (information) based on first 4 or 6 bytes of the record we need to apply the different copybook(we have three cobol copybooks one is to read header records , 2 is to read some base records and 3 is to read some details records)

Question

how can we parse the file using cobrix for different copy books on the same EBCDIC file. can someone help on this?

[yruslan]

Yes, usually is is done by combining multiple copybooks into one like this:

01 RECORD.
   02  HEADER     X(4).
   02  COPYBOOK_1_RECORD.
      04 FIELD1 X(1).
       ...
   02  COPYBOOK_2_RECORD REDEFINES COPYBOOK_1_RECORD.
      04 FIELD100 X(1).
       ...
   02  COPYBOOK_3_RECORD REDEFINES COPYBOOK_3_RECORD.
      04 FIELD200 X(1).
       ...

then, you can make Cobrix automatically filter records based on the header field:

val df = spark
  .read
  .format("cobol")
  .option("copybook_contents", copybook)
  .option("segment_field", "HEADER")
  .option("redefine_segment_id_map:0", "COPYBOOK_1_RECORD => T1")
  .option("redefine_segment_id_map:1", "COPYBOOK_2_RECORD => T2,T3")
  .option("redefine_segment_id_map:1", "COPYBOOK_2_RECORD => T4")
  .load("/path/to/file")

In this example,

• when HEADER='T1', COPYBOOK_1_RECORD will be applied,
• when HEADER='T2' or 'T3, COPYBOOK_2_RECORD will be applied,
• when HEADER='T4', COPYBOOK_3_RECORD will be applied.