Support GRIB2 DRS 42

[rschmunk]

This seems to be a problem with 5.4-MAINT up to the current (7?) development branch.

A user sent me a GRIB2 file that netCDF-Java will open, but when one then tries to extract a variable, an "Unsupported DRS type = 42" exception is thrown by the Grib2DataReader.factory method. (But note that the same exception could be thrown by Grib2DataReader.getData.)

Comparison of the switch statements in these methods to ECMWF's data representation template list, several such templates are not handled. But for the moment, 42 is the one of concern.

A stack trace is appended (using 5.4-MAINT), but note that the line numbers in Grib2Drs and Grib2DataReader are slightly off as I inserted some logging.

A sample file that demonstrates the problem is here on Dropbox.

The user also comments that: "As far as I have understood, this file has been generated by using ECMWF's ECCODES library and it originates from MetCoOp (251) center."

--

java.lang.RuntimeException: Unsupported DRS type = 42
	at ucar.nc2.grib.grib2.Grib2Drs.factory(Grib2Drs.java:39)
	at ucar.nc2.grib.grib2.Grib2SectionDataRepresentation.getDrs(Grib2SectionDataRepresentation.java:83)
	at ucar.nc2.grib.grib2.Grib2Record.readData(Grib2Record.java:319)
	at ucar.nc2.grib.collection.GribDataReader$Grib2DataReader.readData(GribDataReader.java:474)
	at ucar.nc2.grib.collection.GribDataReader.read(GribDataReader.java:273)
	at ucar.nc2.grib.collection.GribDataReader.readDataFromCollection(GribDataReader.java:116)
	at ucar.nc2.grib.collection.GribDataReader.readData(GribDataReader.java:81)
	at ucar.nc2.grib.collection.GribIosp.readData(GribIosp.java:1085)
	at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122)
	at ucar.nc2.Variable.reallyRead(Variable.java:797)
	at ucar.nc2.Variable._read(Variable.java:736)
	at ucar.nc2.Variable.read(Variable.java:614)
	at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:459)
	at ucar.nc2.dataset.VariableDS._read(VariableDS.java:432)
	at ucar.nc2.dataset.VariableDS._read(VariableDS.java:442)
	at ucar.nc2.Variable.read(Variable.java:600)
	at ucar.nc2.Variable.read(Variable.java:546)
	at gov.nasa.giss.data.nc.array.NcArray2D.doSlice(NcArray2D.java:363)
...

[rschmunk]

Also, it seems that this issue involves use of a Lambert Conformal Conic projected grid.

Note that I posted an issue two years ago involving use of a Lambert Azimuthal Equal Area projected grid, although that manifested as an inability to even open the dataset. See #22.

[lesserwhirls]

Comparison of the switch statements in these methods to ECMWF's data representation template list, several such templates are not handled. But for the moment, 42 is the one of concern.

Usually a DRS isn't implemented due to lack of a sample file. That said, DRS 42 looks to be pretty complicated:

https://library.wmo.int/doc_num.php?explnum_id=10722 (pdf page 132)

Let me see if there is a java port of libaec out there that we could use. Otherwise, we'd either have to write our own implementation or start wrapping more C code, neither of which sound like great options.

[rschmunk]

Oh, ugh. Looking at that PDF page. 😱

[rschmunk]

Ping! I just had another request for DRS 42 from a developer at ECMWF.

[shahramn]

Thanks for chasing this up. From cycle 48R1 ECMWF data by default will use CCSDS packing ( for gridded fields )

[rschmunk]

Ping! I just had yet another query about DRS 42 from someone trying to work with ECMWF data.

[haileyajohnson]

@rschmunk it would be great to have this, but unfortunately our dev teams hands are really full right now, so unless we get some outside help on this issue, it's not going to make it to the top of the TODO list for a while.

[rschmunk]

Unfortunately, as @lesserwhirls pointed out, this one looks like a hairball.

Maybe some ECMWF person will get attached to the issue and write some code?

[Yaqiang]

GRIB2 DRS 42 support request + 1.

[NZTSL]

Another GRIB2 DRS 42 support request here, thanks.

[rschmunk]

@haileyajohnson, I hate to say it but I just heard from another user whose ability to view ECMWF data has been crippled because ECMWF is now shipping the product using data representation 42.

Edit: This looks like the same or related product as what @shahramn mentioned on Feb. 20.

[haileyajohnson]

So I 100% hear that this is becoming a major issue. The problems we previously listed though still remain. I think this is going to involve us doing some major work on compression in general in netcdf-java, which wouldn't be the worst thing. I don't know how long it would take to plan/design/develop though, so I guess my question is what is the impact to everyone +1ing this issue if it takes 3/6/9 months before we're able to ship this as a feature?

[cvasquez-github]

GRIB2 DRS 42 support request + 1