EnumTypedef 'null' does not exist in a parent Group?

[rschmunk]

Having enabled my code to tell netCDF-Java to hook into the netCDF-C library, I suddenly find that a sample HDF5 dataset that I was able to open a week ago now fails with the following stacktrace. I have absolutely no idea what to think of this. Is this evidence of a bug somewhere? Do I need to revise my fairly simple nj22config.xml file? Any ideas? The file in question an OPERA L2 data product from JPL.

java.lang.IllegalStateException: EnumTypedef 'null' does not exist in a parent Group
	at ucar.nc2.Variable.<init>(Variable.java:1893)
	at ucar.nc2.dataset.VariableDS.<init>(VariableDS.java:918)
	at ucar.nc2.dataset.VariableDS$Builder.build(VariableDS.java:1175)
	at ucar.nc2.dataset.VariableDS$Builder.build(VariableDS.java:1026)
	at ucar.nc2.Group.<init>(Group.java:924)
	at ucar.nc2.Group.<init>(Group.java:44)
	at ucar.nc2.Group$Builder.build(Group.java:1410)
	at ucar.nc2.Group.lambda$new$2(Group.java:915)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at ucar.nc2.Group.<init>(Group.java:915)
	at ucar.nc2.Group.<init>(Group.java:44)
	at ucar.nc2.Group$Builder.build(Group.java:1410)
	at ucar.nc2.Group.lambda$new$2(Group.java:915)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at ucar.nc2.Group.<init>(Group.java:915)
	at ucar.nc2.Group.<init>(Group.java:44)
	at ucar.nc2.Group$Builder.build(Group.java:1410)
	at ucar.nc2.Group.lambda$new$2(Group.java:915)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at ucar.nc2.Group.<init>(Group.java:915)
	at ucar.nc2.Group.<init>(Group.java:44)
	at ucar.nc2.Group$Builder.build(Group.java:1410)
	at ucar.nc2.Group$Builder.build(Group.java:1402)
	at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:2576)
	at ucar.nc2.dataset.NetcdfDataset.<init>(NetcdfDataset.java:1606)
	at ucar.nc2.dataset.NetcdfDataset.<init>(NetcdfDataset.java:88)
	at ucar.nc2.dataset.NetcdfDataset$Builder.build(NetcdfDataset.java:1807)
	at ucar.nc2.dataset.NetcdfDatasets.enhance(NetcdfDatasets.java:205)
	at gov.nasa.giss.data.nc.NcDataset.enhanceNjfile(NcDataset.java:632)

[lesserwhirls]

Could you point me to a specific sample file that fails, as well as the version of the C library you are using in netCDF-Java?

[rschmunk]

The file that's triggering the exception is on Dropbox. Unfortunately, it's about 250 MB. I'm not quite sure if this an actual in-distribution product or something one of the developers was experimenting with. It's something that was originally sent my way a few weeks ago because of a separate issue in how Panoply was displaying metadata.

The problem is occurring on two Macs, one of which is using netCDF-C installed via MacPorts and the other via Homebrew. Both are apparently 4.9.3.

I'll try to extract some more logging diagnostics on Friday and see if I can track down what specific variable or whatever in the file is triggering the exception.

[lesserwhirls]

Perhaps an interesting datapoint. I can open and read the file using the netCDF-C library for reading in netCDF-Java no problem with:

• netCDF-C: 4.9.3
• HDF5: 1.14.6

and I can see some ENUM variables in the metadata/processing_information/parameters/ group. However, no ENUM variables appear when using:

• netCDF-C: 4.9.2
• HDF5: 1.14.4

I'll also note that the file does not contain the _NCProperties provenance attribute. I believe that means (can you keep me honest here, @WardF?) it was either generated with a version of netCDF-C prior to 4.4.1, or it was written with something that does not rely on the netCDF-C library at all, like h5netcdf, which can produce "invalid" netCDF-4 files (in which case, the _NCProperties is omitted).

[rschmunk]

When you succeed with the 4.9.3/1.14.6 combo, what OS are you using?

[lesserwhirls]

MacOS aarch64 (M2)

[rschmunk]

I'm currently using an M1 MBP with MacPorts, but have also encountered the problem on an M2 Mac Studio with Homebrew.

[rschmunk]

FWIW: After adding some logging to Variable(Builder,Group), I found that the exception occurred when the code got to the enum quality_assurance/rfi_information/VV/is_rfi_info_available.

Prior to getting to that point, the enums in metadata/processing_information/parameters were successfully set up.

[rschmunk]

I just built a copy of toolsUI and the results were… mixed?

toolsUI did open the problem file, but the UI was very flaky. So it was hard/difficult/impossible to confirm what you were seeing in your screenshot.

Then I noticed that my invocation on the CL to start tools UI said

>java -jar toolsUI-5.9.1-SNAPSHOT.jar

current L&F=Mac OS X
[01:35:20.117 WARN  ucar.nc2.ffi.netcdf.NetcdfClibrary] Nc4Iosp: NetCDF-4 C library not present (jna_path='null', libname='netcdf').

which is to say, it didn't use the netCDF-C library location specified in my local file at all. When working my way through #1161 and #1473 a few days ago, I found I had to specify the exact file name for libname rather than expect the system to prepend a 'lib' and append a '.dylib' or '.so' or dot-whatever.

[rschmunk]

Okay, figured out that when using toolsUI, I needed to provide a CL option pointing to my nj22Config.xml file. Having done that, it accesses the local libnetcdf.dylib file and successfully opens the problematic OPERA dataset. I get the same UNKNOWN description of the problem enum as in your screenshot.

(I note that if one does not specify the location of the nj22Config.xml file via CL option, that ToolsUI.java looks for it within the .unidata subdirectory of the current working directory, but it doesn't do anything useful with it if found.)

So, now to figure out what the difference is between how my code tries to acquire the dataset and how toolsUI does so.

[lesserwhirls]

Glad you found that commandline option for specifying the runtime config file. The fact that toolsUI looks for the file in a few known locations but does not do anything with it sounds like a bug to me.

Doing an 'ncdump -h' on the file, I am noticing that the problem enum is described as

/metadata/processing_information/parameters/_AnonymousEnum1 is_rfi_info_available ;

so maybe there's a weird disconnect in backtracking to the previously seem group to get the enum definition? Or not going back at all?

@ethanrd - this sounds suspiciously like the behavior with groups and dimensions that you have been describing.

[rschmunk]

"The fact that toolsUI looks for the file in a few known locations but does not do anything with it sounds like a bug to me." I did wonder about that. It's just a matter of adding one line of code to get that bit to use the file if present.

Also wondered why that bit of code looks for the ".unidata" directory subordinate to the working directory rather than start from System.getProperty ("user.home").

[ethanrd]

@ethanrd - this sounds suspiciously like the behavior with groups and dimensions that you have been describing.

In case it's useful, I recently started a netCDF-C discussion (#3157) on dimension scope in a group hierarchy. It references a PR on user-defined types as well.