Writing a NaN to an integer variable gives inconsistent results upon subsequent read

[ellenjohnson]

Hi netCDF friends! Question regarding netCDF-C behavior when writing NaNs to a integer variable. I tested this with netCDF 4.8.1 on my Linux machine. I see similar results on Windows.

In MATLAB and in my netCDF-C repro program (attached), if I create a netCDF integer variable and write a NaN to it, I see inconsistent results when reading the value back which I suspect is caused by undefined behavior in C library when converting a floating point type to integer type. This stackoverflow post is similar to what I’m seeing: https://stackoverflow.com/questions/3986795/what-is-the-result-of-casting-float-inf-inf-and-nan-to-integer-in-c.

Note MATLAB NaN is the IEEE arithmetic representation for NaN as a double scalar value. Also, MATLAB stores all numeric values as double-precision floating point values.

So for example, if I create an int16 variable, write a NaN, then read it back, I get zero. Because we're passing in NaN (a double) to ncwrite, we end up calling nc_put_var_double() and passing it the C value -nan as a double. So when it gets into the netCDF library, and netCDF is trying to assign the NaN to an int16, I think the zero is the result of undefined behavior:
% write NaN to int16 variable
nccreate('ocean1.nc','temperature','Datatype','int16','Format','netcdf4');
ncwrite('ocean1.nc','temperature',NaN);
data = ncread('ocean1.nc','temperature')
data =

     int16

     0

Note that writing an int32 to a NaN gives a different, non-zero result:
% write NaN to int32 variable
nccreate('ocean2.nc','temperature','Datatype','int32','Format','netcdf4');
ncwrite('ocean2.nc','temperature',NaN);
data = ncread('ocean2.nc','temperature')
data =

     int32

     -2147483648

However, if I first cast the NaN to an int16, the internal are a little different - we end up calling nc_put_var_short (because we cast NaN to int16), and we pass in a value of zero (since in MATLAB, casting a NaN to an integer type always results in zero). Thus in this case, we're basically writing a zero and reading a zero, which makes sense.
% write NaN to int16 variable but cast the NaN to int16
nccreate('ocean3.nc','temperature','Datatype','int16','Format','netcdf4');
ncwrite('ocean3.nc','temperature',int16(NaN));
data = ncread('ocean3.nc','temperature')
data =

     int16

     0

Here’s a summary of the behavior when writing NaN for all the integer types in MATLAB:
For uint8, uint16, uint32, int8, and int16, writing NaN writes zero.
For int32, writing NaN writes -2147483648.
For uint64 and int64, writing NaN writes -9223372036854775808.

My “nanTest.c” program (attached as text file) tests this with int16, int32, and int64 and shows the same behavior.

Question: Is this expected behavior in the netCDF-C library? Can you consider making the behavior consistent, such as returning zeros for all cases of writing a NaN to a variable defined as an integer, or giving a warning?

nanTest.txt

[DennisHeimbigner]

I looked at that stack overflow url. The claim is that all conversions of Nan
(and Inf) to integer are undefined. I do not think that netcdf-c does anything
so I assume this is done by the hardware or the C runtime library.

[ellenjohnson]

Thanks @DennisHeimbigner, that's what I thought - netCDF isn't doing anything special, it's just relying on the C behavior when converting NaNs to integer types.

My question at this point is whether you and your netCDF colleagues are going to leave the netCDF code as-is and close this out saying it's expected behavior, or if you consider it a bug and will modify the code to guard against cases where there is a NaN-to-integer conversion (and handle it in some way like always returning zero, or giving a warning, etc.).

Note this wasn't reported to us by a MATLAB customer - we found it during internal testing.

[WardF]

Hi @ellenjohnson. This is a good question; we've certainly encountered other issues where undefined behavior was encountered. It seems like we may need to raise an 'undefined behavior' warning. Our general guide here is that it's better to raise a warning/error and let the user change the data, rather than quietly changing the data being written.

We'll need to figure out exactly how we want to address this, but 'the data being written/read is an unexpected value' is exactly the situation we want to avoid.

[ellenjohnson]

Thanks @DennisHeimbigner and @WardF for the quick replies! I really appreciate it. I'll add a comment to the bug we're tracking on our side that we're waiting for a fix from the netCDF developers.

[DennisHeimbigner]

Fixing this in the netcdf-c library may be costly. In effect,
we have to examine each value during conversion from float/double
to int to see if it is a NaN value. BTW I wonder how HDF5 addresses this?

[Dave-Allured]

My opinion is that the netcdf library should prioritize speed over numerical consistency. I think this is what most users want and expect, most of the time. The library uses the most efficient available hardware and software instructions for type conversion.

Therefore, result values can not be guaranteed for numerical exceptions such as NaN conversions, unless they are somehow guaranteed by hardware, C runtime, and IEEE numerical standards.

Applications that need full consistency should simply read or write only the original data type, and perform their own controlled type conversion.