Conformance requirement on actual_range

[pvanlaake]

Topic for discussion

In the conventions, section 2.5.1, it is stated:

This convention defines a two-element vector attribute actual_range for variables containing numeric data. If the variable is packed using the scale_factor and add_offset attributes (see Section 8.1, "Packed Data"), the elements of the actual_range should have the type intended for the unpacked data. The elements of actual_range must be exactly equal to the minimum and the maximum data values which occur in the variable (when unpacked if packing is used), and both must be within the valid_range if specified.

The conformance requirements for the same state:

The actual_range attribute must be of the same type as its associated variable unless there is a scale_factor and/or add_offset attribute, in which case it must be of the same type as those attributes.

This is obviously a mistake in the conformance document.

[Armin-RS]

But doesn't this in the end equals to the same statement ?
Since Section 8.1 (https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#packed-data) states:
"When packed data is written, the scale_factor and add_offset attributes must be of the same type as the unpacked data,"

[pvanlaake]

The text in 8.1 is correct. The packed data is typically of NC_SHORT type, while the unpacked data is NC_FLOAT or NC_DOUBLE. The scale_factor and add_offset attributes must be of the type that the data variable is to be unpacked to. On the one hand there is no other way (multiplying a NC_SHORT with another NC_SHORT produces an integer type), and furthermore it flags the data type of the unpacked data which is not otherwise recorded in the netCDF file.

[davidhassell]

Something's not right in the 2.5.1: The text says that actual_range has values that relate to the unpacked data and must be within a valid_range. However, valid_range relates to unpacked values, so we have a contradiction, it seems.

Either actual_range does not need to lie within a valid_range, or it applies to packed values - which one is right?

[pvanlaake]

valid_range is expressed in the same data type as the variable if the data is not packed, in the data type of the packed data otherwise. The actual_range values must lie within the valid_range range, after unpacking.

[pvanlaake]

Considering that actual_range values are expressed in terms of the unpacked data values, perhaps the para in 2.5.1 can be rephrased as follows:

This convention defines a two-element vector attribute actual_range for data variables containing numeric data. The elements of actual_range must be of the same data type as the data variable and exactly equal to the minimum and the maximum data values which occur in the data variable, and both must be within the valid_range (when unpacked if packing is used) if specified. If the data is all missing or invalid, the actual_range attribute cannot be used.

This still leaves open the inconsistency with the conformance requirements.

[davidhassell]

Hi Patrick, I think that's nearly it, but when data are packed then the actual_range should have "the type intended for the unpacked data".

I'm thinking that the reference to valid_range is a distraction. valid_range is just one means of defining missing values (what about valid_min and valid_max?), so really we just need to say that actual_range needs to be the max/min of the (unpacked) mon-missing values. So ...

This convention defines a two-element vector attribute actual_range for variables containing numeric data. If the variable is packed using the scale_factor and add_offset attributes (see Section 8.1, "Packed Data") then the elements of the actual_range have the type intended for the unpacked data and must be exactly equal to the minimum and the maximum of the non-missing data values which occur in the unpacked variable. If the variable is not packed then the elements of the actual_range have the same type as the data and must be exactly equal the minimum and the maximum of the non-missing data values.

With this, the conformance would some changes, but I don't have time to suggest those right now :)

[pvanlaake]

Looks good to me

[ethanrd]

Hi all - The phrase "must be exactly equal" caught my eye. Given there's floating point math involved, it seems potentially problematic.

[pvanlaake]

I don't think this is a major issue. The actual_range values are copied from the data values through min \ max operators, not independently calculated, so a binary equivalence may be assumed. Is it for the conventions to flag such operational details that may be different on the various platforms?

[ethanrd]

For the situation where the data is packed, the data user will be calculating the data values independently and it seems unlikely the min/max of those values and the actual_range values will have binary equivalence.

Yes, I would say including details that help support use across different platforms is a key goal of the conventions.

[pvanlaake]

Ouch, you got me there! I surrender.

[larsbarring]

"...every time floating point values are compared for equality a fairy dies..."

[sethmcg]

Does anyone know what the use case is for actual_range is? And is it compelling?

I ask because one way to resolve the issue would be to deprecate actual_range and remove it from the standard.

That's an extreme measure, but I think it's worth considering, because there's a really big problem with actual_range: it's incredibly fragile. The instant that you make any changes at all to the variable -- particularly subsetting, which is a very common operation -- you can no longer rely on the values in actual_range being valid.

Because they are supposed to be "exactly equal to the minimum and the maximum of the non-missing data values," any time that there's a change to the data, the min and max could change, so you should be recomputing them and updating the attribute. But nobody does that. In particular, nobody does it because (unless I've missed something) none of the major software packages that people use to process netcdf files (NCO, CDO, xarray) support automatic updating of the attribute, and nobody is going to do that update by hand every time that they touch a file.

Which means that, of all the files out there in the world that have an actual_range attribute, there are probably an enormous number of them where the attribute is possibly non-compliant. Sure, untouched files straight from the data provider are probably fine, but for anything else, the odds are good that something has happened to the file that makes the attribute untrustworthy.

Alternately, if it useful to keep around, we could resolve both issues by loosening the requirements, and defining it simply as a range that's appropriate to use for visualization of the data (or whatever other use case may apply), without specifying its type or value.

[davidhassell]

The instant that you make any changes at all to the variable -- particularly subsetting, which is a very common operation -- you can no longer rely on the values in actual_range being valid.

[pvanlaake]

Hi Seth,

I fully agree with your description of the Wild West out there but not so much with the proposed solution. There are many attributes that require updating after any kind of processing, from the rather humble (and often mis-used) history to more critical attributes such as cell_methods and coordinates. And then there is a whole slew of attributes that should be removed or renamed, like source and institution for CMIP data, or binning attributes for MODIS L2b data after de-gathering (?) to a regular grid.

CF is all about attributes and if people and software are serious about their data analysis, they had better improve their operations.

[davidhassell]

(sent too early)

The instant that you make any changes at all to the variable -- particularly subsetting, which is a very common operation -- you can no longer rely on the values in actual_range being valid.

Indeed. This potential for inconsistency data is not good.

I have encountered its use in the wild just the once in the Climate hydrology and ecology research support system meteorology dataset for Great Britain (1961-2019) [CHESS-met]. As to why this dataset has an actual_range, I don't know. It's purpose is to provide variables required to run the JULES land surface model, a use which doesn't seem to me to need it (although I'm not an expert :)). Could this perhaps be a case of "it was there, so I used it?" ...

[ChrisBarker-NOAA]

There are many attributes that require updating after any kind of processing

well, true, but this is not the same as most other cases, because:

they are supposed to be "exactly equal to the minimum and the maximum of the non-missing data values,"

Which means they are a calculated value, from the data itself. It's a good practice in software to NOT store explicitly a calculated value separately, because it's too easy for it to get out of sync.

So why does this attribute exist?

• I suspect it's an optimization (which is when you would make an exception to the "good practice" I mention above -- there may be applications that need to know the range of values to expect, and it could be way to expensive to load the entire variable and calculate it.

But in that case, to @sethmcg 's point, perhaps we could relax the constraint to:

It is recommended that the range be:

exactly equal to the minimum and the maximum of the non-missing data values

And it is required that the range:

encompass the full range of the the non-missing data values.

All the being said, the language should still be correct and clear with regard to the data types :-)

[sethmcg]

That's a solution that makes a lot of sense to me!

[pvanlaake]

But if the actual_range is not equal to the actual range of the data values because it is only recommended, then the attribute loses the guarantee of having the information that it currently has. The user can always delete it; if it is there, then its name should be indicative of what it represents.

[ChrisBarker-NOAA]

actual_range is not equal to the actual range of the data values

well, yeah -- "actual" does have a meaning .... however, it is used as an addendum to:

valid_range - and valid_range is, IIUC, about what "could" be stored in the data -- e.g. what's not a missing value, etc.

So actual_range is about what values to expect in this particular data set. So a bounds on that range could be almost as useful as an exactly actual range.

so does "actual" mean : what's in this exact variable, right now? (the current definition) or what might actually be stored in this variable from this source.

In a way, the second definition seem more useful to me:

When writing code to process the data -- you need to decide what data types to use, what values to use as ranges fro plotting, etc. And for that, you want to know what the range is of values one might expect for this variable in general, not necessarily what the range is for this particular subset.

I guess what I'm getting at is that it is very common for the same type( source, etc) oof data to be presented in multiple files -- spanning time periods, or what have you. IN that case, it seems lot less useful for each file to have an exactly computer actual_range for that one subset of the data than for all files of the same "thing" to have an actual_range for that whole dataset.

Does anyone remember what the original use case was for actual_range ? Why it's there at all?

[ethanrd]

I'm pretty sure the original use case was about hints for visualization. But that's just from memory, I haven't searched for the actual discussion.

[sethmcg]

I agree with all of this.

I found the original trac ticket, and it is indeed intended to support visualization and data mining:
https://cfconventions.org/Data/Trac-tickets/31.html

I missed the discussion of the Trac Ticket at the time, but I flagged this as a problem in a discussion thread on the mailing list a decade ago:
https://cfconventions.org/mailing-list-archive/Data/6208.html

[ChrisBarker-NOAA]

the original use case was about hints for visualization

Thanks!

Then my point stands -- the

"range of values you might get in this variable"

rather than

"the range of values this particular variable contains"

is far more useful.

Of course, this would a potentially breaking change :-(

-CHB

[sethmcg]

Given how unreliable the values are in practice, I can't see how any software that relied upon the attribute wouldn't already be broken.

[pvanlaake]

I note, with certain amusement, that a rather mundane flagging of an inconsistency between two documents is evolving into a discussion on what the conventions should or should not include, which involves digging up 10-yo trac tickets. Great research work, Seth! If there were such a position, I would move to appoint you Chief Custodian of the Conventions.

That said, two things.

1. There is still the inconsistency that needs to be resolved. (Following on Seth's argumentation, I could also advocate for deleting the conformance document altogether because no one has apparently ever applied it on a data set with an actual_range attribute attached to a packed data variable; or the attribute was always in the packed data type which is incongruent with the conventions text.)

2. I am rather uncomfortable with yielding to the flaws of badly written software. That is a sliding scale with no end to it. The conventions are 100% about attributes describing what the data in a netCDF file represent and how to interpret them. One cannot use any CF-conformant netCDF file without looking at the attributes. In true Marxist style, I am all for Verelendung: bad or outdated software will be updated or replaced because the users revolt against the injustice of being stuck with software running CF-1.6 while the data sets have moved on to CF-1.12. So if the values of the actual_range attribute are not kept up-to-date, then that is not something that the conventions need to be concerned with. It is not a required attribute, it can be dropped (or not inspected) at no consequence other than a potential loss of efficiency.

Which brings me to, for good measure, 3. The attribute actual_range, in its strict sense of having values that are numerically equivalent (@ethanrd Ethan, can I say it like this?) to what the data variable or coordinate variable or bounds variable contains, is very useful in the context of data discovery. On a CV it informs you of the spatial or temporal domain of the data variable before you have to read in the CV data. If I am doing research on heat stress in the current climatological normal period, I can check actual_range on my tmax data variable and its time CV and forget about reading the entire 2GB array to cheaply calculate the maximum of the maximum temperature on-the-fly (actually not so cheap with the I/O involved, especially if my netCDF file is on a THREDDS server halfway around the globe) if tmax$actual_range[2] < 298K or time$actual_range outside of 1991-2020.

So I commiserate with Seth and Chris on the plight of having to work with bad software and sloppy data scientists, but I suggest that you disregard the actual_range attribute rather than throw it out or water down its definition and leave the disciplined ones amongst us without a reliable discovery tool.

[larsbarring]

Basically, I agree with all of the above. But a different question is what does the actual_range refer to? Is it per file or is it per dataset? I imagine that back in the days when it was introduced it was the former. And I believe that this would be what is checked by a cf-checker, although as Paul point out a cf-checker probably only checks that the metadata as such is well behaved wrt CF, not that the values are correct in relation to the actual data (I might be wrong here...). Anyway, is this really useful today when CMIP, CORDEX and many other data sources split a specific variable from a specific model run (etc.), into several files covering shorter time periods, i.e. making up a dataset. ? Personally I do not think so.

Instead, when my single variable now is split up into several files it seems reasonable to make the actual_range represent the actual range across those files. But, I also have the same variable from other model runs, should the actual_run span the range of those to? Maybe/probably not?

On the other hand, as CMIP and CORDEX files are split into consistent time periods, wouldn't it be reasonable to have one actual_range spanning all simulations (sharing the same emission scenario, etc.) for each time period?

I imagine that these questions may have very clear answers --- that may be very different depending on who you ask... I.e. no clear answer....

One thing is that we of course should clear up the inconsistency that Paul initially pointed at. Another thing is that now, when the cat is out of the bag, we should as well clarify what actual_rangeis meant to represent and convey.

[pvanlaake]

Hi Lars, Patrick here {;-)

As per Appendix A, actual_range can be used for data variables, coordinate variables and boundary variables (measures variables are conspicuously missing here...). And that makes sense: they are all variables that contain a quantum of numeric data that can usefully be summarised into min/max values.

Wrt a single data set spanning multiple netCDF files, actual_range can still be useful at the file level (basically the heat stress example I used above), but it could also apply to the larger data set. @davidhassell David's CFA work on CFA (issue #508) would be covering that scenario.

[pvanlaake]

Coming back to the initial issue: there is an obvious discrepancy between the conventions and the conformance document.

On 24 March @davidhassell proposed the following text for the conventions, section 2.5.1:

This convention defines a two-element vector attribute actual_range for variables containing numeric data. If the variable is packed using the scale_factor and add_offset attributes (see Section 8.1, "Packed Data") then the elements of the actual_range have the type intended for the unpacked data and must be exactly equal to the minimum and the maximum of the non-missing data values which occur in the unpacked variable. If the variable is not packed then the elements of the actual_range have the same type as the data and must be exactly equal the minimum and the maximum of the non-missing data values.

"Variables containing numeric data" includes more than the currently allowed (Appendix A) coordinate variables, data variables and boundary variables: cell measure variables, auxiliary lat-long grids, geometry containers all contain numeric data as well. I would therefore suggest to qualify that phrase to be more specific.

In terms of changes to the conformance document, I would propose the following:

• The actual_range attribute must be of the same type as its associated variable unless there is a scale_factor and/or add_offset attribute, in which case it must be of the same type as those attributes.

• The actual_range attribute must have two elements, of which the first exactly equals the minimum non-missing valid value occurring in the associated variable after any scale_factor and add_offset are applied, and the second exactly equals the maximum valid value in the same way.

• There must not be an actual_range attribute if all the data values of the associated variable equal the missing_value.

• If both the actual_range and either of valid_range or valid_min/valid_max are specified, the values of the actual_range must be fall in the range of valid values.