UTC & leap seconds (again! again!!!)

[pvanlaake]

Topic for discussion

A Wise Man once said “If you think you know all about Time, you haven’t thought long enough”. Hence I beg your forgiveness for bringing up the UTC / leap seconds issue after the extended discussion (#304) and the recent inclusion in CF 1.12 (not to mention my missing the PR).

First of all, great job on the revamped section 4.4 and happy to see some new “calendars” added.

That immediately brings me to the first issue: Neither UTC nor TAI are calendars, they are clocks. So rather than than saying for utc “A Gregorian calendar with leap seconds as prescribed by UTC”, can this be changed to “Coordinated Universal Time (UTC) which includes leap seconds and where dates are represented using the Gregorian calendar”? And likewise for tai.

In the description of both calendars I would add that time zone offsets are not allowed and that a time zone indication, when given in a datetime, must be 0.

UTC

More concerning, from my perspective, is the definition of utc. As defined, it starts in 1958, but UTC did not exist until 1960 and was only officially adopted in 1963 (it was based on that same epoch as TAI, which is 1958-01-01 00:00:00 UT). The issue of leap seconds in UTC is not meaningfully addressed in the new text. Of course, there is section 4.4.3, but that really only deals with the consequences of having to consider leap seconds. I have seen the arguments being made in issue #542 but it feels unbalanced to have the detailed explanation of figure 4.1 but no basic information on leap seconds (such as where to find an authoritative list of the 27 leap seconds since 1972) or how it differs from the other calendars.

Further, prior to 1972 there were no leap seconds. The divergence of nearly 10 seconds between UTC and UT that accumulated between 1958-01-01 and 1971-12-31 was applied through three mechanisms: (1) a shifting in the length of the UTC second, at various rates over the period 1960-1971; (2) frequent small increments of multiples of 50ms; and finally (3) an anomalous jump of 0.107758 SI seconds to make the difference exactly 10 seconds at the onset of 1972-01-01. In other words, prior to 1972, the second used by UTC is not equal to the SI second. The introduction of the leap second was made exactly to align the UTC second with the SI second and to make time shifts to align the UTC clock with the celestial reality less burdensome (all of this before the advent of modern computer and communication infrastructure). You may find all the details here.

It is then a bit of a mystery to me why CF would want to dive head-first into this murky 1960-1971 business that many Wise Men have tried to obliterate through the introduction of the leap second. Is there a known use case for data sets using UTC that span the early years of that clock?

I would propose to side-step all of this complication by changing to definition of utc to begin at 1972-01-01 00:00:00, being the instant when UTC started using the SI second and leap seconds were introduced, with no time coordinates prior to that instant allowed.

units_metadata

I do not understand the purpose of this argument. The utc calendar was introduced to enable data producers that need second accuracy to record their observations correctly. When leap seconds are of concern, then why would the data producer use the standard or proleptic_gregorian calendar? Having this need for accuracy but not knowing whether or not the data collection equipment properly records and processes leap seconds does not seem like a real-world combination to me.

Why is there reference to the julian calendar here? Leap seconds are defined in the context of UTC and that uses the Gregorian calendar exclusively.

Finally, why an attribute units_metadata with a compound structure? Wouldn’t a simple attribute leap_seconds suffice?

[ChrisBarker-NOAA]

I'm all for making the docs more precise and clear, which I think is what much of this is about.

But is there any of this that might cause actual problems with using the new calendar?

To a few points:

“A Gregorian calendar with leap seconds as prescribed by UTC”, can this be changed to “Coordinated Universal Time (UTC) which includes leap seconds and where dates are represented using the Gregorian calendar”? And likewise for tai.

Sure -- that seems a bit more correct.

In the description of both calendars I would add that time zone offsets are not allowed and that a time zone indication, when given in a datetime, must be 0.

I don't think that's wise. As you point out about above, UTC is not a calendar per se -- a calendar specified days, and the time within in that is kind of a separate thing. However, for CF, they are a single concept a "calendar" is the full spec for how to interpret (and produce) datetime strings. So the difference between, say the standard calendar and utc is leap seconds, that's it -- there is just as much validity to specifying a timezone offset with the UTC calendar as any other. In that case, the time is not UTC, but it is relative to UTC -- so that's fine.

([UTC] was based on that same epoch as TAI, which is 1958-01-01 00:00:00 UT).

exactly, so it's valid from that date, yes? what's the problem here?

The issue of leap seconds in UTC is not meaningfully addressed in the new text. ... but no basic information on leap seconds (such as where to find an authoritative list of the 27 leap seconds since 1972) or how it differs from the other calendars.

I think that was pretty deliberate -- we try not to re-write specifications that others are responsible for. That being said, a bit more detail, and in particular a link to an external source with the full explanation would be great -- can you propose one?

It is then a bit of a mystery to me why CF would want to dive head-first into this murky 1960-1971 business that many Wise Men have tried to obliterate through the introduction of the leap second.

I'm guessing that was out of ignorance ....

I would propose to side-step all of this complication by changing to definition of utc to begin at 1972-01-01 00:00:00, being the instant when UTC started using the SI second and leap seconds were introduced, with no time coordinates prior to that instant allowed.

That sounds good to me -- but I'll let others that may know better weight in.

The goal here is that people can use a UTC timestamp, convert that to a CF-style "time-since" form, (and vice versa) and clearly define what that means. If that's not possible pre-1972, then yes, we shouldn't allow it.

When leap seconds are of concern, then why would the data producer use the standard or proleptic_gregorian calendar? Having this need for accuracy but not knowing whether or not the data collection equipment properly records and processes leap seconds does not seem like a real-world combination to me.

The issue is not that data producers don't know whether leap seconds are included in their timestamps -- the problem is that most of the software out there is not leap-second aware. So if a producer knows they have proper UTC timestamps, but their software (and that of their users) doesn't support leap seconds, then they need to use the standard calendar.

Practicality beats purity, and all that.

Why is there reference to the julian calendar here? Leap seconds are defined in the context of UTC and that uses the Gregorian calendar exclusively.

agreed -- we should remove that.

Finally, why an attribute units_metadata with a compound structure? Wouldn’t a simple attribute leap_seconds suffice?

because units_metadata was already introduced for temperature -- so keeping things a bit more consistent.

However, now that you mention is, I see in section 3.1.2:

"A variable must not have a units_metadata attribute if it has no units attribute or if its units do not involve a temperature unit."

That needs an update.

[pvanlaake]

Hi Chris, thanks for your reply. A few responses to your comments:

there is just as much validity to specifying a timezone offset with the UTC calendar as any other

Here I have to disagree. The U stands for Universal, UTC time is the same all over the world at a single instant, irrespective of where you are. Bangkok, London and Denver all have the same UTC time at the same instant.

UTC is managed by BIPM so I would suggest a reference to https://www.bipm.org/en/work-programme/time as the authority on UTC and TAI would suffice, perhaps with the addition that this is technical background information. On that page is a link to "time department database" where lots of reference material is available. Under "resources", then "time scales" are a number of files, including a list of the 27 leap seconds that have been applied. Relationship between TAI and UTC is instructive in the context of this discussion. The first few lines read:

 RELATIONSHIP BETWEEN TAI AND UTC
 ------------------------------------------------------------------------------- 
 Limits of validity(at 0h UTC)       TAI - UTC  
 
 1961  Jan.  1 - 1961  Aug.  1     1.422 818 0s + (MJD - 37 300) x 0.001 296s
       Aug.  1 - 1962  Jan.  1     1.372 818 0s +        ""
 1962  Jan.  1 - 1963  Nov.  1     1.845 858 0s + (MJD - 37 665) x 0.001 123 2s
 1963  Nov.  1 - 1964  Jan.  1     1.945 858 0s +        ""
 1964  Jan.  1 -       April 1     3.240 130 0s + (MJD - 38 761) x 0.001 296s
       April 1 -       Sept. 1     3.340 130 0s +        ""
       Sept. 1 - 1965  Jan.  1     3.440 130 0s +        ""
 1965  Jan.  1 -       March 1     3.540 130 0s +        ""
       March 1 -       Jul.  1     3.640 130 0s +        ""
       Jul.  1 -       Sept. 1     3.740 130 0s +        ""
       Sept. 1 - 1966  Jan.  1     3.840 130 0s +        ""
 1966  Jan.  1 - 1968  Feb.  1     4.313 170 0s + (MJD - 39 126) x 0.002 592s
 1968  Feb.  1 - 1972  Jan.  1     4.213 170 0s +        ""
 1972  Jan.  1 -       Jul.  1    10s            
       Jul.  1 - 1973  Jan.  1    11s
 1973  Jan.  1 - 1974  Jan.  1    12s	
...

So in 1961-01-01 the difference between TAI and UTC was 1.422818 SI seconds and increasing at a rate of 0.001296 SI seconds per day. That rate was adjusted three more times before 1972 and in addition there were steps of multiples of 0.05 SI seconds. Given that there were still only 86400 seconds in a day, the UTC second did not align with the SI second.

([UTC] was based on that same epoch as TAI, which is 1958-01-01 00:00:00 UT).

exactly, so it's valid from that date, yes? what's the problem here?

The values may have been coincident on that day, but they immediately started drifting apart.

Best, Patrick

[JonathanGregory]

Dear Patrick

Thanks for your comments and the discussion.

• You suggest we should change the first sentence in Sect 4.4.2 defining the utc calendar from "A Gregorian calendar with leap seconds as prescribed by UTC" to "UTC, which includes leap seconds and where dates are represented using the Gregorian calendar." (We don't need to spell out what "UTC" stands for in this sentence, because it is well-known and was anyway already defined earlier in Sect 4.4.2.) Actually, I don't see a difference of meaning between these two versions. Maybe the word "prescribed" is unclear? We could change "with leap seconds as prescribed by UTC" to "with the leap seconds that are included in UTC." Would that be better?

• I agree that it would be helpful to provide a link to Relationship between TAI and UTC. Thanks for suggesting it. We could also link Leap_Second.dat. Both of these files are available for download from BIPM under the CC BY-ND 4.0 licence, I see.

• You're right that we elided the complicated history of adjustments to UTC betwen its invention and 1972. I hadn't grasped that they had "rubber seconds" (as explained in wikipedia more concisely than by BIPM). Because of this, I think you're right that we should prohibit utc before 1972. In CF, a second is always an SI second.

• Like Chris, I don't think it's problematic to use time-zones with utc and tai, because these are CF calendars, not time-zones. A CF calendar defines an ordered set of valid datetimes with integer seconds, as stated in Sect 1.3 and at the start of Sect 4.4.2. We could remind the reader of this in the paragraph defining utc. It's important to be aware of, because the leap seconds are defined in UTC +0 and therefore occur in different hours when there's a non-zero time-zone offset. We should mention this at the end of Sect 4.4.1 as well.

• Chris, thanks for noticing that in Sect 3.1.2 we need to change "A variable must not have a units_metadata attribute if it has no units attribute or if its units do not involve a temperature unit." We overlooked it there. We did remember to update the conformance document, I'm glad to say, which now reads, "A variable must not have a units_metadata attribute if it has no units attribute, or if its units do not involve a temperature unit or a reference time unit."

• We included the julian calendar for completeness in the treatment of leap seconds, because it was easy to do, although there's unlikely to be a CF use-case. The Orthodox churches and some Berber communities (see e.g. Amazigh New Year) use the Julian calendar. Since they don't differ from everyone else in clock time, that means their version of the Julian calendar (in the CF sense) must include UTC leap seconds. Postscript: Acknowledging that leap seconds are unlikely to be needed with the Julian calendar, we could make units_metadata="leap_seconds: none" the default (instead of unknown) for calendar="julian". Then we need not recommend including units_metadata for that calendar. Do you think that would make more sense?

Best wishes and happy New Year (whenever it exactly occurs for you!)

Jonathan

[pvanlaake]

Dear Jonathan,

Happy New Year to you and all other collaborators on the Gregorian calendar! (For those of you on the Julian calendar, do not hesitate to fault me if I forget about your New Year in 9 days' time.)

I remain rather uncomfortable with some of the arguments made in previous comments. As I understand these arguments, they are based more on ease-of-use and every-day practice rather than standards and principles. UTC is one thing and one thing only. Time zones are referenced to UTC so to call a calendar utc and then allow, say, -6 to indicate that the time is actually local time 6 hours behind UTC comes across as a CF-specific quirk that will confuse people more than that it will help them. The fact that in Boulder, CO, there may have occurred an instant of time with the datetime 2015-06-30 17:59:60 -6 doesn't turn Mountain Daylight Time into UTC, so why create that impression through the calendar utc? (This example is not incidental. The conventions document has used an example from the UDUNITS documentation since at least version 1.7 which refers to Mountain Daylight Time, as observed in Boulder, CO, where, probably totally by coincidence, UCAR is located. Perhaps tellingly, the reference to MDT has disappeared in the most recent conventions text. So the explicit reference to daylight savings time has disappeared (and a good thing too!) but now we have the leap second.)

The first use-case that was made for a calendar using UTC would very unlikely be using the local time as reported by a local computer synchronising with an NTP server, and which might result in a datetime as indicated above. And that is probably the case with the large majority of data producers using the CF conventions. In climate and weather forecasting science, practitioners tend to know their clocks, and how to convert from their local time as reported by their instruments and processing environment into UTC. So, again, why the extra whistles and bells?

It should also be considered that utc as currently defined may lead to a confusion between a data producer using utc + timezone, and a data user interpreting that as UTC. Buyer beware!

The CF calendar utc uses the UTC clock and the Gregorian calendar: two separate things. The fact that these two separate entities are conflated into a single CF calendar is the root of the issue we are discussing now.

Best,
Patrick

[JonathanGregory]

Dear Patrick

Thanks for your further thoughts. You comment that the arguments "are based more on ease-of-use and every-day practice rather than standards and principles." I think that's partly true. Perhaps you've seen that the preceding discussions on the issue of leap seconds were extraordinarily long and difficult. It seemed to me that these arguments were often about what things ought to mean "in principle". Because of their different understanding of these principles, the participants in the debate could not agree, and perhaps sometimes could not understand one another properly.

In this case, we could not achieve an consensus about all the principles involved. This is a very unusual outcome for CF discussions, but it's not a show-stopper, because CF is a convention i.e. an agreement, rule, practice or custom. Even though we couldn't agree on what things ought to mean, we did manage to agree some workable conventions. You're not alone in being uncomfortable about some of our choices. That's regrettable, but perhaps can't be helped.

I think the important question to consider is whether there are any hazards with these conventions that are likely to cause data-producers to write incorrect metadata, or data-users to misinterpret the metadata. If there are, we should either clarify the standards document, or consider changing the conventions. This is consistent with one of the principles for design (Sect 1.2):

The conventions should minimise the possibility for mistakes by data-writers and data-readers.

You wrote about some possible hazards before, and I have responded with possible remedies. Thanks for those points. Here, you're suggesting that time zones in utc are a hazard:

It should also be considered that utc as currently defined may lead to a confusion between a data producer using utc + timezone, and a data user interpreting that as UTC.

Timezones have always been allowed in CF datetimes. This convention was iinherited from COARDS, which contains the example in Mountain Daylight Time which you refer to. As COARDS says, that example comes from the UDUNITS documentation. Another of the CF principles is

Conventions are provided to allow data-producers to describe the data they wish to produce, rather than attempting to prescribe what data they should produce.

If you have a data-logger which records local time and is leap-second-aware, the datetimes would most conveniently be expressed in CF using the utc calendar with a timezone offset in the units.

In the example you give, you might have a time coordinate of 64801 with units="seconds since 2015-6-30 0:0:0 -6" and calendar="utc". The convention says, "Subtracting the time zone offset from a given datetime converts it to the equivalent datetime with zero time zone offset." Therefore seconds since 2015-6-30 0:0:0 -6 means the same as seconds since 2015-6-30 6:0:0 in UTC. 64801 seconds since 2015-6-30 6:0:0 is the leap second 2015-6-30 23:59:60 in UTC, hence 2015-6-30 17:59:60 -6.

Where would the confusion arise? Do you think the data-user might ignore the timezone in the units, simply because the calendar is called utc? In that case, maybe we should adopt a different name for this calendar? What about civil, as a suggestion? Wikipedia, Royal Museums Greenwich and Time and Date all use "civil time" to mean local time which differs by an offset from UTC.

Best wishes

Jonathan

[ChrisBarker-NOAA]

I’m on vacation, so brief, but a few notes. Here I have to disagree. The U stands for Universal, UTC time is the same

all over the world at a single instant, irrespective of where you are. Bangkok, London and Denver all have the same UTC time at the same instant.

Of course, but the offset is with respect to UTC, and it is provided as part of the string, so there is no confusion about what it means. And this isn’t a CF-specific oddity — it’s part of the ISO 8601. As you say, CF is indeed focused on “on ease-of-use and every-day practice rather than standards and principles.” “Practicality beats purity” as the Python folks say. Indeed, that’s why UTC has not been supported in CF until now— very little software (and none of the most commonly used tools) support it properly. Note: we should try to use “time zone offset”, or “offset”, rather than “time zone”. Everyone in this discussion knows the difference, but when written down, we should make sure to minimize confusion. One of the practical challenges is that a LOT of people assume UTC means “time at the prime meridian”, I.e. with no time zone offset applied, which it does, but it’s also more specific than that, but that distinction is lost on many people. That rate was adjusted three more times before 1972 …

The values may have been coincident on that day, but they immediately

started drifting apart. Ah, thanks - then I agree— 1972+ makes better sense then.

What about civil, as a suggestion?

I think that would be even more confusing. UTC +- offset is precise and clear and very commonly used.

…

-CHB

[chris-little]

@JonathanGregory I strongly recommend a Best Practice of data creators/servers using UTC timestamps without timezones, as recommended by various restrictive profiles of ISO8601, such as IETF RFC3339. Timezone notation should really be a function of client software that "knows where it is". Timezones, and daylight saving times, can change at the whim of politicians and others. Environmental Data really ought to have a global context. And of course ISO8601 has evolved, and people often reference deprecated versions as they may be freely available unlike the costly current standard.

To quote the W3C: "It is usually desirable to normalize timestamp values to UTC ..." and "When in doubt, use UTC for serializing, storing, and exchanging date and time values."

And too much backward compatability will in the long run build a legacy of technnical debt (aka "rat's nest" or "can of worms")

HTH, Chris

[pvanlaake]

I believe we may be converging on an outcome that is both practical and standards-based; maybe even both practical and pure?

UTC +- offset is precise and clear and very commonly used

Correct, but that is local time or the common understanding of civil time as Jonathan pointed out. So rather than call it utc, I second Jonathan's suggestion to call it the civil calendar (I had been toying with the same idea when walking my dog a few days ago but came up with the local calendar name).

Rather than tossing out UTC as a calendar option, I would propose the following:

• utc calendar, starting 1972-01-01 on the Gregorian calendar, at that instant 10 seconds behind TAI. Time zone must be 00:00 if given. This satisfies the requirements of many data producers publishing global data sets or satellite imagery or basically anything that is not specifically tailored to a small geography.
• An additional civil (or local) calendar, based on UTC and the Gregorian or Julian calendar but using a time zone offset (and possibly supporting daylight savings time (on the data producer side!) through the correct use of tz offsets). Personally I prefer the local name as that conveys the fact that the data may be tied to a specific geography and requiring adjustment to align with other time series.

Would something like that work?

As a final note on standards versus practice: The CF conventions are far more influential than you make it sound. The decisions reached on issues such as calendars reverberate throughout the larger CF community and materialize in data sets that hang around for decades. The netcdf library itself was modified in response to CF conventions. I'm just putting in my 2 cents to propose something that may still be valid and logical in a few decades` time.

[davidhassell]

Hi,

On the 1958-1972 issue: I think we should keep the 1958 epoch. This change in approach before/after 1972 was indeed considered during the creation of the new text, and the wording we used was deliberately chosen to allow for any frequency of leap second application (e.g. daily), and an individual adjustment being any amount of seconds (e.g. negative amounts and non-integer amounts). I think that what we have now is entirely consistent with the messy situation that occurred in 1958-1972, so there is no need to change the minimum valid time. If you can deal with leap seconds applied after 1972, then dealing with the situation before 1972 (should have such data) is logically no different. (Whilst it currently seems unlikely, the leap second situation could get more messy again in the future, in which case the conventions will still hold, and we'dbe glad we didn't arbitrarily remove part of the valid time period.)

On the timezone/calendar-name issue: To me, the use of a timezone offset is merely an encoding choice in the units attribute string value in the dataset, just as choosing "seconds since" or "days since" is your units string is. Therefore I see no reason disallow their use, and no reason to change the CF calendar name. Also, if we changed the CF utc calendar name, surely we'd have to change the tai calendar name, too? (which I don't think we want!).

Cheers,
David

[larsbarring]

This is more of a general comment than a response to any of the previous comments in this discussion.

I think that we have to careful about what words we use in relation to all this. For example what do we mean by the word "calendar" (as a general concept) and when do we mean calendar as a CF construct. Is "time"/"timestamp" different from "date"/"datestamp" and from "datetime"/"datetimestamp"? Are they just the same, even across diffent calendars? What is a "clock", is it just a physical mechanism providing ticks and tocks at regular intervals,? And then what is a "regular interval" if you have two clocks?

These are maybe silly questions. But at the same time, CF is trying to make several different "clocks" and "calendars" come together into one concept; the CF calendar construct as a representation of time coordinates. From my own perspective, I think that Chris Little's presentation⁽¹⁾ at the 2024 CF Conventions Workshop was very useful, and in this context particularly viewgraphs 17-19.

I think that CF would be well served by general discussion about all this. For example, there seems to be a strong push towards having the "datetimestamp" labelling (strings such as "2025-01-07 12:00:00Z") as a vehicle for unifying points in time across different calendars. This can be a very useful and pragmatic solution that is legitimate and accepted in many situations. But in other situations such a solution cannot be justified or even make it invalid, whereas other pragmatic solutions exist. Trying to fold any such pragmatic solution into the CF Conventions is doomed to cause trouble and limit its usefulness.

(1) Little, C. (2024, September 17). OGC Temporal Domain Working Group Activities. 2024 CF Workshop, Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden. https://doi.org/10.5281/zenodo.14194152.

[JonathanGregory]

Dear Lars

You're right that we have to be careful about words. In the world as a whole, the words you mention may be used in various ways. If there is inconsistency, we can't help it. Within the CF convention, however, we can be careful to use our words consistently. I hope that within the convention document we always mean "calendar" and "datetime" in the senses with which they're defined in Sect 1.3. On briefly looking, I believe that "date" separately always refers to just the year-month-day part of the datetime. A "time" coordinate represents a datetime, of course, not just a time of day; there's a lot in Sect 4.4 about it! In the standards document, "stamp" occurs only once, in Appendix B; we could delete it. The word "clock" occurs only once, in Sect 4.4.1; we could replace it. The phrase "regular interval" occurs only once, in App J, where it refers to distance rather than time.

Best wishes

Jonathan

[larsbarring]

Dear Jonathan,

I was more thinking of the different discussions, both here and in all the preceding discussions and issues on time/calendars/etc over the years. It is very easy to misunderstand or simply miss a point just because different understanding of concepts behind words. I am sure that we all have done it. Hence my general view is that CF has not done a very good job in what I mentioned as: "CF is trying to make several different "clocks" and "calendars" come together into one concept; the CF calendar construct as a representation of time coordinates." But this is a huge and difficult topic as we all are aware of. My point is that in order to make some progress in disentangling all this, it is useful to look at what others are doing. Hence my reference to Chris's presentation as food for thought.

[ChrisBarker-NOAA]

From my own perspective, I think that Chris Little's presentation⁽¹⁾ at

the 2024 CF Conventions Workshop was very useful, and in this context particularly viewgraphs 17-19.

I think that CF would be well served by general discussion about all this.

Is there a freely available summary of the new ISO 34000 ?

https://www.iso.org/standard/77019.html

Or do any of you have access to a copy? It seems that could be helpful to reference.

…

-CHB

[ChrisBarker-NOAA]

If you can deal with leap seconds applied after 1972, then dealing with the situation before 1972 (should have such data) is logically no

different. IIUC ( and I may not) it IS logically different — that is, rather than applying a “leap second” integer or not, at particular times, there was a slight adjustment to the size of every second in a range (did someone call them rubber seconds?). Which is a different approach, and would require different code to handle. @pvanlaake: could you please confirm? My concern about the addition of a UTC Calendar is that it is an attractive nuisance -- a lot of people *think* they are using UTC, when they are not (quite) -- and the software they are using may not support it properly. Allowing pre-1972 may make it even more likely for not-really-UTC data to be presented as UTC.

[davidhassell]

Ah, interesting. I though the changes were applied daily, rather than every second. Clarification would be very useful!

However, applying an increment annually, daily, hourly, even every second, is logically the same, surely?

[pvanlaake]

As I mentioned in my initial post in this thread, the second used by UTC in the period 1961 - 1972 is not the SI second. The table in my reply to Chris shows the rate at which the UTC second differs from the SI second used by TAI, expressed as a total amount per day but applied to the length of the second. The whole point of introducing leap seconds as of 1972 was to make UTC run on the SI second, which it previously didn't.

However, applying an increment annually, daily, hourly, even every second, is logically the same, surely?

Surely. As surely as applying an increment every second would make a mockery of claiming to use UDUNITS. And I do recall seeing some discussion about sub-second precision...

[davidhassell]

Hi Patrick,

I'm sorry, but I don't understand the UDUNITS analogy - could you expand (are you agreeing or disagreeing :))?

One of the reasons that the original discussions took so long was trying to understand the relationship between UTC time as stored in a CF dataset and elapsed time in the world (real or idealised) being measured.

Consider the difference between the UTC datestamps 1961-03-01 0:0:0 and 1961-03-01 0:0:1 (a rubber second), and the difference between the UTC datestamps 2016-12-31 0:0:0 and 2017-01-01 0:0:0 (includes the most recently applied leap second). Non-leap-second-aware software would calculate these differences as 1 SI second and 1 SI day respectively. Leap-second-aware software would give the differences as 0.999999985 SI seconds (a reduction of 15 ppb) and 1.000011574074074 SI days (86401 SI seconds), respectively.

I hope this illustrates why I think that the pre-1972 era is logically no different to the post-1972 era. Messier, for sure, but not really different!

Cheers,
David

[pvanlaake]

Hi David,

The "rubber second" is compressed (of shorter duration) relative to the SI second. So it is not a question of "applying an increment every second" - there is no such increment, it is a rate of divergence (my terminology) baked into the second that the UTC clock pre-1972 used. There is a computational solution, for sure, but at a significant cost of complicating code. And again, the leap second was introduced precisely because the pre-1972 system was considered too cumbersome.

Your suggestion has a very awkward fit with the concept of "second" as used otherwise in the conventions. Throughout the #304 thread, it is brought up on a few occasions that the "second" in CF is the SI second.

Is there a compelling need for CF support of the period 1961-1972? (1958-1960 is a real no-no because there was no UTC during that period and there are no published clock rates or step increments.) Can you identify a single data set that has true UTC datetimes that cover that period? Would it not be better to exclude this period until someone expresses that compelling need?

[davidhassell]

Hi Patrick,

Firstly, I'm more than happy to restrict ourselves to after 1972. It will also prevent anyone from the unlikely event of re-writing existing data from before 1972 using the utc calendar (which is almost certainly not what they want to do!).

This doesn't change my assertion that the text is "not wrong" for the pre-1972 period, but I think we're splitting hairs on different conceptual views of the same thing, so I'll stop going on about it, now :).

With regards use cases, as I recall the original use case was for a tai calendar in the remote sensing community, but we felt you can't introduce TAI with dealing with the concept of leap seconds, which we felt was best done by introducing a corresponding utc calendar to properly define it.

Cheers,
David

[ChrisBarker-NOAA]

“”” Non-leap-second-aware software would calculate these differences as 1 SI second and 1 SI day respectively. Leap-second-aware software would give the differences as 0.999999985 SI seconds “”” Well, “rubber seconds” aren’t the same as leap seconds, though they serve a similar purpose. With Leap seconds, seconds are always an integer. So we *could* say the “fully UTC compliant software could manage that” it can’t be handled with integer seconds. In fact, it looks like you’d need finer than microseconds, so pretty problematic. (And single precision float wouldn’t work either). And in any case when mapping between calendars, you couldn’t do it at all if the otgg bf er calendar only gandkk k ed integer seconds … So yes, it’s still a well defined transform, but from a practical perspective it’s a pretty different beast. Again, I’m concerned about the attractive nuisance — is there ANY software that can handle this correctly? So wouldn’t it be better to disallow it? Hmm — thinking now, it might actually work better to use the standard calendar in that case. You could use unaware software to transform back and forth, and it would be lossless, unlike leap seconds, where that almost works, except when there is an actual leap second, as most (all) non-leap second aware software doesn’t allow a 60th second.

[pvanlaake]

it might actually work better to use the standard calendar

This is also my thought. Allowing the inclusion of 23:59:60 into a datetime of a standard, proleptic_gregorian or julian calendar with units_metadata = "leap_seconds:utc" would solve many of the issues we have been discussing.

That opens up the issue of older software not being able to properly read newer data sets but that is a different discussion. (Software tends to follow new conventions. Should older software, i.e. of the kind that is no longer kept up-to-date, be actively "safe-guarded"? That is a rather unusual arrangement in the computing world and really holding back these conventions.)

[larsbarring]

CF is usually enhanced, expanded or changed based on concrete use cases. Over the years it has been argued, and accepted, that there are enough use cases to include the UTC and TAI calendar, where the latter was more of an complement to the UTC calendar.

However, concrete use cases are missing, even though there have been many references to that the satellite folks need it. And to that there are high-resolution measurement that may need this kind of precision. When I asked a well-placed EUMETSAT person I got the response that they reset their clocks at midnight (we did not go into any further details, so this is a bit of hearsay). And the only recent concrete high-frequency data collection example was from David @davidhassell, where they used UTC but was not even aware of the concept of leap seconds (David please correct me if I have mis-interpreted something). Way back I was involved in field measurements using sonic anemometers where the raw 20 Hz output was stored and the system clock was in the measurement hardware. But the final data was then reduced to 5- or 10-min covariances (etc), meaning that the second precision was not needed anymore. Similar systems today I imagine would either use NTP or GPS as clock.

All in all, without concrete uses cases we are not on firm ground, and I would really like to have at least one concrete use case to help us here.

[JonathanGregory]

Dear all

I think David is correct in asserting that the current text is not wrong, because in Sect 4.4.3 we say, for integer-second datetimes, "a time coordinate value expressed in seconds equals the number of valid (integer-second) datetimes after (not including) the reference datetime in the units up to (and including) the datetime that the time coordinate represents." That is, the time coordinate value is not the length of the time interval in SI seconds between the two instants. It is actually the count of the number of valid integer-second datetimes between them. These two quantities are equal except when a leap second intervenes in calendars where leap seconds are not valid datetimes, most importantly the standard calendar.

In that sense, rubber seconds are less problematic than leap seconds because, as Chris remarked, the standard calendar has the correct set of datetimes for that situation. There are always 86400 integer-second datetimes in a day in UTC between 1960 and 1972, so the encoding and decoding of time-coordinates by counting integer datetimes is easy!

However, the current text isn't adequate in Section 4.4.2, because we don't mention that seconds aren't SI seconds in UTC before 1972, or about the 0.1 s and 0.2 s jumps. As a curiosity, how did they deal with these in datetimes? If 0.1 s was inserted at the end of a day, for instance, did the clock jump from 23:59:60.1 to 0:0:0?

The simplest solution is to disallow the utc calendar before 1972, as Patrick suggested. I think we must also insert a further comment, however, about time-coordinates before 1972-1-1 0:0:0 UTC in all real-world calendars except tai, which is precise, and allowed back to 1958.

In other calendars, before 1972, when real-world datetimes are encoded as time-coordinates, the difference between two time-coordinates is in general not exactly equal to the length of the time-interval between them in SI seconds. Obviously it can't be, before the atomic clock was invented. The further back you go, the less accurate and universal the second was. For me, this underlines that the correct general intepretation of a real-world time-coordinate is as an encoded datetime. This encoding is additionally useful because, since 1972, and except for leap seconds, the difference between two time-coordinates equals the length of the intervening interval, but that's not the fundamental definition. Of course, in idealised model calendars, it's always exactly true.

Best wishes

Jonathan

[pvanlaake]

Recapping the discussion of the past weeks to see where we stand on the different issues:

• There appears to be a consensus to restrict the utc calendar to start at the datetime "1972-01-01 00:00:00".
• An argument has been made to not allow time zone offsets on either utc or tai calendars because the clocks used by these calendars do not use the concept of time zones. Where a time is expressed with a time zone offset from UTC it is "civil" or "local" time. Using utc as the name of a CF calendar suggests that datetimes are in "real" UTC. Perhaps have a "real" utc calendar and an additional civil or local calendar? These arguments are not supported by several contributors so the issue is still open (from my perspective anyway).
• Concrete use cases should guide the definition of CF calendars. (An existing data set using TAI with a time zone offset comes to mind...)

Perhaps for CF-2:

• Base the description of the entire section 4.4 on ISO 34000 and/or OGC terminology.

[ChrisBarker-NOAA]

Because now you also need to define julian_utc and julian_tai.

Actually, that was my point -- there is no use case for julian_utc or juliam_tai

Why in the world would anyone want to use the Julian calendar with that level of precision in modern times??

But this brings up an idea for names -- "gregorian_utc" and "gregorian_tai" ?

( I guess a bit like how we use a naming scheme like that in standard names ...)

Lars -- I can't recall what your concerns were, but I think we're close here. What's still on the table:
now:

1. What name to use -- there's concern that simply "utc" isn't ideal, because it's not exactly the same in definition as utc -- that doesn't bother me, but to keep quoting teh zen of python (sorry, Patrick!) -- explicit is better than implicit, so if we can find a better name -- great!

2. Whether to allow timezone offsets -- I think that only Patrick is worried about that -- but maybe with a new name, he'll be OK with it ?

Have I missed siomething?

[davidhassell]

I agree, Chris, I suggested similar names in https://github.com/orgs/cf-convention/discussions/400#discussioncomment-11891114 (noting in https://github.com/orgs/cf-convention/discussions/400#discussioncomment-11894036 that "standard" is preferred to "gregorian", meaning that it might be better to add a suffix to the preferred name, rather than a deprecated one)

[JonathanGregory]

I think that your (1) and (2) are linked, Chris, since Patrick's objection to utc as a name is partly that UTC does not have time zones.

I recall the proposal of calendar names like standard_utc. In CF1.12 we adopted units_metadata leap_seconds to make the distinction instead.

[JonathanGregory]

Dear all

The CF committee discussed this matter today. We decided it would be helpful to convene an online group to talk about it, because we think that would be more effective than discussion in writing. The group should consider:

• Whether there are any flaws in the CF1.12 convention for time coordinates (sec 4.4). That means conceptual mistakes that we made in formulating it, such that data written according to the convention might not be interpretable correctly and unambiguously. We're not aware of any flaws (in this strict sense), but if there are any we must rectify them.

• Whether there are any respects in which CF1.12 is not interoperable with ISO and OGC time standards, and how we can make the correspondence clearer where it is useful to do so. We don't expect it to include all their concepts, nor that they should include all of CF's, because each standard has a distinct set of applications.

• Where the text of CF1.12 should be clarified or the substance improved. Some possibilities have already been discussed here, such as to exclude datetimes before 1972 from the CF utc calendar.

If the group identifies changes for consideration, they will be proposed and discussed on GitHub as usual.

If you'd like to join this group, please say so here within the next three weeks (before 18th February). Thanks

Jonathan

[pvanlaake]

Dear Jonathan,

Thank you and the Committee for your support for this. I would like to join the group.

Best,
Patrick

[larsbarring]

Please include me. / Lars

[sethmcg]

I'm in!

[cofinoa]

I would like to join the group.

[davidhassell]

I would like to join the group.

[davidhassell]

Thanks, Jonathan. I would like to join the group.

[ChrisBarker-NOAA]

please add me, too -- I'll join if I can.

[JonathanGregory]

Thanks to the above for expressing interest in this discussion. I have sent you a poll to find a date. If there's anyone else who would be interested, please say so.

[JonathanGregory]

Dear @chris-little

I'm replying to your comment above, where you wrote, "I strongly recommend a Best Practice of data creators/servers using UTC timestamps without timezones, as recommended by various restrictive profiles of ISO8601." The group which convened to continue this discussion viva voce agrees with you! As a result of our discussions, we hope to have revised text of the calendar section before the CF workshop, and definitely in time for CF 1.13. Among other things, we agreed:

• That CF 1.13 disallow the use of time-zone offsets in the reference date-time of the units of a time coordinate variable in the utc and tai calendars. This is a backward-incompatible change (in the generous interpretation of principle 9 of Sect 1.2), but we don't think it matters because these calendars were newly introduced in CF 1.12 and they probably have not yet been used in archived data.

• In the other calendars (which all existed before CF 1.12) numeric time-zone offsets are allowed (as they always have been), but we propose recommending in CF 1.13 that time-zones should not be used. This is because it's easy to make mistakes with them, especially with the sign and in allowing for seasonal changes in clocks. (Principle 7 in Sect 1.2 is "The conventions should minimise the possibility for mistakes by data-writers and data-readers.") A time-zone offset is simply a number of hours to be added or subtracted when converting between time coordinates and date-times. Exactly the same can be achieved by modifying the time coordinates i.e. converting them to the zulu time-zone. Note that named time-zones e.g. GMT and MDT are never allowed. CF 1.12 says so in 4.4.1, but we could say it more strongly.

Best wishes

Jonathan

[ChrisBarker-NOAA]

... we propose recommending in CF 1.13 that time-zones should not be used.

I must have missed that meeting. I disagree. I think offset are totally appropriate when used for data that are already "local time" , e.g. have an offset.

This is because it's easy to make mistakes with them, especially with the sign and in allowing for seasonal changes in clocks. (Principle 7 in Sect 1.2 is "The conventions should minimise the possibility for mistakes by data-writers and data-readers.")

Exactly -- so why invite mistakes by requiring people to convert when the save the data??? I"m found it's a helpful principle to record data the way it come in, without doing any conversion -- use the units they data come in as, and use the timezone offset the data comes in with.

I work with this issue a LOT -- and in my work, folks usually want data in "local time" -- and it a whole lot easier (and less error prone) to use an offset than to have to convert when writing the data and then convert back.

Personally, I think using an offset with the UTC calendar is also reasonable -- what differs the UTC calendar from the standard calendar is the issue of leap seconds, not timezone offsets. But I do understand that principle, and the confusion around what "UTC" means so I'm not going to argue that one.

Note that named time-zones e.g. GMT and MDT are never allowed.

Totally agree with that.

Anyway, I believe this is a discussion for a hackfest, yes?

[pvanlaake]

I must have missed that meeting

Seeing that you commented on the section on "agreements reached" in the minutes of that meeting, I am afraid you are having a momentary memory lapse here. The points that Jonathan flagged are pretty much verbatim taken from that section.

[ChrisBarker-NOAA]

I am afraid you are having a momentary memory lapse here

All too likely :-(

Though I don't think I would have agreed to not recommending time zone offsets in any context at all.

I guess it comes down the final choice of words.

This isn't a blocker for me in any case.

[chris-little]

@ChrisBarker-NOAA @JonathanGregory I think the difference of opinion and experience are because of differing use cases. I was coming from the perspective of making data and services available globally, such as from widely used archives or experiments, etc. The local data creation case is also valid, but when that data is "published" to a wider range of consumers in multiple time zones, UTC "Z" makes more sense. If conversion to local time is needed, then that is an task for client software that knows when and where it is, as jursidictions do seem to change daylight saving hours and even time zones. HTH

[ChrisBarker-NOAA]

@chris-little: well, yes and no. I agree that the CF use case is about "making data and services available globally".

Which is why it is critical that time data have a fully defined specification -- e.g. no "naive" date-times. (or "local", or a political timezone, like US Pacific Time)

But UTC with an offset is a perfectly correct and precise specification.

The use-case I have in mind:

A user has data that was recorded in a "local" time. They know what the offset is, so , for example the timestamps may be somethign like (in ISO format):

2025-09-16T05:56:34-8:00

The question is: on whom should the burden be to convert that to UTC (or for that matter, any other timezone offset)?

I think it's a good principle of data management to store data as it's recorded -- e.g. if something is recorded in grams, report it in grams, rather than converting to kg -- there is room for error in any conversion.

Following that principle, if timestamps are recorded with an offset, they should be stored with that offset, rather than converting to UTC t store it.

All that being said -- this is about recommendations, so I'd have to see the final wording of the recommendation before I complain ;-).

[pvanlaake]

A date-time like 2025-09-16T05:56:34-8:00 is perfectly valid. But as we have discussed in this thread before, that is not UTC but local time which is generally based on UTC but under the "management" of politicians or bureaucrats that do funny things, like adding or changing daylight savings time. So adding a calendar local or civil would cater to that specific scenario and be fully aligned with ISO guidance.

[ChrisBarker-NOAA]

@larsbarring: is there an issue to be solved? As long as the data is recorded with a known offset from zulu time, then all is good, yes? Are there cases where someone used "local" time without clearly defining what "local" meant in that particular context? (i.e. what the offset was)

The fact is, if you use anything other than zulu time and don't define what that is, then all bets are off anyway.

Which brings up an question: should there be a way to store "naive" datetimes in CF? what that would mean is "no idea what the timezone was" or "look elsewhere to figure out what the timezone was". There may be a use case for that, as unfortunate as it is.

The danger is that someone simply uses the standard calendar, and does not specify an offset (which ISO says means local time), and then the data consumer interprets it as Zulu time.

Another option, which I really don't like, is to allow a civil timezone specification, e.g.: "Australian Central Time" -- which could be used as a true "local" time. These CAN be worked with, thanks to the Time Zone Database, but what a pain!

[larsbarring]

@ChrisBarker-NOAA: "As long as the data is recorded with a known offset from zulu time, then all is good, yes?" Yes, this is exactly what I meant. But equating that with local time (or civil time) may not always be true. A concrete example: If I am getting some data date-timestamped 2025-09-17T10:00:00+01 is that referring to:

• local time in UK, British Summer Time (BST = UTC+1),
• to data from a system not observing the CET to CEST change (i.e. not local time),
• or is it referring to CET+1, i.e. the date-time part is not UTC, and it actually refers to Swedish local time
• or is there a typo/technical fluke accident in the offset part (or even the month part) that needs to be corrected?

But I absolutely agree with your conclusion that, as you write,

Exactly why the concept of "local" time should not be in CF

[chris-little]

Declaration of Interest: I am a meteorologist, who takes it for granted that WMO Regulations are adhered to and all internationally exchanged data is time stamped with UTC Zulu time. Clocks in Forecast Offices stay on that time, irrespective of the local tme. And an extreme example, admittedly, but my wristwatch has two hour hands - one on UTC Z wherever in the world that I happen to be! (But we won't talk about Australian Post Offices - another story.)

[pvanlaake]

Presumably your watch also checks with BIPM twice a year if a leap second is coming up so that you can maintain second-level accuracy!

[chris-little]

@pvanlaake Actually it gains about one second every few weeks, so I occassionally check the radio beeps. I did once stay up late when there was a leap second declared - 7 beeps instead of 6 - (not) very exciting :-(

[pvanlaake]

With CF-1.13 published, inclusive of the issues discussed in this thread, I think we may close this discussion. Having opened the issue on 2024-12-22, we are about to complete a lap around the Sun, not counting any leap seconds. May the next lap bring more fruitful discussions!

[JonathanGregory]

I have added the frequently asked question to this discussion, although it wasn't a question in the first place, because it may be useful for reference in future.

[JonathanGregory]

In January 2025 the CF committee decided to invite anyone who was interested in this Discussion to join a group to meet online in order to resolve the problems identified and disagreements about the text concerning time coordinates, calendars and leap seconds in CF 1.12. The members of the group which formed in response to the invitation were Chris Barker, Lars Bärring, Antonio Cofiño, Jonathan Gregory, David Hassell, Seth McGinnis and Patrick Van Laake. The group met on 10th March, 30th April, 2nd June 2025. The attached record of the group’s deliberations is in reverse order, starting at the bottom with the agenda for 10th March. Between the meetings the record includes views contributed by the members of the group on the matters to be discussed at the following meeting. The conclusions of the group, agreed by email after the third meeting, are shown at the top of the record. These were the basis for the changes made to the conventions in CF 1.13 agreed in conventions issue 613. This record may be useful if similar issues are discussed in future.

[davidhassell]

Thank you for preparing and uploading these notes Jonathan. If similar issues are discussed in future, I'm sure that they will be useful :)