What is the difference between calendars standard/leap_seconds:none and tai in CF 1.12

[Alexander-Barth]

Question

I am considering to add support for the new calendar types to the julia package CFTime.jl. I am wondering what would be the difference between a time variable using the calendar standard (or proleptic_gregorian) with leap_seconds: none as units_metadata and the calendar tai which is also Gregorian and without leap seconds.

In example 4.1, both represent 2017-1-1 0:0:0 because no leap seconds in the timeline:

variables:
  float time_tai ;
    time_tai:standard_name = "time" ;
    time_tai:long_name = "Satellite data" ;
    time_tai:calendar = "tai" ;
    time_tai:units = "seconds since 2016-12-31 23:59:58" ;
  float time_stdnone ;
    time_stdnone:standard_name = "time" ;
    time_stdnone:long_name = "Model data with no leap seconds" ;
    time_stdnone:calendar = "standard" ;
    time_stdnone:units = "seconds since 2016-12-31 23:59:58" ;
    time_stdnone:units_metadata = "leap_seconds: none" ;
[...]
data: // time coordinate variable and the datetime it represents
  time_tai = 2; // 2017-1-1 0:0:0 because no leap seconds in the timeline
  time_stdnone = 2; // 2017-1-1 0:0:0 because no leap seconds in the timeline
[...]

As far as I can see, a validator should return an error or warning if dates prior to 1958-01-01 are used for tai. But are there other differences (for the purpose of computing durations between time instances, adding duration to time instances, computing year, month, day, hour... from time instances) ?

[JonathanGregory]

Dear @Alexander-Barth

Thanks for the question. You're right that dates before 1958-1-1 should give errors with tai (both as dates to encode/decode, and as reference dates after since). Apart from that the algorithm for conversion between date-times and time coordinates is exactly the same for standard none and tai [JMG deleted the following because it's confusing - please see my next answer: and in both of them the difference between two time coordinates equals the interval of time between the instants]. The difference between them is that a date-time in tai refers to the real-world timeline, whereas in standard none it does not - it's an idealised world. The standard none calendar is commonly used in climate models for historical integrations, ignoring leap seconds; in such cases it is likely that the date-times are intended to correspond to UTC, not TAI, although of course the difference is only a matter of seconds. But this doesn't affect the manipulations; it's just the meaning which is affected. Does that answer the question?

Best wishes

Jonathan

[Alexander-Barth]

Thank you for your reply :-)
I am struggling a bit with the concept that calendar: standard, leapsecond: none is an idealized world as opposed to a real timeline. Does it imply that we cannot compare dates in UTC/TAI and dates using dates in the standard/proleptic_gregorian calendars with a precision of seconds (assuming that the leap seconds handling is specified using units_metadata)?

For example, users of the julia package CFTime.jl can compare and convert dates in different calendars:

using CFTime
DateTimeStandard(2000,4,24) == DateTimeJulian(2000,4,11)
# output: true
convert(DateTimeJulian, DateTimeStandard(2000,4,24))
# output: DateTimeJulian(2000-04-11T00:00:00)

I would like to give the users the ability to do that with the tai and utc calendars. In the future, scientists will likely need to combine data using the standard calendar with data using utc or tai. It would then be necessary to convert/compare dates expressed in UTC/TAI and standard/proleptic_gregorian.

I am wondering if it is really appropriate, to consider data as likely UTC (with leap_seconds included), if, for example, the data provider is explicit that leap seconds are not accounted for (units_metadata = "leap_seconds: none"). I assumed that units_metadata was introduced to avoid this ambiguity.

Or a related question: if UTC/TAI are the only real-world timelines (only applicable after 1958-01-01), how would I encode the (real) moment that predates 1958-01-01 by 1 SI second if the calendars standard/proleptic_gregorian do not apply to the real-world timeline?

Thanks a lot for your insights and clarification!

[JonathanGregory]

Dear @Alexander-Barth

These are good questions too! They will help us clarify the text, at least, and I ought to have been clearer in my first answers.

The essential reason for introducing units_metadata is to avoid the ambiguity about the length of intervals between instants. calendar="standard" with units_metadata="leap_seconds: none" asserts that there are no leap seconds in the timeline. The standard none calendar should not be used for datasets from the real world, with UTC date-times, because there really are leap seconds in that timeline. That's why I called it idealised.

To use the example in Fig 4.1, if the standard calendar has been used to encode date-times, then the time coordinate 3 seconds since 2016-12-31 23:59:58 should always be decoded as 2017-01-01 00:00:01, regardless of units_metadata. In the real world, the length of the time interval between 2016-12-31 23:59:58 UTC and 2017-01-01 00:00:01 was 4 seconds, because of the intervening leap second. The data-writer should put leap_seconds: utc if they have encoded UTC date-times using the standard calendar, to indicate that the time coordinate value may not equal the elapsed interval between the two instants; to get the exact interval, you have to correct for leap seconds. If the data-writer puts leap_seconds: none, they are asserting that the interval is really 3 seconds. That's not the real world.

Thus, the CF metadata tells you how to decode the date-times and how to calculate time-intervals between date-times, but it does not indicate whether the date-times in the standard calendar should be interpreted as real-world UTC date-times. The standard calendar is sometimes used for simulations that are intended to be comparable with observations, by climate models which use the Gregorian calendar but do not include leap seconds. In such a simulated world, it really is 3 seconds between 2016-12-31 23:59:58 and 2017-01-01 00:00:01. This is not the real world. However, comparison with observations is only possible by interpreting the model date-times as UTC. For climate purposes, the stretching or squeezing of the time axis by a few seconds does not matter.

This isn't different in principle from more serious distortions that are commonly applied to the timeline of climate models. Many models do not use the Gregorian calendar exactly; instead, they omit leap days. In that case, an exact comparison of observed data for 29 February 2016 with the simulation is clearly impossible, but a time-mean for February 2016 can reasonably be compared, since one day doesn't make an important difference for many purposes. However, this is a judgement for the data-user to make.

You write

In the future, scientists will likely need to combine data using the standard calendar with data using utc or tai. It would then be necessary to convert/compare dates expressed in UTC/TAI and standard/proleptic_gregorian.

Yes, I agree. Conversion between utc and tai is unproblematic. If the observed date-times are UTC, the user can certainly choose to compare them with model data, by treating model date-times as though they were UTC. In doing so, the user will have judged that the differences between the real and model timelines that I've just mentioned are immaterial for their purposes.

if UTC/TAI are the only real-world timelines (only applicable after 1958-01-01), how would I encode the (real) moment that predates 1958-01-01 by 1 SI second if the calendars standard/proleptic_gregorian do not apply to the real-world timeline?

When I said that standard none is idealised, not realistic, I meant since 1958. Before 1958, there were no leap seconds, even in the real world! Hence, as you'd expect, this instant could be encoded as 0 seconds since 1957-12-31 23:59:59, among infinitely many possibilities [in the standard or proleptic_gregorian calendars].

Does that help?

Best wishes

Jonathan

[pvanlaake]

Dear Alexander,

I am not sure that Jonathan’s assertion that the standard calendar is idealised, whatever the reference period, is very helpful to you. By the same logic, the clock on each and every computer is idealised, being based on the POSIX implementation which uses 1970-01-01T00:00:00 as its epoch and blissfully ignoring leap seconds, relying instead on its next synchronisation with an NTP server (at some point in time) to align with local time which is typically based on UTC, and then blissfully ignoring past leap seconds when calculating intervals. By the same logic UTC is idealised, as it is a time scale that is continuous with discontinuities (as per ISO 34000:2023), relative to UT1, which itself is idealised in a relativistic sense…

The utc and tai calendars were introduced in CF to enable users to have a second-accuracy time recording framework, with leap second consideration in the case of the utc calendar. TAI, the time scale underlying the tai calendar, is effectively (algorithmically) identical to the implicit (in the sense of not being formally defined) time keeping on the standard calendar. The only restriction (in CF) on the tai calendar is that the earliest valid instant is 1958-01-01T00:00:00. A second prior to that is 1957-12-31T23:59:59 on the standard and proleptic_gregorian calendars. No question about that. TAI may not have existed prior to 1958, but time certainly did.

IMHO, the definition of the utc calendar in CF has multiple problems. For starters, it did not exist until 1962 and only from 1972 onwards did it use SI seconds and were 27 (not 37) leap seconds inserted (in addition to 10 seconds difference relative to TAI at 1972-01-01). Furthermore, as currently defined, both utc and tai calendars can use a time zone offset, which is possibly deliberate, maybe accidental, but apparently considered appropriate. Effectively this means that utc is not UTC and tai is not TAI by default, but only in the special case that the time zone offset is 00:00. For utc, use of a time zone offset makes it local time (again as per ISO 34000:2023, but also the common understanding of time), or, more conveniently, your local computer time (noting the parenthesised caveat on NTP synchronisation above). For tai, use of the time zone offset is part of the imaginary (idealised) world of CF writers, again IMHO. But I’ll repeat: utc is not UTC and tai is not TAI.

The units_metadata is also problematic. So a data writer can indicate that the time scale includes leap seconds, but they cannot be represented in the respective calendars. So what? How about this text from CF-1.12, section 4.4.3:

If it is essential for leap seconds to be counted in time coordinates, so that they exactly equal time intervals, you must use the utc calendar.

Clear. Followed by:

For many applications of the standard, proleptic_gregorian, and julian calendars, these inaccuracies are too small to matter, but there are some applications where it is necessary to know about them. Therefore it is recommended that for the standard, proleptic_gregorian, and julian calendars the appropriate treatment of leap seconds should be indicated by giving the time coordinate variable a units_metadata attribute containing a leap_seconds keyword with one of the permitted values none, utc or unknown.

Huh? First we are told to use the utc calendar if we care about leap seconds, and then we are told that we can use some other calendars as well but using an attribute having a potential value of “unknown” if “it is necessary to know about them”.

So talking from one CFtime (R) developer to another CFTime (jl) developer: (1) ignore the units_metadata attribute until someone starts complaining on your issues page (my money is on not before Kingdom come); (2) disallow time zone offsets for the utc and tai calendars until someone starts complaining…; (3) start utc at 1972-01-01 until…

Please note that the current definition of both calendars is being discussed (issue #400) and there may be changes coming.

[ChrisBarker-NOAA]

I am not sure that Jonathan’s assertion that the standard calendar is idealised, whatever the reference period, is very helpful to you.

Huh? First we are told to use the utc calendar if we care about leap seconds, and then we are told that we can use some other calendars as well but using an attribute having a potential value of “unknown” if “it is necessary to know about them”.

I think the key point is that the standard calendar is an ugly, poorly defined mess. We shouldn't pretend it's anything else. It's a legacy result of incomplete software.

What it is is defined by how it is (was) used:

Time-stamps (day, month, year, time) are encoded/decoded to/from the time axis (timedelta since ...) using the Gregorian calendar with no leap seconds.

That's it -- that's all we know. a few CF versions ago.

The problem is that with only that, we don't know what exactly the data mean:

A) The time axis could have been the primary source -- e.g. for model results.
B) The time axis could have been computed from timestamps, e.g. for measurements from an instrument.

If (B) -- the timestamps may or may not have had leap-seconds included.

That's simply the legacy facts.

What we do know:

For (A): If you decode the time axis using software that doesn't include leap seconds, you will get timestamps that don't include leap seconds, and if you compare those results to proper UTC time, you may be off by up to 37 (I think) seconds -- for model results, that probably doesn't matter -- certainly no one has cared in the past.

For (B): If you decode the time axis using software that doesn't include leap seconds, you will get timestamps that match the timestamps originally started with. - the limitation here is that it is impossible to encode timestamps that include an actual leap second (i.e. a 60th second). But what you don't know is whether the original timestamps did or did not include leap seconds.

So that's where we were a couple CF versions ago.

To address the uncertainty about leap seconds in the standard calendar:

units_metadata="leap_seconds: none"
and
units_metadata = "leap_seconds: utc"

were introduced -- now users could know what the original timestamps contained.

NOTE: some of us suggested that we introduce a new calendar name for: "leap_seconds: utc", but given all the legacy data, it was decided to handle it this way -- old datasets, and old software that didn't know about units_metadata would retain the same old ambiguous definition. Anyway ....

Back to the original question:

What is the difference between calendars standard/leap_seconds:none and tai

I think for practical purposes -- the answer is: there is no difference. Theleap_seconds:none concept was introduced before the tai calendar -- if we had introduced tai first, maybe we would't have introduced it at all -- I'm not totally sure.

For new datasets, I'd simply use tai is you know it's tai. A reason to use 'standard' would be if you want older software that isn't aware of tai to still work.

(note: there's a downside there -- I don't know if the tai calendar has made its way into downstream tools yet...)

However, leap_seconds:utc IS still useful -- and maybe necessary -- because most software does not handle leap seconds -- so if you want users to be able to decode your files correctly with most software -- leap_seconds:utc is what you need to use.

Now -- how to get this all into the CF doc clearly ----.

NOTE:

If it is essential for leap seconds to be counted in time coordinates, so that they exactly equal time intervals, you must use the utc calendar.

Patrick wrote:

First we are told to use the utc calendar if we care about leap seconds,

Well, this is a problem -- because, of course, while using the utc calendar is the correct thing to do:

1. software that does that properly is hard to come by (I know of no Python library for instance)

2. There's a LOT of legacy data out there that was encoded with the "standard" calendar -- one could fairly easily add an attribute to clarify the leap-seconds issue, but re-encoding the data is bigger lift.

-CHB

[JonathanGregory]

I agree with Chris's comments except that I would say there is a practical difference between leap_seconds: none and tai. Here I restate my first answer, and try to be clearer this time:

Dates before 1958-1-1 should give errors with tai (both as dates to encode/decode, and as reference dates after since). Apart from that, the algorithm for conversion between date-times and time coordinates is exactly the same for standard none and tai.

The difference between them is that date-times in the tai calendar identify instants of International Atomic Time of the real world, whereas date-times in the standard none calendar identify instants of time in an idealised world. tai can be used both for observations and for models, provided they use exactly the same timeline as the real world, but standard none can only be used with models.

When the standard none calendar is used for model simulations of real history, a given date-time refers to an instant in the model timeline which corresponds (to the extent that a simulation can be compared with reality) to the instant that has the same date-time in UTC. Thus, for example, the standard none date-time of 2025-05-05 16:55:00 identifies the instant in the model which corresponds to the instant with 2025-05-05 16:55:00 as its UTC date-time in the real world. In the real world, this same instant is labelled 2025-05-05 16:55:37 in TAI. Thus, a given instant in the real world has different date-times in the tai and standard none calendars, when the latter is used for a model simulation that is supposed to represent real-world history. Equivalently, a given date-time refers to different instants of real-world time in these two calendars.

That is why tai and standard none are distinct calendars, even though their algorithms for conversion are the same. They differ because, in the world of the standard none calendar, the rotation of the Earth is constant, unlike in the real world, so there are no leap seconds. That makes it slightly unrealistic, but the difference is so small that it's irrelevant for nearly all physical simulations.

Although we haven't discussed it (because no use-case has made it relevant), the standard none world must be idealised in other ways as well, if it's assumed to be appropriate for simulating a constant climate with present-day insolation forever. I think this implies that the standard none world has a solar year of exactly 365 days + 5 hours + 56 minutes + 24 seconds = 365.2475 * 86400 seconds, whereas in the real world the solar year is approximately 365.242189 * 86400 seconds = 365 days + 5 hours + 48 minutes + 45 seconds. With the standard none length of solar year, the Gregorian calendar is exactly right to remain synchronised indefinitely with the equinoxes.

[Alexander-Barth]

That is why tai and standard none are distinct calendars, even though their algorithms for conversion are the same. They differ because, in the world of the standard none calendar, the rotation of the Earth is constant, unlike in the real world, so there are no leap seconds.

If I understand this correctly, with calendar=standard leap_seconds: none, it means that one apparent revolution of the sun in the sky ("day") is exactly always 86400 SI seconds. With the UTC and TAI calendars the earth rotation is considered as variable. So the calendar encodes information about how the daily cycle in the model is implemented (analogous to calendar 360_day, where one revolution of the earth around the sun is 360*86400 seconds).

Is this correct?

I am now wondering what is said about the Earth rotation when we use calendar=standard leap_seconds: utc. Is it also constant at 86400 SI seconds (one apparent revolution of the sun in the sky) except when there is a leap second, then it is exactly 86401 SI seconds?

[pvanlaake]

There are two principal physical phenomena that govern time on Earth.

The first one is the rotation of the Earth around its own axis. That rotation is not constant due to a variety of celestial (gravitational effects from the Moon and large planets, for instance) and human-induced (melting of the glaciers is moving mass from the poles towards the equator) factors. Because of all the ensuing problems and the need to have a universally (read: globally) agreed time, a number of time scales have been developed over the last 150 years, usually in reference to the positions of far-away stars that have no apparent movement with respect to the Sun and the Earth. More recent time scales are based on the SI second. TAI was introduced in 1958 and is a continuous time scale, ticking away SI seconds forever; for convenience of notation only it uses the Gregorian calendar with a day length of exactly 86400 SI seconds. Formally, TAI has nothing to do with the Earth's rotation, nor with the length of the day or the year; as a result TAI time is drifting away from the seasons as we know them (if you wait long enough, the boreal summer will be in December). UTC, on the other hand, is based on TAI but occasionally corrected by a full leap second to maintain synchronicity with the tropical year (i.e. the seasons stay where they are during the year).

The second physical phenomenon is the revolution of the Earth around the Sun on the equatorial plane. This is what real-world calendars are about. The current common Gregorian calendar is based on having exactly 365.2475 days in one year. This differs from the mean tropical year, which is variable over time and was 365.242189 days long in the year 2000. For all practical intents and purposes, these two year lengths are considered equal. The CF conventions do not in any way consider this difference, nor is it likely that they ever will. The Julian calendar has a year length of 365.25 days and is used by the standard calendar for periods prior to the introduction of the Gregorian calendar in 1582, as well as by the julian calendar.

So to answer your questions: in calendar standard with leap_seconds: none a day has 86400 seconds, as well as in the tai calendar. In the utc calendar a regular day has 86400 seconds as well; only when a leap second is inserted does the day have 86401 seconds.

[JonathanGregory]

@Alexander-Barth. I agree with Patrick. You are correct that calendar="standard" with units_metadata="leap_seconds: none" means that all days are 86400 seconds exactly. When calendar="standard" with units_metadata="leap_seconds: utc", the algorithm for converting between time coordinate values and date-times (year, month, day, hour, minute, second) is the same one as in calendar="standard" with units_metadata="leap_seconds: none" and calendar="tai" (which you asked about originally). Therefore, for example, the time coordinate for midnight on one day is always 86400 seconds greater than the time coordinate for midnight on the previous day. However, when units_metadata="leap_seconds: utc", the date-times are UTC. This means that, when a leap second occurs, it is actually 86401 seconds between successive midnights. In this situation, therefore, the difference between two time coordinate values (one before and one after a leap second) is not exactly equal to the actual elapsed interval between the date-times they represent (it is one second smaller). In all other calendars, and for intervals not involving leap seconds, the difference between two time coordinates exactly equals the elapsed interval between the date-times they represent. We introduced units_metadata leap_second so that the two different uses of the standard calendar could be distinguished, in order to provide sufficient metadata to allow time intervals to be calculated exactly. (By the two different uses, I mean the idealised world with no leap seconds where the standard calendar is the actual one, and the use of the standard calendar to encode UTC date-times.)

Is that clear? Thanks again for your questions. Arising from #400, a group of us are currently meeting (two meetings so far) to discuss how we can improve the text and/or clarify the convention itself regarding leap seconds, so any comment you can offer on what's the clearest way to understand or explain these issues would be useful.