Remove leap seconds conversion rule because it is non significant

[bernhardreiter]

Current draft has a leap second conversion rule in 9.1.5:

csaf/csaf_2.1/prose/share/csaf-v2.1-draft.md

Lines 12479 to 12480 in 76fb206

	In addition, the converter outputs a warning that leap seconds are now prohibited in CSAF and the value has been replaced.
	The CVRF CSAF Converter SHOULD indicate in such warning message whether the value was a valid leap second or not.

and 9.1.18:

csaf/csaf_2.1/prose/share/csaf-v2.1-draft.md

Lines 13048 to 13051 in 76fb206

	* value type `string` with format `date-time`: If the value contains a `60` in the seconds place, the CSAF 2.0 to CSAF 2.1 Converter MUST replace
	the seconds and their fractions with `59.999999`.
	In addition, the converter outputs a warning that leap seconds are now prohibited in CSAF and the value has been replaced.
	The CSAF 2.0 to CSAF 2.1 Converter SHOULD indicate in such warning message whether the value was a valid leap second or not.

I propose to remove these rules on the basis that they cover an extreme edge case.

They do not have the significance to have their own conversion rule nor test of this rule. This would all be additional code and contents that is not helping CSAF's mission.

There is not and will not be (with > 99% confidence) any leap second in a real world CSAF 2.0 document.

Rationale:

• The last leap second was from 2016 to 2017. It is unlikely there will be some in the next years. There were 27 leap seconds over 34 years so far.
• date-time is used in the revision history for the CSAF documents and CSAF is newer than 2017, so valid 2.0 document cannot have a leap second in there.
• date-time can be used in discovery_date which holds when the vulnerability was originally discovered. The chance for a leap second to be hit is tiny. A (unrealistic high) upper limit is 0.81% : We have about 320000 CVE#s available. Let us assume the leap seconds and the discovery dates were evenly distributed, this means we have a change of 27 leap seconds /(34 years *365 days *24 hours *60 minutes *60 seconds) for hitting one on a single discovery. For all CVE#s to miss, we'd have (1-27/(24*365*60*60*34))**320000 that is about 99.194%. The non-even distribution (more vulnerabilities found in recent years) will make that number go higher in practice.
• CSAF 2.1 will supersede 2.0 soon.

Checking on our testing ISDuBA instance, on all 376000 documents, there is no leap second.

psql -t -c "select count(*) from documents" isduba
 376429

psql -t -c -t -c "SELECT count(*) FROM documents WHERE original LIKE '%23:59.60Z%'" isduba
  0

In the very unlikely case that such a real world 2.0 document still gets created, the CSAF 2.1 validation at the end of the conversion would fail and the person converting would need to handle this manually.

While the saved time for readers of the standard is good.
And the saved time and saved code (with potential defects) for handling this in the converter and the test documents for the converter is an upside.

[bernhardreiter]

Mentioned: https://groups.oasis-open.org/discussion/remove-leap-second-conversion-rules-1348

[tschmidtb51]

IMHO, the TC should not remove these conversion rules:

• After a long and heated discussion, the TC decided to deviate from RFC 3339 and prohibit the representation of leap seconds in CSAF 2.1 due to the lack of support in various programming languages. As they were valid in CSAF 2.0 and CSAF CVRF 1.2, the tools need to have clear guidance how to handle files that contain leap seconds. Otherwise, the conversion into CSAF 2.1 is non-deterministic (and we try our best to prevent that).
• There are several publishers that provide data in CSAF 2.0 (and CSAF CVRF 1.2) dating back before 2017. Superseding CSAF CVRF 1.2 and CSAF 2.0 does not intent to render that data useless - instead the converters should help to make a faster move to CSAF 2.1.
• Different multipliers could automatically convert old documents into CSAF 2.1 - if we leave the decision how to handle the situation to the implementers, different outcomes are very likely.
• Those rules only have to be implemented once (in the reference implementation). A tool maker can always decide to not fulfill the conformance target. Now, they can at least flag what would be missing to fulfill the conformance target which provides clarity for users.

Off-Topic

I guess that won't change something in your result, but:

1. Leap seconds could occur in any field of type date-time. It looks like you are running the statement against the original JSONs. But those values are most likely not all in UTC... (Otherwise, they wouldn't be the original.)
2. Most likely, the database is incomplete - we certainly don't know of all CSAFs. (I'm pretty sure about that as I know about at least 5 document issuers that publish CSAFs as TLP:GREEN and higher.)
3. There is no complete repository of CSAF CVRF 1.2 documents as far as I know.

Nevertheless, I flagged it for TC discussion - feel free to close if my arguments convinced you.

[bernhardreiter]

Thanks for taking the time to respond. I think it is okay to keep the leap seconds out, this is just about the conformance target and corresponding implications to converter implementations. Someone will write unnecessary code with potential defects for this if it is in the standard, so it should not be in there. Because there will (>99,9%) be no documents where this matters.

It looks like you are running the statement against the original JSONs. But those values are most likely not all in UTC... (Otherwise, they wouldn't be the original.)

Can you explain further? My (probably very limited) understanding is that YYYY-MM-DDT23:59:60Z is the only accepted RFC3339 leap second representation, as stated in https://www.rfc-editor.org/rfc/rfc3339#appendix-D .

[tschmidtb51]

Thanks for taking the time to respond. I think it is okay to keep the leap seconds out, this is just about the conformance target and corresponding implications to converter implementations. Someone will write unnecessary code with potential defects for this if it is in the standard, so it should not be in there. Because there will (>99,9%) be no documents where this matters.

Agree to disagree. IMHO, we value consistency over probability. I don't see a good reason to remove a rule the TC came up with just because that case might seem rare. (I know, this is not the Mars Climate Orbiter...)
And: As you create the test cases, I'm confident that you provide a well structured approach that covers all potential defects ;-)

Can you explain further? My (probably very limited) understanding is that YYYY-MM-DDT23:59:60Z is the only accepted RFC3339 leap second representation, as stated in https://www.rfc-editor.org/rfc/rfc3339#appendix-D .

Let me cite from Appendix D (emphasis done by me):

When required, insertion of a leap second occurs as an extra second at the end of a day in UTC, represented by a timestamp of the form YYYY-MM-DDT23:59:60Z. A leap second occurs simultaneously in all time zones, so that time zone relationships are not affected. See section 5.8 for some examples of leap second times.

As the leap second happens in every time zone at the same time, it can't be always at "23:59:60". Also, it must be able to be represented in the local time. The section 5.8 also provides the examples:

Here are some examples of Internet date/time format.
[...]
1990-12-31T23:59:60Z

This represents the leap second inserted at the end of 1990.

 1990-12-31T15:59:60-08:00

This represents the same leap second in Pacific Standard Time, 8
hours behind UTC.

Please note that there is an implication, that is easy to miss:

 1937-01-01T12:00:27.87+00:20

This represents the same instant of time as noon, January 1, 1937,
Netherlands time. Standard time in the Netherlands was exactly 19
minutes and 32.13 seconds ahead of UTC by law from 1909-05-01 through
1937-06-30. This time zone cannot be represented exactly using the
HH:MM format, and this timestamp uses the closest representable UTC
offset.

The leap seconds are also one reason (but not the only one), why it is so hard to get the regex complete and correct for RFC 3339. (And such regex will never be future proof. Therefore, regex in this usually only covers the basics - the rest is normally done in a proper time library.)

[bernhardreiter]

Thanks for the long explanation, I've missunderstood the Appendix D from RFC3339. From there my quick test was wrong.

IMHO, we value consistency over probability. I don't see a good reason to remove a rule the TC came up with just because that case might seem rare.

It is feedback from trying to implement that part of the specification, supported by @s-l-teichmann and myself. We expect the explicit test and conversion rule for lead second to have small negative impact on the 2.1 standard.

For a standard (or any application) to be successful, it should be good in the main area of usage. That is where the energy of the committee and everybody should be spend. From our understanding the leap seconds are not relevant for any of CSAF intended use cases. 60 seconds will not make any difference here. That is the reason to briefly see if this feedback of implementers is to be followed: Make implementations and standards easier to remove an unneeded piece.

The rest of the argument was to discuss what would happen in the rare case of cases: Nothing bad. The validator will barf after conversion, the person supervising the converter will adjust the time in question once (in a reasonable way) and it is done. So the rare edge case is taken care of.

If there is any discussion in the TC, it is not worth having a discussion about leap seconds (they are irrelevant). They could only serve as an example and study point about what is relevant and what is not. Maybe establishing a rule of thumb of what to not discuss in detail, but just remove because of irrelevance in other cases.