Clarify use case for sorting of changes.csv

[bernhardreiter]

For changes.csv files to be useful for implementers, the understanding of its properties must be good.

A corollary from the current_release_data requirement (https://docs.oasis-open.org/csaf/csaf/v2.1/csaf-v2.1.html#document-property-tracking-current-release-date) and the requirement that changes.csv is sorted with newest at the top, is that changes.csv should only grow at the top. New entry will be freshly "published" aka "released" and so they can only be added at the top.

To implement a simple incremental downloader, someone could then start reading and downloading each line of changes.csv from the top, until a line is hit with an entry that I already have or that is elder then the last time I've downloaded things.

The matches a mental model which I have as an implementer: "changes.csv" is like a log and can only grow at the top (and be truncated at some point). (This nicely matches the fact that changes.csv exists at all, because the sorting and the publication date is what makes a difference to the index.txt. What would it be good for, if it is not to make implementation's life easier. )

If entries would be added to the middle of the file, then the above algorithm wouldn't work. I had to parse everything and diff it to the last revision to see if there is a new entry with an elder publication date that has been added. ISDuBA does something like this and the implementation of what do download has become more complicated as a result.

For simplification I suggest to add a note about the above corollary.

(And similar to CSAF ROLIE, but for this I need to check the ROLIE details first, it may already be specified in the standards it is build upon.)
(There is the related question of "recent" means for changes.csv. It is only useful if the timespan is significant, so maybe giving a minimum that implementation can rely on like 3 months or 12 months. Probably worth another issue.)

[bernhardreiter]

Notice send to https://groups.oasis-open.org/discussion/please-consider-to-clarify-the-changescsv-use-cases-1280

[bernhardreiter]

Considering what happens if a changes.csv file is recreated from scratch (for whatever reasons, e.g. for an update), looking at a simplified file (which later turns out to be wrong - see below (edit 20260211):

4.json, 2026-02-03
5.json, 2026-02-02
5.json, 2026-02-01
2.json, 2026-01-02
4.json, 2026-01-01
3.json, 2025-12-10
2.json, 2025-12-10
1.json, 2025-12-01

on disk are all five documents in its latest revision
(= latest current_release_date), so when regenerating changes.csv
from disc we get (deleted lines kept empty for display):

4.json, 2026-02-03
5.json, 2026-02-02

2.json, 2026-01-02

3.json, 2025-12-10

1.json, 2025-12-01

The information is lost that there have been previous revisions published in the file, though this information is still with the revision_historyhistory.

The lost lines violate the mental model of a log file. So I'd say they SHOULD be kept.

Considering the incremental downloading algorithm (suggested above):

start reading and downloading each line of changes.csv from the top, until a line is hit with an entry that I already have or that is elder then the last time I've downloaded things.

Let us consider all cases:

1. No previous downloads: all is downloaded (anyway)
2. Last download 5.json, 2026-02-01: down to 5.json, 2026-02-02 is downloaded because that revision is newer than the revsion we have.
3. Last download 2.json, 2026-01-02: downloading up to there, because we have it.
4. last download 4.json, 2026-01-01: down to 2.json, 2026-01-02 is downloaded, because 3.json, 2025-12-10 is elder.

-> the algorithms works, even if old lines are deleted, after a regeneration

Suggestion: Entries of changes.csv SHOULD be kept over time until they are removed from the bottom when not recent anymore.
(Depending on how the minimum interval of 'recent' is prescribed.)

[tschmidtb51]

The lost lines violate the mental model of a log file. So I'd say they SHOULD be kept.

Disagree: The changes.csv was never intended to keep track of every change to the file but only of the latest change for each file. Especially if you have many changes this files would grow fast and known other changes does not provide a value to the users as they always only download the latest version.

[tschmidtb51]

Note to self:

• work with "recently changed" => maybe SHOULD contain at least last year?
• but allow longer (up to all)
• clarify use case: incremental download
• change to "insert on top only"
• add consequence: sorted by current_release_time
• add requirement for csaf provider => MUST reject uploads if newer file exists

[bernhardreiter]

Thinking about this today, after discussing the issue with @tschmidtb51 yesterday, here is a better suggestion:

• Remove changes.cvs completly.
• change index.txt to serve all purposes

this would simplify writing and reading for non-ROLIE users, except for unlikely cases.

update index.txt to include datetime

New format for index.txt:

2023-07-01T10:09:07Z,2023/esa-2023-09953.json
2023-07-01T10:09:01Z,2021/esa-oo-2021-03676.json
2022-04-17T15:08:41Z,2022/esa-2022-02723.json
2022-03-01T06:01:00Z,2021/esa-oo-2021-00042.json

now index.txt can be used to

• download all files
• incremental downloads
• downloads up to a datetime (aka recent)

Having the datetime in front has a main technical advantage:

1. it is a fixed length, so the 21 characters can be cut away directly. File names are trailing, as they may have different length. (That means a CSV reading library can be dropped from the dependencies)

To get the old index file back, it is a simple string operation, e.g. the POSIX tool box can easily do it:

 cat index.txt |  cut --characters=22- | sort >index.old.txt

Small technical advantages

• for technical analysis index.txt can directly be displayed by web browsers.
• for writers, the needed sort would work directly on it.
• the file system space for the old index file is saved.

The minor drawbacks:

• very simple downloaders that just take index.txt to get everything will have to add the "cutting" and potentially "sorting" part. Those are simple operations. I expect taking index.txt directly to be a rare case as the most interesting CSAF providers are huge and simple downloading won't work in many cases anyway. Rate limiting will get in the way for instance. Sequential downloading will take very long and some connections will fails so you need retry possibilities.
• changes.csv users will have to change the download to download index.txt instead and have to simplify the match to the comma to the more simple cut of 21 characters. Again I suspect changes.csv to not be used very often. csaf_downloader and isduba downloading kernel prefer ROLIE. And the missunderstanding about the changes.csv use cases discussed above are an indicator that usage is low.

scratch requirement 13

(but do not renumber, so 13 stays unused, but other requirement numbers stay stable)

The big advantages is: simplification of the standard:
One file less to care about, one requirement less that is to be kept in mind and be understood.
Less file mechanics that can be implemented wrongly and would need testing.

[bernhardreiter]

To explain why comment #1280 (comment) and some of my initial thinking was in the wrong direction:

• CSAF 2.0 mandates that changes.cvs has all entries. It does not say but it means only all entries once. So it is not a log-file. It cannot have entries more than once.
• The new mentioning of "recent" in CSAF 2.1 (current main) is confusing, as is the use case note for the index.txt. Both changes.csv and index.txt can also be used to download all files as they have all entries.

[tschmidtb51]

1. it is a fixed length, so the 21 characters can be cut away directly. File names are trailing, as they may have different length. (That means a CSV reading library can be dropped from the dependencies)

This is not 100% true: You can have time-secfrac (optional part) and the timezone can also be presented as a six character string.

And the misunderstanding about the changes.csv use cases discussed above are an indicator that usage is low.

Not necessarily true: It could also indicate that other implementers didn't came across that situation (as they publish their CSAFs always according to the current_release_date) or interpreted the standard in the same way as the TC ;-)

[bernhardreiter]

You can have time-secfrac (optional part) and the timezone can also be presented as a six character string.

My suggestion is to fix the datetime to the ISO format variant: yyyy-mm-ddThh:mm:ssZ then the length is fixed. :)

(Yes, I cannot really say how much usage there is for changes.csv now, but with index.txt in the suggestion fully supporting the use cases it can be used for in 2.0 there is no loss of functionality. At most there would be a small change in coding. ;-) )

[tschmidtb51]

Given that discussion, I see multiple ways forward for the TC:

1. Leave this as is except for fixing the "recent" wording (default, but possibly a bad decision as it does not improve anything for the users)
   • pros: no change at all
   • cons: no improvement of the situation
2. Change the wording around the changes.csv and ROLIE feeds to enforce "each file only once, insert on top only"; keep index.txt as is
   • pros: easy rule
   • cons: no safeguard if a CSAF provider does not stick to the rule of publication = current_release_date
3. Change the wording around the changes.csv and ROLIE feeds to enforce "each file only once, insert on top only"; keep index.txt as is; add additional requirement for CSAF Provider software to reject CSAF Documents if they contain a CSAF Document with a newer current_release_time (note: same time would still be inserted on top)
   • pros: software safeguard prevents accidental late publication
   • cons: CSAF provider software needs to implement rule, harder in automation due to rejection
4. Change the wording around the changes.csv and ROLIE feeds to enforce "each file only once, insert on top only"; change content of changes.csv to contain at least filenames of documents that change during the course last year - but it may contain more up to containing all; keep index.txt as is; add additional requirement for CSAF Provider software to reject CSAF Documents if they contain a CSAF Document with a newer current_release_time (note: same time would still be inserted on top)
   • pros: software safeguard prevents accidental late publication; limited time frame allows for smaller, but still useful changes.csv; not a heavy burden on CSAF providers that release files only a couple times a year: they can still update the changes.csv whenever they have time
   • cons: CSAF provider software needs to implement rule, harder in automation due to rejection (order is important when regenerating)
5. Change the index.txt as suggested by @bernhardreiter above; make the changes.csv optional, extend filename by time frame (e.g. last hour, last 24h, last 7 days, last month, last year)
   • pros: more options to provide useful changes.csv windows for CSAF providers
   • cons: more potential files to generate and check; changes.csvs not reliably available; index.txt with "hidden structure" (i.e. not just a list of files); no fallback option if changes.csv is corrupt
6. Drop index.txt; change the wording around the changes.csv and ROLIE feeds to enforce "each file only once, insert on top only"; add additional requirement for CSAF Provider software to reject CSAF Documents if they contain a CSAF Document with a newer current_release_time (note: same time would still be inserted on top)
   • pros: consistent as all files are sorted ones in the changes.csv; somewhat easy to implement
   • cons: no fallback option if changes.csv is corrupt; human-readable option vanishes
7. Change the index.txt as suggested by @bernhardreiter above, drop the changes.csv completely
   • pros: human-readable option still available
   • cons: no fallback option if index.txt is corrupt; index.txt with "hidden structure" (i.e. not just a list of files)
8. Change structure of changes.csv (and ROLIE feed) to track inserted at timestampand sort by that value; keepindex.txt` as is
   • pros: easy to detect inconsistent CSAF provider, never miss a CSAF document again
   • cons: more data to store; harder to regenerate from scratch; potentially more overhead for consumers

[tschmidtb51]

For any suggestion, we need to clarify that a CSAF incremental downloader probably can't just rely on the time but needs the last line downloaded as well.

[bernhardreiter]

To clarify a little bit further: option 7. has the big advantages that it is the only option that actually makes the standard shorter and simpler. (I do not think that anyone will compare the full contents of index.txt and changes.csv now to detect differences. At least that is not an idea I would easily have. And there is the ROLIE feed for those who offer both.)