add clarification on data updates

[tomkralidis]

Add to section for data publishers:

If a data publisher issues an update on a data granule, the associated WNM should:

• keep the same properties.data_id
• update the link object to use rel=update to refer to the updated granule (not rel=canonical)
• update properties.pubtime

[6a6d74]

Also add section for Implementation and operation of a Global Service / Global Cache.

Expected behaviour is described below:

The first aim of updates is to make sure that the (potentially) faulty data in the initial message will not be used by downstream processing by the users.
It means that, the initial download link in the cache should either be made unavailable or the file is updated 'in place' and reuse the same link.

Using an example:

        "links": [
            {
               "rel": "canonical",
               "type": "application/x-bufr",
               "href": "[http://wis2bra.inmet.gov.br/data/2024-05-22/wis/br-inmet/data/core/weather/surface-based-observations/synop/WIGOS_0-76-0-1709500000000161_20240522T030000.bufr4](http://wis2bra.inmet.gov.br/data/2024-05-22/wis/br-inmet/data/core/weather/surface-based-observations/synop/WIGOS_0-76-0-1709500000000161_20240522T030000.bufr4)",
               "length": 240
            },
            {
               "rel": "update",
               "type": "application/x-bufr",
               "href": "[http://wis2bra.inmet.gov.br/data/2024-05-22/wis/br-inmet/data/core/weather/surface-based-observations/synop/WIGOS_0-76-0-1709500000000161_20240522T030000.bufr4](http://wis2bra.inmet.gov.br/data/2024-05-22/wis/br-inmet/data/core/weather/surface-based-observations/synop/WIGOS_0-76-0-1709500000000161_20240522T030000.bufr4)",
               "length": 240
            },

when you get this message from Brazil the cache must:

• redownload the data using the link
• 'replace' the original file so
  a. delete the old one and create a new file with a new link
  b. replace the existing link and overwrite the content
• change all the links in the WNM that sends back to the origin
• publish a new message with a new id and the same data_id.

So, using this example above, the canonical and the update href must be changed to send to your cache.

If there is only a link with rel=update the GC should add a copy of the link to the WNM, albeit with rel=canonical

Note that client applications should always be looking for a link with rel=update first, then rel=canonical.

[antje-s]

I have the following questions about the behaviour described:

1. Should the cache really change the filename of updates to match the canonical when the centre sends a new, different filename? In my view, the responsibility here should lie with the centre that sends the updates. The same filename should already be used here.

2. In my opinion, no canonical is necessary for rel=update and rel=delete. Since rel=update and rel=delete are searched for first and rel=canonical is only used if there are no hits.

[tomkralidis]

2. In my opinion, no canonical is necessary for rel=update and rel=delete. Since rel=update and rel=delete are searched for first and rel=canonical is only used if there are no hits.

Note that TT-WISMD is addressing this as per wmo-im/wis2-notification-message#82 (comment)

[6a6d74]

Add to section for data publishers:

If a data publisher issues an update on a data granule, the associated WNM should:

• keep the same properties.data_id
• update the link object to use rel=update to refer to the updated granule (not rel=canonical)
• update properties.pubtime

@tomkralidis - has this change been documented?

[6a6d74]

Also add section for Implementation and operation of a Global Service / Global Cache.

Expected behaviour is described below:

The first aim of updates is to make sure that the (potentially) faulty data in the initial message will not be used by downstream processing by the users.
It means that, the initial download link in the cache should either be made unavailable or the file is updated 'in place' and reuse the same link.

[...]

when you get this message from Brazil the cache must:

• redownload the data using the link
• 'replace' the original file so
  a. delete the old one and create a new file with a new link
  b. replace the existing link and overwrite the content
• change all the links in the WNM that sends back to the origin
• publish a new message with a new id and the same data_id.

[...]

For a Global Cache to implement the behaviour outlined about, it would have to keep track of all the previous data granules it has published. This would incur significant overhead. Suggest that updates are managed at the client end - i.e., the client receives the update message, downloads a new version and overwrites any version of the data granule previously downloaded.

@golfvert - what do you think?

[tomkralidis]

Add to section for data publishers:
If a data publisher issues an update on a data granule, the associated WNM should:

• keep the same properties.data_id
• update the link object to use rel=update to refer to the updated granule (not rel=canonical)
• update properties.pubtime

@tomkralidis - has this change been documented?

Latest status is in wmo-im/wis2-notification-message#172 (comment). Once this is merged I will PR the Guide accordingly.

[6a6d74]

Latest status is in wmo-im/wis2-notification-message#172 (comment). Once this is merged I will PR the Guide accordingly.

Thanks @tomkralidis - this covers off the expected behaviour for the data publisher.

[6a6d74]

ET-W2IT meeting 24-Nov-2025

WIS2 doesn't behave like GTS where bulletins propagate from message switch to message switch over a period of time, allowing those message switching systems recompile bulletins when updates are received. In WIS2, messages are delivered to subscribers in seconds.

ET-W2IT conclude that managing updates in WIS2 should be resolved at the client end. For example, when a data consumer receives an update, they should prioritise the message with the latest properties.pubtime. Data consumers should discard any WNMs with identical properties.data_id but earlier properties.pubtime that have not yet been processed. And, where a previous WNM with identical properties.data_id but earlier properties.pubtime has previously been processed, (meta)data previously written to local systems should be overwritten. Data consumers should not assume that messages arrive in order (i.e., it is possible, albeit unlikely, that a WNM with an earlier properties.pubtime may arrive after a more recently published WNM). As a consequence, a data consumer should keep track of WNMs that have been processed so that WNMs with identical properties.data_id can be identified and the properties.pubtime of each can be compared to identify the latest version. Given that messages propagate rapidly through WIS2, a record of previous messages need not exceed a 60-minute duration.

(By way of example, the Sensor Centre Global Cache assumes that messages delayed by more than 10-minutes when re-published by a Global Cache are considered to be "missed").

Data consumers should take care when replaying earlier messages via (WIS2 Global Replay Service)[https://github.com/wmo-im/wis2-grep] to ensure that older messages from the replay service don't overwrite good data.

ET-W2IT agree that there is no need for Global Caches to overwrite cached data items if an update WNM is received.

Germany's Global Cache (de-dwd-global-cache) does keep track of messages received and overwrites updated data. However, the UK/USA (data-metoffice-noaa-global-cache) and Chinese (cn-cma-global-cache) Global Caches do not - they simply write a new object to the cache with a new identifier. If we assume that client applications are managing updates (as described above) there are no negative affects from the differing GC behaviour.

Global Cache operators may choose how they manage updates - i.e., they may or may not overwrite objects in their cache.

[6a6d74]

Relating to the discussion above, the Guide needs to be updated with information for data publishers and data consumers. We also need to record the ET-W2IT decision that Global Cache operators may choose how they manage updates.

• Update Guide for data publishers as per clarify data updates (#160) wis2-notification-message#172 (comment) (@tomkralidis)
• Update Guide for data consumers as per text below - or similar (@6a6d74)
• Record ET-W2IT decision (done - see Should Global Caches overwrite cached objects when update received? et-w2it#45)

Changes to the Guide should target the FT2026-1 release.

When a data consumer receives an update, they should prioritise the message with the latest properties.pubtime. Data consumers should discard any WNMs with identical properties.data_id but earlier properties.pubtime that have not yet been processed. And, where a previous WNM with identical properties.data_id but earlier properties.pubtime has previously been processed, (meta)data previously written to local systems should be overwritten. Data consumers should not assume that messages arrive in order (i.e., it is possible, albeit unlikely, that a WNM with an earlier properties.pubtime may arrive after a more recently published WNM). As a consequence, a data consumer should keep track of WNMs that have been processed so that WNMs with identical properties.data_id can be identified and the properties.pubtime of each can be compared to identify the latest version. Given that messages propagate rapidly through WIS2, a record of previous messages need not exceed a 60-minute duration.