handling empty observation

[mer-a-o]

The check_obs function in SWELL checks whether an observation file is empty. If it is, the observation is then removed from the observation list
to prevent the file from being used downstream.

This behavior causes issues in cases where only one type of observation is being assimilated and some assimilation cycles contain no observations. For example, TEMPO observations are only available during the daytime, so each day includes at least one assimilation window without any observations. In these cases, SWELL builds an incomplete JEDI executable YAML, leading to a runtime failure:

...
observations:
  observers: []
...

A more robust approach could be to create empty observation files during the ingestion (storage) step in R2D2. These files would then be fetched during assimilation, and the JEDI executable YAML could be configured in the same way as for non-empty files.

JEDI can handle empty observation files by generating empty feedback files and storing them in R2D2. This would simplify the SWELL logic and eliminate the need for special checks and conditional handling.

This approach might also be preferable since the ignore_missing capability was removed in R2D2 v3. However, users would need to be aware that empty observation files will not trigger any distinct warning messages, so additional validation or logging may be helpful.

I'm opening this issue so we can discuss and identify the best way moving forward. Please let me know what you think.

[Dooruk]

From a general perspective, having empty files is fine but it will be waste of lot of ionodes, especially now the disk space is limited. Also we still don't know R2D2 performance for a lot of local users with thousands of files (scalability).

From marine only perspective, one complication is having to run DA in different window lengths (6-hr, 1-day or 5-days)

I have the following method to combine observations, assuming they are all ingested in 6-hourly chunks:

swell/src/swell/tasks/get_observations.py

Line 420 in 29ebff8

def read_and_combine(self, input_filenames: list, output_filename: str) -> None:

Some PT6H observations might be missing from the full period, say while combining five 6-hourly observations for a 1-day DA window. That handling is not perfect.

[jeromebarre]

I agree with Doruk with the potentail limitation of the ionodes...
One tradeoff I am seeing is to have one template of empty file stored somewhere protected.
The workflow would then still have the check in place but would then still place the observation in the observer list and then create the obsspace class but pointing to that empty file.

This could be done during this check obs loop or even during the fetchobs task with the try/except section where the r2d2 query could fall back to linking to this empty file. This could be a bit dangerous as if the necessary warning are not in place one careless or beginner user could run experiments without assimilating anything while thinking they do.