Skip to content

Conversation

@bouweandela
Copy link
Member

@bouweandela bouweandela commented Dec 10, 2025

Description

Move the preprocessor output filename template to the new configuration.

Upgrade instructions

If you have defined projects in config-developer.yml that are not in the default config-developer.yml file shipped with ESMValCore, then you will need to move the value of 'output_file' to

projects:
  <insert project name here>:
    preprocessor_filename_template: <insert output_file value here>

Closes #2919

Link to documentation:


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.


To help with the number pull requests:

@bouweandela bouweandela force-pushed the configure-preprocessor-output-file branch from f01c6d1 to 6770eee Compare December 10, 2025 11:47
@bouweandela bouweandela force-pushed the configure-preprocessor-output-file branch from 6770eee to fd5992d Compare December 10, 2025 11:49
@codecov
Copy link

codecov bot commented Dec 10, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.59%. Comparing base (dc340e9) to head (d8df0ac).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2923      +/-   ##
==========================================
+ Coverage   95.57%   95.59%   +0.01%     
==========================================
  Files         266      266              
  Lines       15520    15556      +36     
==========================================
+ Hits        14834    14870      +36     
  Misses        686      686              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bouweandela bouweandela marked this pull request as ready for review December 10, 2025 14:34
@bouweandela bouweandela added this to the v2.14.0 milestone Dec 10, 2025
Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks very good - I am wondering if we want to start thinking about allowing preprocessor files that are not netCDF4 for regular cases ie allow a bit of flexibility when it comes to appending that .nc extension - probably, but in the future, and definitely outside the scope of this PR. Cheers @bouweandela 🍻

@bouweandela
Copy link
Member Author

I considered the option of adding a DataSink similar to the esmvalcore.io.protocol.DataSource but then for writing data to, so you could write your data as zarr to an object store, but the difficulty is that diagnostic scripts wouldn't support this, so that would require some further work on the ESMValTool side as well, especially for languages other than Python, and I'm not sure how useful this is anyway as diagnostic scripts are typically expected to work with small amounts of data after the heavy lifting has been done by the preprocessor functions.

@bouweandela bouweandela merged commit d4ce7a1 into main Dec 12, 2025
4 checks passed
@bouweandela bouweandela deleted the configure-preprocessor-output-file branch December 12, 2025 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move the preprocessor output filename template to projects configuration

3 participants