Update error thresholds.

[odiazib]

After @mjschmidt271 updated the script to check for errors, many tests were reported as skipped because they did not pass the error threshold. I reviewed all the skipped tests and decreased the tolerance to pass a few tests where the relative error was smaller than 1e-8.

We still have many tests that are marked as skipped, which have a significant relative error:

64 - validate_stand_modal_aero_calcsize_sub_update_ptend (Skipped)
66 - validate_stand_calcsize_aero_model_wetdep_ts_379 (Skipped)
74 - validate_ma_precpprod (Skipped)
90 - validate_compute_massflux_small (Skipped)
134 - validate_pcarbon_aging_1subarea (Skipped)
202 - validate_calc_1_impact_rate_ts_0 (Skipped)
210 - validate_stand_aero_model_calcsize_water_uptake_dr_ts_379 (Skipped)
214 - validate_clddiag (Skipped)
222 - validate_wetdep_prevap_130 (Skipped)
234 - validate_wetdep_scavenging_true (Skipped)
236 - validate_wetdep_scavenging_false (Skipped)
240 - validate_rain_mix_ratio (Skipped)
246 - validate_wetdep_resusp_130 (Skipped)
248 - validate_wetdep_resusp_230 (Skipped)
370 - validate_newton_raphson_iter_ts_355 (Skipped)
512 - validate_chm_diags_ts_355 (Skipped)
538 - validate_calc_het_rates_merged (Skipped)
546 - validate_sethet_merged (Skipped)
616 - validate_mam_soaexch_1subarea_ts_379 (Skipped)
620 - validate_mam_gasaerexch_1subarea_ts_379 (Skipped)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.35%. Comparing base (34c4d5c) to head (993fcad).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #483      +/-   ##
==========================================
- Coverage   93.43%   93.35%   -0.08%     
==========================================
  Files         303      303              
  Lines       25177    24283     -894     
  Branches     2766     2767       +1     
==========================================
- Hits        23523    22669     -854     
+ Misses       1654     1614      -40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[odiazib]

After updating the src/validation/mam_x_validation/scripts/compare_mam4xx_mam4.py script:

The following tests FAILED:
64 - validate_stand_modal_aero_calcsize_sub_update_ptend (Failed)
66 - validate_stand_calcsize_aero_model_wetdep_ts_379 (Failed)
222 - validate_wetdep_prevap_130 (Failed)
234 - validate_wetdep_scavenging_true (Failed)

[singhbalwinder]

Thanks, Oscar. Do you know what their required tolerances are to pass?

[mjschmidt271]

@odiazib I believe this line in mam_x_validation should use np.all() rather than np.any(), since this is to determine whether a test passes, in contrast to the test above that to determine if it fails.

if np.all(np.array(errvec_o[0:3]) < error_threshold):

To be repetitive for the sake of exactness/posterity, in words:

If absolute errors (errvec_o[:3]) are all less than the error threshold, then the absolute-error pass condition (abs_error_pass) is true.