VRMSE Metric Behavior for Gray-Scott Steady State Cases

[shyams2]

Hi,

I'm working with the Gray-Scott Reaction Diffusion examples for operator learning model training. I've noticed a potential issue with the VRMSE (Variance-normalized Root Mean Square Error) metric for steady state cases.

Specifically, when evaluating predictions on steady state examples:

• The MSE values are consistently low across all examples
• The VRMSE values are disproportionately large
• This appears to be due to the denominator (variance) approaching zero in steady state cases, which inflates the VRMSE

Could you clarify:

1. How the baseline results handle these steady state cases?
2. Whether the reported metrics include or exclude these cases?

Thank you for your help!

[mikemccabe210]

Hello,

This is expected behavior. VRMSE is a normalized metric that captures error relative to the magnitude of the overall variation in the field, so it intentionally penalizes error larger than the field variance which may occur if a model is unable to recognize that a system is approaching a steady state.

In all cases, the reported baselines include all data. If steady-state scenarios aren't relevant to your application, since this is a fairly common scenario in Gray-Scott, we do list the trajectories that reach steady state in the readme so it is possible to exclude them manually. During training, the min_std parameter of the dataset can be used to avoid penalizing steady state predictions too much (this is basically an epsilon term in the normalization).

Short version:

1. The baselines are trained with steady state data included - no special allowances were given for these cases.
2. The reported metrics do include these cases.

[rubenohana]

Hello,

We did not compared the predictions steady-state vs non-steady state of gray_scott_reaction_diffusion as this was not the goal of the paper. We are aware that this would affect the metrics. However, as we suggested in Appendix D of the paper, the steady-state was an interesting behavior to be studied!

Regarding the benchmark, it seems that you are not selecting the optimal learning rate. In the optimizer config file, the standard learning rate is 1e-2 whereas we report for FNO on gray_scott_reaction_diffusion that the optimal learning rate is 1e-3. We would advise you to try this value, and if you can't reach the desired loss, come back to us so we can discuss it!

[shyams2]

I see! Thank you for the clarification! I'll give that a go, and report back.

[shyams2]

@rubenohana I ran the code again using a reduced learning rate and I'm seeing much better results now! It seems like the losses are around ~1e-5 now. However, I wanted know how I can evaluate the performance of this trained model now on the test set. Specifically I want to know how I can check the VRMSE performance over the entire test set. Thanks in advance!

[rubenohana]

Now that your model is trained, you can run the same script with validation_mode=True and this will run the validation and test runs using the checkpoint associated to the model and learning rate. For example:
python train.py experiment=fno lr=1e-3 validation_mode=True server=local data=gray_scott_reaction_diffusion will launch the validation script with the checkpoints of the FNO pretrained with lr=1e-3. This should add the validation and test metrics in wandb.

[shyams2]

@rubenohana
Okay, got it! Thank you very much for your help.

I had another question regarding the appropriate metric to focus on. In my WandB console, under the "test_gray_scott_reaction_diffusion" section, I'm currently seeing full_VRMSE_T=all. However, I'm unsure if this is the metric we should be targeting for reproduction. Specifically, is this a rollout VRMSE calculated over the entire trajectory, or is it the average VRMSE calculated over all single-step ahead predictions?

The reason I'm confused is that there's also a separate "rollout_test_gray_scott_reaction_diffusion" section in my WandB logs that mentions full_VRMSE_T=0:1. This makes me uncertain about which full_VRMSE_T is the relevant metric to evaluate performance.

I've shared my WandB run log here for reference. Please note that I'm aware that these may not represent ideal results, as I've only been able to train for 16 hours on an A6000.

Any clarification on which metric I should be focusing on would be greatly appreciated!