Questions regarding Walrus evaluation and request for numerical data

[hassony2]

Hi !

Thank you for the great work on the Walrus paper and thank you for releasing the code an the pretrained checkpoint :)

I was wondering if you could share the following details:

• Numerical results for Figure 15 In the Appendix whichshows the VRMSE loss over time for the pretrained models. This would be extremely helpful to facilitate direct comparison with the pretrained checkpoint.

• Metric Clarification. Could you confirm the calculation for the "median VRMSE". Specifically that the VRMSE first averages over physical variables and (optionally) averages across all time horizons/steps before computing the median across test trajectories ?

• Could you confirm that results for tables 7 and 8 are are computed for finetuned Walrus while figure 15 is for pretrained model and table 8 for finetuned models ? (In page 9 of the paper it is written "Full loss over time plots can be seen in Appendix Figure 15 while direct numerical versions of these plots can be found in Appendix Table 8.", but it seems to me that Figure 15 is on pretrained model and Table 8 is for the finetuned one as mentioned page 31 "Table 8 shows the finetuning dataset numbers").

Thank you for the clarifications you might provide :)

All the best !

Yana

[mikemccabe210]

Hi Yana,

• Probably not in this case. It's not really feasible to plug that many numbers into a paper and we're not planning on releasing a dump of all computed metrics. What we can do is make the finetuned weights used for each of these accessible so they can be easily recomputed (or used for further computations), but that will take a little while to get done.
• Yep. The one exception to this being one-step computations where it's treated as a sliding window so the median is over all possible one-step ahead predictions over all trajectories.
• Splitting up this since there's a couple different answers:
  • Table 7 is just the pretrained model, but as page 5 notes, it's also a small random subset of the validation data that's being used as a quick ablation, so it's not directly comparable to full results listing.
  • Table 8 is finetuned - those are all from out of the training set. There are no zero shot experiments in the paper.
  • Yeah, that should say F14 which is the corresponding downstream data. Thanks for pointing that out. F15 on the other hand has both. "Walrus-PT" in the key corresponding to PT only.

Hope that helps!

Best,
Mike