Clarifications About Evaluations

[shyams2]

I have some questions regarding the evaluation methodology:

1. VRMSE Calculation: Are the preliminary VRMSE metrics computed using de-normalized quantities?

2. Masked Domains: For problems with masked domains (e.g., helmholtz_staircase and acoustic_scattering_maze), how are evaluations performed? Are they:
   a) Accounting for the mask explicitly, or
   b) Conducted on the whole domain?

3. Rollout Loss Clarification: Could you confirm if my understanding of the VRMSE loss calculation for rollouts is correct?

   • Initial input: 4 timesteps [u_{-3}, u_{-2}, u_{-1}, u_{0}]
   • Autoregressive prediction: [u(1), u(2), u(3), u(4), ..., u(30)]
   • 6:12 VRMSE losses: Computed using [u(6)...u(12)]
   • 13:30 VRMSE losses: Computed using [u(13)...u(30)]

Thank you for your assistance

[rubenohana]

Hello,

Thank you for your questions! Here are the answers:

1. The VRMSE is computed in normalized space.

2. For the datasets with a mask, the evaluation is performed on the whole domain (the mask being set to zeros in the physical field + added to the t0_fields with NaNs where the mask should be and zeros elsewhere). However, because the mask is present is constant_field_names in the data, it is appended to the data in DefaultChannelsFirstFormatter here but is not being predicted.

3. I think this is not correct. It would be more something like this:

   • Initial input: 4 timesteps [u(0), u(1), u(2), u(3)]
   • Autoregressive prediction: [u(4), u(5), u(6), u(7), ..., u(34)]
   • 6:12 VRMSE losses: Computed using [u(4+6)...u(4+12)]
   • 13:30 VRMSE losses: Computed using [u(4+13)...u(4+30)]

Let me know if you have more questions!

[shyams2]

Thank you for your response! I have a follow-up question regarding the handling of masked domains. I understand that the mask is included as an input to provide geometric information, but the model doesn't predict the mask itself. Just clarifying again if my understanding is correct:

1. Even in these cases, I should still be proceeding in the usual sense and capture:

    vrmse = calculate_vrmse(gt, pred)

2. Or, should I be using the mask to filter out irrelevant regions in my predicted output before calculating VRMSE? Specifically, would the correct approach be to apply the mask to both ground truth and predictions before VRMSE calculation, like this:

   masked_gt = gt * ~mask # ~mask to only finding values in the physical domain of interest
   masked_pred = pred * ~mask
   vrmse = calculate_vrmse(masked_gt, masked_pred)

Your insights on this would be greatly appreciated. Thanks again!

[rubenohana]

I think it really depends on your task and objective. If I remember correctly, in our benchmarks, we computed the vrmse as in 1. This forces the model to learn to predict zeros where the mask is. However, you could try to compute it as in 2., I think this would influence the computation of the vrmse?

Let me know if you have more questions!

[shyams2]

I see! I'll try out using both for the computing the VRMSE. Thanks so much for your clarification!

[dawonahn]

Hello!

I think you're making a minor mistake in indexing here for the autoregressive predictions. It should start from the last index of the input i.e., this

6:12 VRMSE losses: Computed using [u(4+6)...u(4+12)]
13:30 VRMSE losses: Computed using [u(4+13)...u(4+30)]

should read

6:12 VRMSE losses: Computed using [u(3+6)...u(3+12)]
13:30 VRMSE losses: Computed using [u(3+13)...u(3+30)]

Feel free to correct me if I'm mistaken but it does not make sense since it starts from u(4) while you're only inputting from u(0) to u(3).