Pretest data

[wkirgsn]

Dear hosts,
my team is going through the pretest data set and we study the provided material and scripts to better understand the evaluation metrics.
Two things seem weird - maybe you can illuminate these issues?

1. Padded HDF5 Loss key
   When extracting the HDF5 data for the "loss" component, it appears that both hdf5 data sets {mat}_Testing_Padded.h5 and {mat}_Testing_True.h5 contain the same loss values.

padded_loss[msk_N].ravel()=array([103.30951663, 103.30951663, 103.30951663, ...,  12.04704808,
        12.04704808,  12.04704808], shape=(2100,))
true_loss[msk_N].ravel()=array([103.30951663, 103.30951663, 103.30951663, ...,  12.04704808,
        12.04704808,  12.04704808], shape=(2100,))

Looking at the webinar slides, on the formula for the normalized energy relative error, I understand that one provided loss should represent the full time series core loss and the other the shorter time series loss that define the pretest data.
Is the HDF5 data correctly provided?

2. Filtering of time series
   In lines 79-81 of Error_Distribution_Plot_Template.m it seems the time series are always cropped from start to a certain scenario-specific marker. Shouldn't it rather be a crop from a scenario-specific marker to the end of the time series as that's where the predictions are?
   If I am not mistaken, the reported error distributions are too optimistic then.

[wkirgsn]

I found in the git history that there was an update to the matlab file 2 weeks ago, and the results PDF was also updated.
However, the fix is only touching B but misses to fix the prediction and measurement variable as well.
The reported error distributions are still too optimistic then, in my opinion. Can we have a check on this?

[PaulShuk]

1. The core loss values provided were the same just to make sure the two files have consistent indexing. The energy calculation is done simply with the provided measurement sequence as benchmark.

2. In regards to the fixing of B and H. We fixed B because of the reasons you mentioned. However, the H that we use here only contains the cropped segments from scenario-specific marker to the end. We did not include them in the predicted and measurement segments. Hence, a 10% info predicting 90% info scenario would generate only 900 H values from the python code. This is the reason why H is taking it from the beginning to the end. I hope this helps.