Using train_set for evaluation in MackeyGlass LSTM benchmark?

[paulhaider]

Hello NeuroBench team,

I was reviewing the lstm_benchmark.py example for the Mackey-Glass dataset and noticed a potential issue regarding the evaluation setup. In the benchmark function, specifically at line 168, it appears the train_set is being used for evaluation:

    test_set_loader = DataLoader(train_set,
                                 batch_size=mg.testtime_pts,
                                 shuffle=False)

In addition, I cannot make sense out of why batch size is set to testtime_pts.
The actual test_set defined above in line 117 is not used anywhere else in the benchmark script.

I have also changed it to use the actual test_set but then performance gets much worse with sMAPEs of around 30.

Is it just me not understanding the code or is there a potential issue? I'd appreciate if you could clarify the reason behind using train_set for this particular evaluation step in the benchmark, or if this might be an oversight?

[YounesBouhadjar]

Thank you so much @paulhaider for pointing this out.
The idea was to use the training set as a warmup to initialize the LSTM’s hidden states (see section 3.1 in https://doi.org/10.1016/j.mlwa.2022.100300), but something went wrong in the implementation.

We've just corrected the benchmark #263 to evaluate the LSTM on the test set, using the hidden states warmed up at the end of training. As far as I can see, the performance is almost similar to before.

Regarding your other question, the batch size above refers to the number of predicted points, which is used for evaluating the performance during the autonomous mode.

[paulhaider]

Great!
When I evaluated on test I forgot to warmup so that explains the discrepancy there.
And thanks for the clarification!