Reproducible Results

[azencot]

Problem

Hello,

Great work!

The trained models in HF do not correspond to git's code (e.g., max_patches is wrong). Moreover, I could not reproduce your reported results (Table 2 in the paper). In general, it would be great if you could report the hyperparameters to increase chances of reproducibility.

Thank you so much for all your hard work!

Solution

Clearly report all hyperparameters used to produce Table 2 in the paper

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines

[RealLast]

Hi @azencot,

thank you for your interest and your kind words, we really appreciate it! Sorry for the late response, it has been a really busy couple of weeks recently.

We have now uploaded all model checkpoints to HuggingFace, including TSQA and M4:
OpenTSLM huggingface

Additionally, we have a branch with demo scripts showing how to use these checkpoints and the corresponding datasets. Will merge them in soon, but feel free to try them out already. If you use these scripts and our checkpoints, you should be able to reproduce our results:
Huggingface demo scripts

In general, regarding the hyperparameters, we did not change them and the ones in the curriculum_learning.py should still be valid. However, depending on the dataset, you need to update the max_patches parameter to either 1024 (for TSQA, M4, HAR-CoT and Sleep-CoT), and to 2600 for ECG-QA. We changed max_patches to 2600 to support the longer high-frequency time-series in the ECG dataset (30s @ 100Hz).
You can set this parameter (max_patches) for OpenTSLM-SP here and for OpenTSLM-Flamingo here.

Please let us know if it works or if you run into any issues. Happy to help debugging.

Best

[mltrinhammer]

Hi @RealLast!
Thank you so much for making the full range of models available on the hub, that is a huge help. I have been training them manually the past few months, on two H100s, yes, its been painful! 😆

I am experiencing an issue when trying to use any Flamingo-suite model from your stage5 (ECG). I can only load the weights from the hub when the CNNTokenizer max_patches argument is set to 1024, not 2600 as is expected. Please see this file in my fork for a reproduction.

The error when max_patches is set to 2600 is RuntimeError: Error(s) in loading state_dict for OpenTSLMFlamingo: size mismatch for model.vision_encoder.pos_embed: copying a param with shape torch.Size([1, 1024, 128]) from checkpoint, the shape in current model is torch.Size([1, 2600, 128]). size mismatch for llm.vision_encoder.pos_embed: copying a param with shape torch.Size([1, 1024, 128]) from checkpoint, the shape in current model is torch.Size([1, 2600, 128]).

In your previous reply to @azencot, you are referencing the files to change the max_patches argument. I get confused when I follow your links. As I understand your architecture, the CNNTokenizer is used in the Flamingo-suite, whereas the TransformerCNNEncoder.py is used in the SoftPrompt. The links are the other ways around? Please confirm if I am confused here.

Again, thanks for your incredible contribution; I am working on extending the architecture to my own domain day-and-night! All the best.

[RealLast]

Hi @mltrinhammer,

thank you very much for reporting this issue, and exciting to see you are currently training your own models! Regarding the CNNTokenizer vs. TransformerCNNEncoder, you are right: The CNNTokenizer is for Flamingo, and the TransformerCNNEncoder is for SoftPrompt. I updated my comment above to link correctly.

Regarding the mismatch in the model checkpoints: Basically the error you reported means that the checkpoint contains max_patches=1024, which should be true only for non-ECG models. If you tell me the exact model ID that you are working with, I can look into this and see whether we maybe uploaded a wrong checkpoint.

Best

[mltrinhammer]

Hi @RealLast, thank you for this!
The model I used is this one at model_checkpoint.pt. I wonder why the link has two model checkpoints, of equal size, one of them named softprompt-gemma_3_270m-ecg.pt?
Let me know if you need anything else!
All the best - Martin

[mltrinhammer]

Hi @RealLast
I am taking the liberty of checking in again here with regards to the above item.
I would really like to use your Flamingo-ECG model for an upcoming publication, but I find it cumbersome to keep the max_patches at 1024 when the architecture is built for 2600.
Any chance you would have a quick look sometime soon?
All the best - Martin

[RealLast]

Hi @mltrinhammer, thank you for following up!

I looked into this again and can confirm that all released ECG models were trained with max_patches = 1024, which, as I confirmed with my colleagues, is also intended.
To clarify the history: at an earlier stage of development, we indeed had max_patches set to a higher value to support 500 Hz ECG data. For these we would require 10 s × 500 Hz / 4 = 1250 patches

However, before final training and release, we switched to 100Hz ECG data, requiring 10s x 100Hz / 4 = 250 patches, fitting well within the 1024. The released checkpoints were therefore trained consistently with this lower value.

TL;DR: Sorry for the confusion, the checkpoints should be correct, and you don't need to change the max_patches parameter for 100Hz ECG. Don't know why I came up with 2600. If anything, we would need to set max_patches = 1250 for the 500Hz ECG samples, which is also the value I could find in some older checkpoints where we still trained on 500Hz data.

[mltrinhammer]

No problem, thanks for checking!