Performance reproduction issue

[Minsoo2022]

Hi, and thanks for the great release. With the released paper weights, I can reproduce the reported HEST-Bench numbers on my side. However, when I train from scratch using the repo defaults (with the provided filtering), my scores are noticeably lower than in the paper. Could you help confirm which non-default training settings are required to reach the paper numbers?

Questions

1. Best checkpoint metric - Did the paper select lowest val MSE (vs. Pearson)? The script picks best by Pearson; should I switch to val MSE for faithful reproduction?

2. GPU/batch -Your paper notes an 8-GPU environment—did training actually use all 8 GPUs with batch size=2 per GPU (effective 16), or a single GPU with global batch size=2?

3. Species filter - For pretraining, is human-only the intended setting (my slide count aligns with the paper when using human-only), or should mouse be included as well?

4. Validation split - The code uses utils_data/val_sample.txt, which seems HEST-1k-only; since pretraining uses HEST + STImage1K4M, did you validate only on HEST-1k, or also hold out from STImage1K4M (if so, how were those IDs/counts constructed)?

5. Anything else to reproduce weights - Beyond the defaults, are there any required non-default settings or preprocessing steps needed to reproduce the released weights?

[huangtinglin]

Thanks for your interest. Are you using Gigapath as the image encoder? Could you also let me know the results when training from scratch?

For the questions:

1. I used Pearson as the evaluation metric. We tend not to use MSE since this metric can be biased by different normalization strategies.
2. Single GPU with batch size = 2
3. Human only.
4. Yeah, all the samples in the validation set are from HEST.
5. How many epochs did you train the model for? I think it might be the case that the model is not trained sufficiently, so you can try to increase the early-stopping epoch.

I will also double-check the code to see if there is anything missing.

[Minsoo2022]

Thanks so much for the quick reply!

Yes—I'm using Gigapath as the image encoder. From-scratch results on my side (single GPU, batch size = 2, human only, validation = HEST only) are:

epoch: 50, avg_loss: 0.195, pearson_mean 0.1669584740679628
epoch: 100, avg_loss: 0.169, pearson_mean 0.18220077291357614
epoch: 150, avg_loss: 0.163, pearson_mean 0.1939311971576327
epoch: 200, avg_loss: 0.154, pearson_mean 0.2069965821194429
epoch: 250, avg_loss: 0.153, pearson_mean 0.21366051405758688
epoch: 300, avg_loss: 0.146, pearson_mean 0.22696399492255295

I also suspected early stopping was too aggressive, so I increased patience and am training longer. That said, at 100 epochs I’m still below the paper’s validation Pearson (~0.266 vs 0.182 here), and even at 300 epochs I’m around 0.227.

If you notice anything missing or any non-default toggles I should enable, I’d really appreciate pointers. I’ll keep running more experiments and share updates—thanks again!

[Minsoo2022]

Thanks again for the quick follow-up—really appreciate it!

A couple of updates and a hypothesis I’d like your help with:

With Gigapath and your defaults (single GPU, batch=2, human-only, HEST-only validation), I now reach validation Pearson > 0.27 around epoch 1000.
However, evaluating that checkpoint on HEST-Bench is still ~10% lower than the paper’s numbers.
Also, my validation trajectory does not resemble Fig. 2(f); I don’t see the same steady rise.

Preprocessing/training follows the repo. The only deviations are:

Removed Human_Brain+Kidney_10X_02212023_Visium from STImage1K4M (reported as Mouse mislabeled as Human on HF).

Removed NCBI865 from HEST due to barcode mismatch.
After this, I have 983 WSIs total. I used your utils_data/val_sample.txt (28 IDs) for validation and 955 for training.

The paper states: “excluded downstream samples, resulting in 983 WSIs… we set aside 5% for validation, leaving 928 WSIs for pretraining.”
This makes me suspect my validation split might not match the one you actually used in the paper.

Could you please confirm:

The exact validation split used in pretraining (ID list and count). Is it the same as utils_data/val_sample.txt, or a different 5% (~49 slides)?

Whether the two removals above (mis-annotated STImage sample, HEST NCBI865) are expected in your pipeline.

The precise exclusion-list versions you used. If they differ from the repo, could you share them (or a checksum)?

(If feasible) Could you try a from-scratch run with the current repo to confirm it reproduces your paper curve and HEST-Bench scores?

Happy to share my logs, the exact IDs I’m training/validating on, and the HEST-Bench eval outputs if helpful. Thanks again!

[huangtinglin]

For the validation set, I just used the samples shared in the repo, and the NCBI865 is indeed removed due to the mismatched barcode. Human_Brain+Kidney_10X_02212023_Visium wasn't removed in my experiments.

I suspect that the 10% performance drop you mentioned may be due to performance on individual datasets. The model’s performance can fluctuate across different datasets at different checkpoints (you might try evaluating several checkpoints to observe this). In my experiments, the maximum number of epochs was 500.

Regarding the validation curve, I personally do not recommend chasing an identical trajectory, since differences in experimental environments can also introduce bias. What matters more is that the curve converges, and that the final performance it converges to is comparable. I will also double-check my training code, as something may have been missed when I cleaned up the code.

[Minsoo2022]

Thanks again for the guidance!

I’ve now trained up to 1,500 epochs, saving a checkpoint every 50 epochs and evaluating each on HEST-Bench. The best I obtain is Pearson ≈ 0.337, which is still below the ≈0.37 from your released weights.

If the hyperparameters are aligned, my guess is that something may have been lost during the code clean-up. If you could double-check the training code, I’d really appreciate it. I’m happy to test any patch or settings you suggest, and I’ll look out for updates. Thanks again!

[huangtinglin]

Sorry about the late reply. I have been quite occupied recently. I will make some updates on the model implementations. Stay tuned.

[DaIGaN2019]

@Minsoo2022
Hi！Could you tell me where you found the file utils_data/val_sample.txt? I didn't find this file in the utils_data directory.

[Minsoo2022]

@DaIGaN2019 @huangtinglin
Hi—thanks for pointing this out. I initially thought utils_data/val_sample.txt was checked into the repo, but in my runs it was auto-generated by app/pretrain/train.py via random 5% sampling when the file doesn’t exist (see the --val_sample_path logic). This randomness may explain part of the reproducibility gap with the released weights. It would be very helpful if the authors could share the exact val_sample.txt (ID list) used in their experiments. Thanks again!

[DaIGaN2019]

Thank you for your prompt reply. @Minsoo2022