reproduce

[wawpaopao]

hello!
I am currently engaged in replicating the C.Origami model and have encountered some difficulties.
I have diligently replicated the training process detailed on GitHub, employing the same datasets and code across two distinct servers, and have done multiple times. I've noticed considerable randomness in the training process, and the weights saved automatically by PyTorchlightning do not seem to effectively predict the Hi-C matrices in the training, validation, and test datasets. Using the checkpoints provided on GitHub, I found that there were no issues with the predictions.
I am curious to know if there have been previous reports of such variability, and if not, I plan to continue repeating my experiments to understand this phenomenon better.
thanks!

[tanjimin]

Hi @wawpaopao , did you use the processed ATAC-seq CTCF ChIP and Hi-C provided from Zenodo https://zenodo.org/record/7226561? If that is correct, you can check your loss curve, it should converge at around 0.15. The original model is trained for about a day on 4 V100s with batch size of 8. You can check your machine setup and make sure you leave sufficient time for training.

[wawpaopao]

I might have some differences in training time. I trained for 60 epochs, so the total batch size would be 8 * 4, right?

[wawpaopao]

So, the total number of samples is approximately 5000. Using 4 NVIDIA RTX 3090 GPUs, it takes about 5 minutes per epoch.

[wawpaopao]

I adjusted the batch_size to 32, and on a single 3090 GPU, I was able to replicate the results fairly well within about 10 epochs. Thank you.However, we can also observe that the batch size has a significant impact on the results.