Reproducibility and Open Science Practices in Machine Learning

A workshop for the AI-DOC doctoral researchers, 11/11/2025, Aalto University
This repository contains the materials used in the workshop and examples related to machine learning reproducibility.

Schedule

• 9:00 || Motivational intro: Fundamentals of reproducibility in Machine Learning (Enrico Glerean)
• 9:20 || Practicalities (EG):
  • https://noppe.2.rahtiapp.fi/welcome -> HAKA login (noppe workspace password given during the workshop)
• 9:45 || Jupyter reproducibility (Luca Ferranti)
• 10:00 || Environment reproducibility (Simo Tuomisto)
  • Motivation: environment reproducibility good [pip install in default bad]
    • Demo:
      • Show environment
      • Create container from environment: https://github.com/simo-tuomisto/micromamba-apptainer
      • Create apptainer in CSC machine https://coderefinery.github.io/hpc-containers/intro_and_motivation/#
• 10:30 || break
• 10:45 How do you track your work / training: MLflow works for trad ML and DL (Hossein Firooz and ST)
  • Why do we track the training process? What are the approaches? WandB, MLFlow (HF)
  • Examples demo with MLflow (ST)
• 11:30 || Overfitting & overreproducing (LF)
• 12:00 || Lunch - on your own
• 13:00 || How do you create reproducible DL training? How do you reproduce the training & creation? Lightning (ST) (30min)
  • Motivation:
    • Dataset
    • Model
    • Trainer
    • CLI: “main()”
    • configuration management
    • checkpointing
  • Example model in PyTorch Lightning
• 13:45 || break
• 14:00 || Model sharing How do you share models? Huggingface (ST)
  • Model card
    • Model parameters
    • Model structure as a code
  • Model weights & used tokenizers
    • Storage formats: safetensors
• 14:15 || Scaling: How do you read how big players are doing it and how do you get there? (HF)
  • How to understand the hardware?
  • DataLoader
    • Parallelism and workers.
    • Sampling and generator
    • Random number handling
  • DP -> DDP -> Model / Tensor Parallel -> Deepspeed
    • Demo on triton
• 14:55 || Outro “What next?”, good reproducibility practices of any research project (ie. the coderefinery workshop). https://coderefinery.github.io/reproducible-research/ (EG)
• 15:00 || The end