Merge pull request #95 from pythonhealthdatascience/dev

Dev

Author: amyheather

README.md

@@ -10,10 +10,18 @@
 [![Coverage](https://github.com/pythonhealthdatascience/pydesrap_mms/raw/main/images/coverage-badge.svg)](https://github.com/pythonhealthdatascience/pydesrap_mms/actions/workflows/tests.yaml)
 </div>
 
+<br>
+
+This repository is an example accompanying the [**DES RAP Book**](https://github.com/pythonhealthdatascience/des_rap_book) — an open educational resource on reproducible discrete-event simulation (DES) in Python and R. The book demonstrates best practices for building, validating, and sharing DES models within a reproducible analytical pipeline (RAP). The `pydesrap_mms` model illustrates how these principles can be applied to a simple queueing model.
+
+<br>
+
 ## Repository overview
 
 This repository provides a reproducible analytical pipeline (RAP) for a simple **M/M/s queuing model** implemented in Python using SimPy. The model simulates patients arriving, waiting to see a nurse, being served, and leaving. All code is structured as a local Python package.
 
+![](images/nurse_des.drawio.png)
+
 An M/M/s queueing model is a classic mathematical model for systems where:
 
 * Arrivals happen at random, following a Poisson process - and the time between arrivals follows an exponential distribution (the first "M", which stands for "Markovian" as it is memoryless - arrivals are independent).
@@ -24,7 +32,9 @@ This type of model is widely used for studying waiting lines in healthcare, call
 
 <br>
 
-## Installation
+## Usage and reproduction instructions
+
+<details><summary><b>Installation</b></summary>
 
 Clone the repository:
 
@@ -42,9 +52,11 @@ conda activate
 
 There is also a `requirements.txt` file which can be used to set up the environment with `virtualenv`, but this won't fetch a specific version of Python - so please note the version listed in `environment.yaml`.
 
+</details>
+
 <br>
 
-## How to run
+<details><summary><b>How to run</b></summary>
 
 The simulation code is in the `simulation/` folder as a local package. Example analyses and model runs are in `notebooks/`.
 
@@ -87,48 +99,12 @@ To run one notebook from the command line (with the same settings - clearing the
 bash run_notebooks.sh notebooks/notebook_name.ipynb
 ```
 
-<br>
-
-## How does the model work?
-
-This section describes the purposes of each class in the simulation.
-
-**Model Run Process:**
-
-1. **Set Parameters:** Create a `Param` instance with desired model parameters.
-2. **Initialise Model:** Instantiate `Model` using the parameters. During setup, `Model` creates `Exponential` instances for each distribution.
-3. **Run Simulation:** Call `model.run()` to execute the simulation within the SimPy environment, running two processes:
-
-    * `generate_patient_arrivals()` to handle patient creation, then sending them on to `attend_clinic()`.
-    * `interval_audit()` to record utilisation and wait times at specified intervals during the simulation.
 
-**Runner Class Usage:**
-
-Having set up `experiment = Runner()`...
-
-* **Single Run:** Use `experiment.run_single()` to execute a single model run.
-* **Multiple Runs:** Use `experiment.run_reps()` to perform multiple replications of the model.
+</details>
 
 <br>
 
-![Model structure diagram](images/model_structure.png)
-
-*Illustration of model structure created using [draw.io](https://draw.io/).*
-
-<br>
-
-## Reproducing results
-
-To generate the figures and tables from the paper (`mock_paper.md`), execute:
-
-* **Figures 1-4**: `notebooks/analysis.ipynb`
-* **Figures A.1-A.2**: `notebooks/input_modelling.ipynb`
-* **Figure B.1**: `notebooks/choosing_warmup.ipynb`
-* **Figures C.1-C.3**: `notebooks/choosing_replications.ipynb`
-
-<br>
-
-## Input data
+<details><summary><b>Input data</b></summary>
 
 **Patient-level data** for our system is provided in the file: `inputs/NHS_synthetic.csv`.
 
@@ -145,74 +121,50 @@ This dataset is released under the MIT licence. If you use this data, please cit
 
 The code for input modelling is in: `notebooks/input_modelling.ipynb`. Model parameters are determined in this file and then stored in: `simulation/model.py`. Description for each parameter can be found in the class docstring within this file.
 
-<br>
+</details>
 
-## GitHub actions
-
-GitHub actions in `.github/workflows/` automate testing and code checks.
+<br>
 
-* **tests.yaml** runs the tests on Ubuntu, Windows, and Mac after each push to main.
-* **lint.yaml** checks code style in python scripts and .ipynb files to maintain code quality.
+<details><summary><b>Reproducing results</b></summary>
 
-<br>
+To generate the figures and tables from the paper (`mock_paper.md`), execute:
 
-## Repository structure
+* **Figures 1-4**: `notebooks/analysis.ipynb`
+* **Figures A.1-A.2**: `notebooks/input_modelling.ipynb`
+* **Figure B.1**: `notebooks/choosing_warmup.ipynb`
+* **Figures C.1-C.3**: `notebooks/choosing_replications.ipynb`
 
-```
-repo/
-├── .github/workflows/    # GitHub actions
-├── docs/                 # Documentation
-├── images/               # Image files and GIFs
-├── inputs/               # Folder to store any input data
-├── notebooks/            # Run DES model and analyse results
-├── outputs/              # Folder to save any outputs from model
-├── simulation/           # Local package containing code for the DES model
-├── tests/                # Unit and back testing of the DES model
-├── .gitignore            # Untracked files
-├── .pylintrc             # Pylint settings
-├── CHANGELOG.md          # Describes changes between releases
-├── CITATION.cff          # How to cite the repository
-├── CONTRIBUTING.md       # Contribution instructions
-├── environment.yaml      # Conda environment (includes Python version)
-├── LICENSE               # Licence file
-├── lint.sh               # Bash script to lint all .py and .ipynb files at once
-├── pyproject.toml        # Metadata for local `simulation/` package
-├── README.md             # This file! Describes the repository
-├── requirements.txt      # Virtual environment (used by GitHub actions)
-└── run_notebooks.sh      # Bash script to run all .ipynb from the command line
-```
+</details>
 
 <br>
 
-## Run time and machine specification
+<details><summary><b>Run time and machine specification</b></summary>
 
 Run times from our analyses (on Intel Core i7-12700H, 32GB RAM, Ubuntu 24.04.1):
 
-* `analysis.ipynb` - 23s
-* `choosing_cores.ipynb` - 19s
-* `choosing_replications.ipynb` - 33s
-* `choosing_warmup.ipynb` - 4s
-* `generate_exp_results.ipynb` - 7s
+* `analysis.ipynb` - 35s
+* `choosing_cores.ipynb` - 34s
+* `choosing_replications.ipynb` - 46s
+* `choosing_warmup.ipynb` - 38s
+* `generate_exp_results.ipynb` - 1s
 * `logs.ipynb` - 0s
-* `time_weighted_averages.ipynb` - 1s
-
-<br>
+* `time_weighted_averages.ipynb` - 2s
 
-## Community
-
-Curious about contributing? Check out the [contributing guidelines](CONTRIBUTING.md) to learn how you can help. Every bit of help counts, and your contribution - no matter how minor - is highly valued.
+</details>
 
 <br>
 
-## Citation
+## Project details and credits
+
+### How to cite the repository
 
 If you use this repository, please cite either the GitHub repository or Zenodo:
 
 > Heather, A. Monks, T. (2025). Simple M/M/s queuing model: Python DES RAP. GitHub. https://github.com/pythonhealthdatascience/pydesrap_mms.
 >
 > Heather, A. Monks, T. (2025). Simple M/M/s queuing model: Python DES RAP. Zenodo. https://doi.org/10.5281/zenodo.14622466
 
-**Contributors:**
+### Contributors
 
 **Amy Heather** - developed the repository.
 
@@ -224,15 +176,15 @@ If you use this repository, please cite either the GitHub repository or Zenodo:
 * [![ORCID](https://img.shields.io/badge/ORCID-0000--0003--2631--4481-A6CE39?style=for-the-badge&logo=orcid&logoColor=white)](https://orcid.org/0000-0003-2631-4481)
 * [![GitHub](https://img.shields.io/badge/GitHub-TomMonks-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/TomMonks)
 
-<br>
-
-## Licence
+### Licence
 
 MIT Licence. See `LICENSE` for details.
 
-<br>
+### Community
+
+Curious about contributing? Check out the [contributing guidelines](CONTRIBUTING.md) to learn how you can help. Every bit of help counts, and your contribution - no matter how minor - is highly valued.
 
-## Acknowledgements
+### Acknowledgements
 
 This repository was developed with thanks to several others sources. These are acknowledged throughout in the relevant notebooks/modules/functions, and also summarised here:
 
@@ -243,11 +195,9 @@ This repository was developed with thanks to several others sources. These are a
 | Sammi Rosser and Dan Chalk (2024) HSMA - the little book of DES (https://github.com/hsma-programme/hsma6_des_book) (MIT Licence) | `simulation/model.py`<br>`simulation/patient.py`<br>`simulation/runner.py`<br>`notebooks/choosing_cores.ipynb` |
 | Tom Monks (2025) sim-tools: tools to support the Discrete-Event Simulation process in python (https://github.com/TomMonks/sim-tools) (MIT Licence)<br>Who themselves cite Hoad, Robinson, & Davies (2010). Automated selection of the number of replications for a discrete-event simulation (https://www.jstor.org/stable/40926090), and Knuth. D "The Art of Computer Programming" Vol 2. 2nd ed. Page 216. | `simulation/confidence_interval_method.py`<br>`simulation/onlinestatistics.py`<br>`simulation/plotly_confidence_interval_method.py`<br>`simulation/replicationsalgorithm.py`<br>`simulation/replicationtabulizer.py`<br>`notebooks/choosing_replications.ipynb` |
 | Tom Monks, Alison Harper and Amy Heather (2025) An introduction to Discrete-Event Simulation (DES) using Free and Open Source Software (https://github.com/pythonhealthdatascience/intro-open-sim/tree/main). (MIT Licence) - who themselves also cite Law. Simulation Modeling and Analysis 4th Ed. Pages 14 - 17. | `simulation/monitoredresource.py` |
-| Tom Monks (2024) [HPDM097 - Making a difference with health data](https://github.com/health-data-science-OR/stochastic_systems) (MIT Licence). | `notebooks/analysis.ipynb`<br>`notebooks/choosing_replications.ipynb`<br>`notebooks/choosing_warmup.ipynb` |
+| Tom Monks (2024) [HPDM097 - Making a difference with health data](https://github.com/health-data-science-OR/stochastic_systems) (MIT Licence). | `simulation/warmupauditor.py`<br>`notebooks/analysis.ipynb`<br>`notebooks/choosing_replications.ipynb`<br>`notebooks/choosing_warmup.ipynb` |
 | Monks T and Harper A. Improving the usability of open health service delivery simulation models using Python and web apps (https://doi.org/10.3310/nihropenres.13467.2) [version 2; peer review: 3 approved]. NIHR Open Res 2023, 3:48.<br>Who themselves cite a [Stack Overflow](https://stackoverflow.com/questions/59406167/plotly-how-to-filter-a-pandas-dataframe-using-a-dropdown-menu) post. | `notebooks/analysis.ipynb` |
 
-<br>
-
-## Funding
+### Funding
 
 This project was developed as part of the project STARS: Sharing Tools and Artefacts for Reproducible Simulations. It is supported by the Medical Research Council [grant number [MR/Z503915/1](https://gtr.ukri.org/projects?ref=MR%2FZ503915%2F1)].

docs/hsma_changes.md

@@ -170,7 +170,7 @@ In the template, results are instead saved as a dictionary into a list as the ru
 * Avoids initialising an empty dataframe.
 * Is consistent with how the patient-level results were also generated in the template.
 
-Also, some of the calculations have been performed directly during the `run_single()` method, instead of from a seperate method `calculate_run_results()`. This is to help simplify the code, as it makes clear how each metric was calculated in one place, rather than needing to refer elsewhere.
+Also, some of the calculations have been performed directly during the `run_single()` method, instead of from a separate method `calculate_run_results()`. This is to help simplify the code, as it makes clear how each metric was calculated in one place, rather than needing to refer elsewhere.
 
 ```
 run_results = {

docs/quality_assurance.md

@@ -0,0 +1,31 @@
+# Quality Assurance (QA)
+
+## Roles and responsibilities
+
+**Analyst:** Amy Heather (developed the model and implemented the analysis).
+
+**Assurer/Approver:** Tom Monks (provided independent review proportionate to a small teaching example).
+
+## QA when scoping the project
+
+Scoping was agreed verbally between analyst and assurer: construct an M/M/s queueing model in SimPy using synthetic data as a worked example, rather than a decision‑critical tool. No formal written scoping document was produced, which would be expected for larger or higher‑risk work, but was judged unnecessary here.
+
+No explicit QA plan (who, what, how much) was written at the outset; this document provides a retrospective QA summary and lessons learned for future projects.
+
+## QA when designing the analysis
+
+The analytical approach was intentionally simple: an M/M/s model implemented in SimPy, using synthetic data consistent, with no external data sources. This constrained the design space, so there were few substantive design choices beyond implementation details.
+
+Verification and validation strategy was not agreed in advance but was developed iteratively during the analysis as issues and checks were identified.
+
+Design decisions were not recorded in a separate design document; for future, non‑toy models, capturing key decisions in a short design or analysis‑plan file would be preferable.
+
+## QA when performing the analysis
+
+Verification and validation activities were carried out during development, and checked in a [summary GitHub issue](https://github.com/pythonhealthdatascience/pydesrap_mms/issues/84) (serving as part of the QA log). Checks were proportionate to the simplicity and teaching purpose of the model.
+
+Assurance of code and workflows was ensured by following the [STARS Reproducibility Recommendations](https://doi.org/10.1080/17477778.2025.2552177) and the [NHS Levels of RAP Framework](https://nhsdigital.github.io/rap-community-of-practice/introduction_to_RAP/levels_of_RAP/).
+
+Code is documented with docstrings and comments to aid understanding and reuse. User and technical documentation are currently provided via the [DES RAP Book](https://github.com/pythonhealthdatascience/des_rap_book). To avoid duplication of this material, the repository does not have it's own standalone user instructions or detailed technical description of the model structure, which would be expected for a model in practice.
+
+GitHub issues act as an informal QA plan and log; for future projects, a more explicit QA project board and short summary of decisions and changes would strengthen the audit trail.

docs/heather_2025.md …stars_reproducibility_recommendations.mddocs/heather_2025.md renamed to docs/stars_reproducibility_recommendations.md docs/heather_2025.md renamed to docs/stars_reproducibility_recommendations.md

@@ -1,6 +1,10 @@
-# Reproducibility recommendations from Heather et al. 2025
+# STARS Reproducibility Recommendations
 
-As part of the project STARS (Sharing Tools and Artefacts for Reproducible Simulations), a series of computational reproducibility assessments were conducted by [Heather et al. 2025](https://doi.org/10.48550/arXiv.2501.13137). From these, several recommendations were shared to support reproducibility of healthcare discrete-event simulation (DES) models. These are copied below. Those marked with a star (⭐) were identified as having the greatest impact in Heather et al. 2025.
+As part of the project STARS (Sharing Tools and Artefacts for Reproducible Simulations), a series of computational reproducibility assessments were conducted and described in:
+
+> Heather, A., Monks, T., Harper, A., Mustafee, N., & Mayne, A. (2025). On the reproducibility of discrete-event simulation studies in health research: an empirical study using open models. Journal of Simulation. https://doi.org/10.1080/17477778.2025.2552177.
+
+From these, several recommendations were shared to support reproducibility of healthcare discrete-event simulation (DES) models. These are copied below. Those marked with a star (⭐) were identified as having the greatest impact in the paper.
 
 ## Recommendations to support reproduction