A Governed, Reproducible Pipeline for LLM-Generated Radiology Learning Content
MeducAI is a research pipeline for systematically, reproducibly, and auditably generating radiology learning content (e.g., Anki cards, tables, visual materials) using large language models (LLMs).
The core goal of this project is not content generation itself, but rather to achieve fair model comparison (QA), IRB-friendly documentation, and reproducible execution while maintaining clear definitions of what each stage can and cannot decide.
MeducAI consists of clearly separated stages (S0–S6, FINAL):
S0 : QA / Model Comparison (fixed payload, arm fairness)
S1 : Structure only (LLM, no counts)
S2 : Execution only (LLM, exact N cards)
S3 : Selection & QA gate (state-only)
S4 : Rendering & presentation (image only)
S5 : Validation & Triage (multi-agent review, regeneration decisions)
S6 : Positive Instruction (visual regeneration with feedback)
FINAL: Allocation & deployment (card counts decided here only)
Core principles:
- LLMs do not make "decisions".
- Card counts, selections, and policies are determined only in code and Canonical documents.
- Each stage has unique responsibilities, and boundary violations result in immediate failure (Fail-Fast).
This repository is not simply a code repository, but a "Constitutional" repository for research decision-making.
- For each concept, only one final document exists at any time
- Previous versions are not deleted but archived and frozen
- Code, experiments, papers, and IRB explanations always reference Canonical documents as the source of truth
- Level 0–1: Pipeline constitution, Fail-Fast / Abort rules
- Level 2: Step (S0–S4) roles, boundaries, and card count contracts
- Level 3: Implementation details, reference documents (for reference only)
This structure enables external reviewers (IRB, peer reviewers) to verify "why this design was chosen" by reading documents alone, without examining code.
This repository intentionally excludes the following:
- ❌ Learner personal information or raw data
- ❌ Post-hoc adjusted analysis results
- ❌ LLM chain-of-thought exposure
- ❌ Logic relying on implicit rules or verbal agreements
- ❌ Ad-hoc workarounds to "make things work"
In other words, "explainable research design" takes priority over "working demos".
MeducAI was designed from the outset with IRB, auditing, and reproducibility as prerequisites.
- Fail-Fast & Abort Policy Error handling scope (RUN / ARM / GROUP / SET) is fixed in advance
- Runtime Artifact Manifest Each execution (RUN_TAG) mechanically records "what should be generated for a normal run"
- Fairness guarantee for Arm comparison RAG / Thinking must be verified with actual execution + metadata recording, not just labels, before comparison is allowed
- Unified card count decision authority S0 uses fixed payload; only FINAL decides quotas
This ensures Methods / Supplement / IRB response text is guaranteed by pre-existing documents, not post-hoc.
This README is written for the following audiences:
- Supervisors / Co-authors: "What has this research fixed, and what is it comparing?"
- IRB reviewers: "To what extent do LLMs participate, and how is control maintained?"
- External reviewers: "Is reproducibility and experimental fairness demonstrated through documentation?"
This single document is designed so that reading it alone allows understanding of the project's philosophy, boundaries, and safety mechanisms.
MeducAI is not about letting LLMs generate content freely. It is about proving—document by document—that they are used safely, fairly, and reproducibly.
💡 Update Log (Latest: v1.3) Live update history. Please refer to the latest version of the materials.
- v1.3.1 (Current)
2026/01/09- [Organization] Repository structure consolidated and cleaned up (translation workflow completed; Paper 3 communications separated)
- Translation workflow scripts archived (
3_Code/archived/translation_workflow_2026-01-07/) - Paper 3 communication documents separated (
0_Protocol/06_QA_and_Study/Communications/) - README files added to three handoff document locations
- S5/S6 pipeline stages added
- v1.3
2026/01/08- [Added] Print-ready PDF files added (4-volume split set, including print-ordering guidance)
- Split PDF files prepared for print ordering
- Cover design, page counts, and included specialist information provided for each volume
- See
6_Distributions/MeducAI_Final_Share/PDF/print_ready/README.mdfor detailed print-order instructions
- v1.2
2025/01/08 01:00- [Improved] Medical terminology in the Anki deck changed to English only for clarity and professional consistency
- v1.1
2025/01/07 13:00- [Updated] PDF page order rearranged
- [UI] Improved line breaks and layout readability in Anki deck cards
- v1.0
2025/01/07 12:00- Initial release
Research portfolio consolidation:
- ✅ Established a 3-paper research portfolio structure
- ✅ Updated
Study_Design.mdto v5.0 for the 3-paper framework - ✅ Created the
MeducAI_3Paper_Research_Index.mdmaster index - ✅ Expanded the Visual Modality Sub-study to include resident realistic-image evaluation
- ✅ Created the Table Infographic Evaluation Plan with a 5-rater design
3-paper structure:
| Paper | Title | Data source |
|---|---|---|
| Paper 1 | Reliability of the S5 multi-agent review and rewriting system | FINAL QA (1,350 resident + 330 specialist evaluations) |
| Paper 2 | Reliability of MLLM-generated images | Visual Modality Sub-study + Table Infographic |
| Paper 3 | Prospective observational study of educational effectiveness | Baseline + FINAL survey (IRB-approved) |
Document organization:
- 91 operational and temporary documents archived (HANDOFF, S0_QA, AppSheet configuration, etc.)
- Only core canonical documents retained
- Paper-specific document guidance added to
DOCS_REGISTRY.md
Key documents:
0_Protocol/06_QA_and_Study/MeducAI_3Paper_Research_Index.md(master index)0_Protocol/06_QA_and_Study/Paper3_Educational_Effectiveness/Paper3_Study_Design.md0_Protocol/06_QA_and_Study/Paper1_S5_Validation/Paper1_Paper2_Research_Design_Spec.md0_Protocol/06_QA_and_Study/Paper2_Image_Reliability/Paper2_Table_Infographic_Evaluation_Design.md
Repository consolidation completed:
- ✅ Translation workflow scripts archived (
3_Code/archived/translation_workflow_2026-01-07/) - ✅ Medical term English-only policy implemented and completed
- ✅ Paper 3 communications separated (
0_Protocol/06_QA_and_Study/Communications/) - ✅ Handoff documentation indexed (3 locations with READMEs)
- ✅ Pipeline overview updated to include S5/S6 stages
- ✅ Main documentation updated for v1.3.1
Key organizational changes:
- 27 translation/debugging scripts moved to archive with comprehensive README
- Paper 3 emails and announcements organized in dedicated Communications folder
- Handoff folders now include scope documentation and cross-references
- Production Anki/AppSheet export scripts remain in active tools
Language policy:
- Medical terms now English-only across all Anki decks and AppSheet exports
- Sentence structure and formatting preserved during translation
- Applied consistently to baseline and regenerated cards
Reference documents:
0_Protocol/CONSOLIDATION_SUMMARY_2026-01-09.md(consolidation summary)3_Code/archived/translation_workflow_2026-01-07/README.md(archived scripts)0_Protocol/06_QA_and_Study/Communications/README.md(communications index)0_Protocol/01_Execution_Safety/handoffs/HANDOFF__MEDTERM_ENGLISH_ONLY__S2_APPSHEET_ANKI__2026-01-07.md
Repository organization completed:
- ✅ Legacy metadata/code archived (
2_Data/metadata/legacy/,3_Code/archived/) - ✅ Protocol root documents consolidated into
00_Governance/ - ✅ Scripts/Configs/Notebooks organized (moved to
legacy/folders) - ✅ Tools migration completed (
3_Code/src/tools/) - ✅ System files removed (
.DS_Store, etc.) - ✅
9_Future_Work/added to.gitignore
Key organizational changes:
- All Protocol summary/analysis documents moved to
0_Protocol/00_Governance/ - Legacy scripts moved to
3_Code/Scripts/legacy/ - Utility scripts migrated to
3_Code/src/tools/and3_Code/src/tools/qa/ - Legacy configs moved to
3_Code/configs/legacy/ - Legacy notebooks moved to
3_Code/notebooks/legacy/
Git milestones:
- Frozen tag:
protocol-freeze-v1.1(2025-12-22) - Default branch:
protocol-freeze-main
Reference documents:
0_Protocol/00_Governance/archived/LEGACY_AND_CLEANUP_DOCUMENTS_INDEX.md(archived)0_Protocol/00_Governance/archived/LEGACY_CLEANUP_SUMMARY.md(archived)0_Protocol/00_Governance/archived/LEGACY_ARCHIVE_SUMMARY.md(archived)
What changed:
- ✅ Robust PDF parsing that recovers wrapped lines / page breaks / deep nesting (notably pediatric sections)
- ✅ Deterministic LLM steps using
gemini-3-flash-preview(translation/enrichment) with MI‑CLEAR‑LLM run logs - ✅ Text normalization for PDF artifacts (e.g., split English tokens like
R enal→Renal) - ✅ Versioned SSOT outputs ready for S1–S4 with no schema changes required
Key outputs (v2):
2_Data/processed/Radiology_Curriculum_Weight_Factor_v2.xlsx(SSOT)2_Data/metadata/groups_canonical_v2.csv(+.sha256,.meta.json)2_Data/metadata/translation_map_v2.json(expanded coverage)- MI‑CLEAR logs:
2_Data/processed/logs/<run_id>.jsonland<run_id>.system_prompts.txt
How to run (v2 upstream pipeline):
3_Code/src/preprocess/run_pipeline_v2.py(end-to-end)
Legacy note (S0 interpretation vs future runs):
- Keep v1 snapshots under
2_Data/metadata/legacy/and2_Data/processed/legacy/for S0 interpretability. - Promote v2 artifacts to the default names only when you intentionally switch the operational SSOT.
Full document organization completed:
- ✅ All subfolder documents organized and structured
- ✅ Conflict documents resolved and marked as Historical Reference
- ✅ Superseded documents clearly marked
- ✅ Filename cleanup completed (
Prompt_Rendering_Safety_Rule.md) - ✅ Document status classification clarified (Canonical / Reference / Historical Reference)
Major cleanup work:
00_Governance/: Document status clarified, relationships organized01_Execution_Safety/: Filename cleanup, duplicate removal02_Arms_and_Models/: Superseded documents clarified03_CardCount_and_Allocation/: Experimental documents clarified04_Step_Contracts/: Superseded documents marked05_Pipeline_and_Execution/: README updated, deprecated items marked06_QA_and_Study/: S0 Non-Inferiority policy unified
Cleanup summary documents:
- Cleanup summary documents archived to respective
archived/folders (2026-01-04)
S3 (Policy Resolver & ImageSpec Compiler):
- Q2 image policy change: Q2 now also requires images (
image_required = True) - S1 table visual spec added: Group-level table/visual image generation support
- Prompt improvements: Enhanced image generation quality by including card text (front/back) and extracted answers
- Answer extraction logic: Q1 uses "Answer:" parsing, Q2/Q3 use
correct_index-based extraction
S4 (Image Generator):
- Image generation model changed:
models/nano-banana-pro-preview(Gemini 3 Pro Image Preview) - Spec type branching: Card images (4:5, 1K) vs table visuals (16:9, 2K)
- Fail-fast extended: Q2 and table visuals also treated as required images
- Image extraction logic improved: PNG header validation and debugging enhanced
Pipeline tools:
run_6arm_s1_s2_full.py: Added--armsoption (select specific arms only)- Report storage location changed:
2_Data/metadata/generated/{run_tag}/directory check_models.py: Updated togoogle.genaiSDK
Documentation:
S3_S4_Code_Documentation.md: Complete code behavior documentation0_Protocol/04_Step_Contracts/Step03_S3/S3_Implementation_Update_Log_2025-12-20.md0_Protocol/04_Step_Contracts/Step04_S4/S4_Implementation_Update_Log_2025-12-20.md0_Protocol/05_Pipeline_and_Execution/Implementation_Update_Log_2025-12-20.md
Detailed information:
- Implementation log:
0_Protocol/00_Governance/archived/Implementation_Change_Log_2025-12-20.md(archived) - Code documentation:
0_Protocol/04_Step_Contracts/S3_S4_Code_Documentation.md
03_CardCount_and_Allocation/Experimental/S0_STABILIZE_MULTI_Allocation_Artifact_Spec.md01_Execution_Safety/stabilization/S2_Stabilization_Plan_and_Preflight_Checks.md
MeducAI/
├── 0_Protocol/ # [READ-ONLY] Canonical Protocols & Governance
│ ├── 00_Governance/ # Governance documents, cleanup summaries
│ ├── 01_Execution_Safety/ # Safety rules and policies
│ ├── 02_Arms_and_Models/ # Arm configurations and model specifications
│ ├── 03_CardCount_and_Allocation/ # Card count policies
│ ├── 04_Step_Contracts/ # Step contracts (S1-S4)
│ ├── 05_Pipeline_and_Execution/ # Pipeline execution plans
│ └── 06_QA_and_Study/ # QA framework and study design
├── 1_Secure_Participant_Info/ # [RESTRICTED] PII, Consent Forms (IRB)
├── 2_Data/ # Data and metadata
│ ├── metadata/
│ │ ├── generated/ # Runtime artifacts (excluded from git)
│ │ └── legacy/ # Archived legacy metadata
│ └── processed/ # Processed curriculum data (SSOT)
├── 3_Code/ # Source Code & Pipelines
│ ├── src/ # Main source code
│ │ └── tools/ # Utility tools and QA scripts
│ ├── Scripts/ # Execution scripts
│ │ └── legacy/ # Archived scripts
│ ├── configs/ # Configuration files
│ │ └── legacy/ # Archived configs
│ └── notebooks/ # Jupyter notebooks
│ └── legacy/ # Archived notebooks
└── 6_Distributions/ # QA Packages (Blinded/Unblinded)
└── MeducAI_Final_Share/
└── PDF/
└── print_ready/ # print-ready PDFs (4-volume split set with ordering guide)
- Python 3.8+
- GitHub account (for private repository access)
- Access credentials for LLM providers (Gemini, OpenAI, etc.)
-
Clone the repository:
git clone https://github.com/Yoojin-nam/MeducAI.git cd MeducAI -
Install dependencies:
pip install -r requirements.txt
-
Configure environment:
cp .env.example .env # Edit .env with your API keys -
Review Canonical documents:
- Start here:
0_Protocol/README.md - Start with:
0_Protocol/00_Governance/meduc_ai_pipeline_canonical_governance_index.md - Check operating status:
0_Protocol/01_Execution_Safety/stabilization/Weekly_Integrated_Conclusion_Operating_SSOT.md
- Start here:
0_Protocol/README.md- Start-here reading order (minimal set)0_Protocol/00_Governance/meduc_ai_pipeline_canonical_governance_index.md- Canonical document hierarchy0_Protocol/05_Pipeline_and_Execution/Pipeline_Canonical_Specification.md- Pipeline philosophy0_Protocol/05_Pipeline_and_Execution/Pipeline_Execution_Plan.md- Execution plan
0_Protocol/06_QA_and_Study/MeducAI_3Paper_Research_Index.md- master index for the 3-paper research program0_Protocol/06_QA_and_Study/Paper1_S5_Validation/QA_Framework.md- QA framework0_Protocol/06_QA_and_Study/Paper1_S5_Validation/Paper1_Paper2_Research_Design_Spec.md- research design for Papers 1 and 20_Protocol/06_QA_and_Study/Paper3_Educational_Effectiveness/Paper3_Study_Design.md- research design for Paper 3
6_Distributions/MeducAI_Final_Share/PDF/print_ready/- print-ready PDFs (4-volume split set)- Split PDF files for print ordering
- Cover design, page counts, and included specialist information for each volume
- Detailed print-order instructions and email templates
- See
print_ready/README.mdfor details
Paper 1 (S5 multi-agent reliability):
0_Protocol/06_QA_and_Study/Paper1_S5_Validation/Paper1_Paper2_Research_Design_Spec.md0_Protocol/06_QA_and_Study/Paper1_S5_Validation/Paper1_QA_Endpoints_Definition.md0_Protocol/05_Pipeline_and_Execution/S5_Decision_Definition_Canonical.md
Paper 2 (MLLM image reliability):
0_Protocol/06_QA_and_Study/Paper1_S5_Validation/Paper1_Paper2_Research_Design_Spec.md(Section 8: Visual Modality)0_Protocol/06_QA_and_Study/Paper2_Image_Reliability/Paper2_Table_Infographic_Evaluation_Design.md0_Protocol/06_QA_and_Study/Paper2_Image_Reliability/Paper2_Visual_Turing_Test_Design_Detailed.md
Paper 3 (prospective educational-effectiveness study):
0_Protocol/06_QA_and_Study/Paper3_Educational_Effectiveness/Paper3_Study_Design.md0_Protocol/06_QA_and_Study/Paper3_Educational_Effectiveness/Paper3_Survey_Overview.md0_Protocol/06_QA_and_Study/Paper3_Educational_Effectiveness/Paper3_Statistical_Analysis_Plan.md
0_Protocol/00_Governance/supporting/Prompt_governance/Prompt_Engineering_and_Cognitive_Alignment.md- Prompt governance SSOT (Methods-ready + change control)3_Code/prompt/_registry.json- Current active prompt bundle (code truth)
0_Protocol/IRB/README.md- IRB documentation index0_Protocol/01_Execution_Safety/Prompt_Rendering_Safety_Rule.md- Prompt safety rules
This is a private research repository. Access is limited to authorized collaborators.
For co-authors and collaborators:
- Contact the repository owner for access
- Review the Canonical documents before making changes
- Follow the Fail-Fast policy and document any modifications
- Update relevant documentation when making protocol changes
This project is part of a research study. All rights reserved.
For questions about the pipeline, protocol, or access requests, please contact the principal investigator.
Last Updated: 2026-03-30
Frozen Tag: protocol-freeze-v1.3
Default Branch: protocol-freeze-main