CPGPrompt - Headache Referral

This project implements a pipeline to convert clinical text guidelines into a structured decision tree, generate synthetic clinical vignettes, and traverse the decision tree to simulate classification or referral suggestions based on the vignettes.

Project Structure

• text_headache_guideline.txt
  The original guideline document describing headache diagnosis and referral criteria in free text.

Pipeline Overview

1. text2tree.py
   Parses and converts the raw guideline text into a set of decision tree substructures.

   • Output: decision_trees_output_full.json
   • Note: The guideline text is split into smaller segments to improve model performance during parsing.

2. merge_tree.py
   Merges all subtrees into a unified structured decision tree.

   • Output: merged_decision_tree.json

3. generate_vignette.py
   Automatically generates clinical vignettes with various combinations of positive and negative features.

   • Output: Multiple JSON files containing vignettes in the vignettes_part/ directory

4. merge_data.py
   Merges all vignette JSON files into a single CSV for easy processing and analysis.

   • Output: vignettes_data.csv

5. traverse_tree.py
   Main execution script that:

   • Loads the merged decision tree
   • Uses each vignette as input
   • Generates prompts for each decision node
   • Traverses the tree according to model decisions
   • Output results are saved in the results/ directory

6. results.ipynb
   Jupyter Notebook for evaluating the outputs. Includes analysis of the traversal and classification performance.