Skip to content

Commit 97ba047

Browse files
Andy-jqaAndy-jqa
committed
updated TrialGPT code
1 parent ab7d9cc commit 97ba047

File tree

23 files changed

+672771
-0
lines changed

23 files changed

+672771
-0
lines changed

‎LICENSE‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
PUBLIC DOMAIN NOTICE
2+
National Center for Biotechnology Information
3+
4+
This software/database is a "United States Government Work" under the terms of
5+
the United States Copyright Act. It was written as part of the author's
6+
official duties as a United States Government employee and thus cannot be
7+
copyrighted. This software/database is freely available to the public for use.
8+
The National Library of Medicine and the U.S. Government have not placed any
9+
restriction on its use or reproduction.
10+
11+
Although all reasonable efforts have been taken to ensure the accuracy and
12+
reliability of the software and data, the NLM and the U.S. Government do not and
13+
cannot warrant the performance or results that may be obtained by using this
14+
software or data. The NLM and the U.S. Government disclaim all warranties,
15+
express or implied, including warranties of performance, merchantability or
16+
fitness for any particular purpose.
17+
18+
Please cite the author in any work or product based on this material:
19+
Qiao Jin, Zifeng Wang, Charalampos S. Floudas, Fangyuan Chen, Changlin Gong, Dara Bracken-Clarke, Elisabetta Xue, Yifan Yang, Jimeng Sun, Zhiyong Lu.
20+
Matching Patients to Clinical Trials with Large Language Models.

‎README.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# TrialGPT: Matching Patients to Clinical Trials with Large Language Models
2+
3+
## Introduction
4+
Clinical trials are often hindered by the challenge of patient recruitment. In this work, we introduce TrialGPT, a novel end-to-end framework for zero-shot patient-to-trial matching with large language models (LLMs). TrialGPT consists of three key components: it first performs filtering of irrelevant clinical trials at scale (TrialGPT-Retrieval), then predicts the patient eligibility on a criterion-by-criterion basis (TrialGPT-Matching); and finally aggregates criterion-level predictions into trial-level scores for ranking the clinical trials (TrialGPT-Ranking). We evaluate TrialGPT on three publicly available cohorts of 183 synthetic patients with over 75,000 trial eligibility annotations. TrialGPT-Retrieval can efficiently recall over 90% of relevant clinical trials using only less than 6% of the initial clinical trial collection. Over 1,000 patient-criterion pairs were manually annotated by three physicians to evaluate TrialGPT-Matching, which achieves a criterion-level accuracy of 87.3% with faithful explanations, close to the expert performance (88.7%–90.0%). For TrialGPT-Ranking, the aggregated trial-level scores are highly correlated with human eligibility judgments, and they outperform the best-competing models by 28.8% to 53.5% in ranking and excluding clinical trials. Furthermore, our user study reveals that TrialGPT can significantly reduce the screening time by 42.6% in a real-life clinical trial matching task. Taken together, these results have demonstrated promising opportunities for clinical trial matching with LLMs via the TrialGPT framework.
5+
6+
![image](https://github.com/ncbi-nlp/TrialGPT-Revision/assets/32558774/3382c2ec-b24b-4ab7-af69-82e0fb2ef511)
7+
8+
## Configuration
9+
10+
To run TrialGPT, one needs to first set up the OpenAI API either directly through OpenAI or through Microsoft Azure. Here we use Microsoft Azure because it is compliant with Health Insurance Portability and Accountability Act (HIPAA). Please set the enviroment variables accordingly:
11+
12+
```bash
13+
export OPENAI_ENDPOINT=YOUR_AZURE_OPENAI_ENDPOINT_URL
14+
export OPENAI_API_KEY=YOUR_AZURE_OPENAI_API_KEY
15+
```
16+
17+
The code has been tested with Python 3.9.13 using CentOS Linux release 7.9.2009 (Core). Please install the required Python packages by:
18+
19+
```bash
20+
pip install -r requirements.txt
21+
```
22+
23+
## Datasets
24+
25+
We used the clinical trial information on https://clinicaltrials.gov/. Please download our parsed dataset by:
26+
27+
```bash
28+
wget -O dataset/trial_info.json https://ftp.ncbi.nlm.nih.gov/pub/lu/TrialGPT/trial_info.json
29+
```
30+
31+
Three publicly available datasets are used in the study (please properly cite these datasets if you use them; see details about citations in the bottom):
32+
- The SIGIR 2016 corpus, available at: https://data.csiro.au/collection/csiro:17152
33+
- The TREC Clinical Trials 2021 corpus, available at: https://www.trec-cds.org/2021.html
34+
- The TREC Clinical Trials 2022 corpus, available at: https://www.trec-cds.org/2022.html
35+
36+
The SIGIR dataset is already in `/dataset/`, please download the corpora of TREC CT 2021 and 2022 by:
37+
38+
```bash
39+
wget -O dataset/trec_2021/corpus.jsonl https://ftp.ncbi.nlm.nih.gov/pub/lu/TrialGPT/trec_2021_corpus.jsonl
40+
wget -O dataset/trec_2022/corpus.jsonl https://ftp.ncbi.nlm.nih.gov/pub/lu/TrialGPT/trec_2022_corpus.jsonl
41+
```
42+
43+
## TrialGPT-Retrieval
44+
45+
Given a patient summary and an initial collection of clinical trials, the first step is TrialGPT-Retrieval, which generates a list of keywords for the patient and utilizes a hybrid-fusion retrieval mechanism to get relevant trials (component a in the figure).
46+
47+
Specifically, one can run the code below for keyword generation. The generated keywords will be saved in the `./results/` directory.
48+
49+
```bash
50+
# syntax: python trialgpt_retrieval/keyword_generation.py ${corpus} ${model}
51+
# ${corpus} can be sigir, trec_2021, and trec_2022
52+
# ${model} can be any model indices in OpenAI or AzureOpenAI API
53+
# examples below
54+
python trialgpt_retrieval/keyword_generation.py sigir gpt-4-turbo
55+
python trialgpt_retrieval/keyword_generation.py trec_2021 gpt-4-turbo
56+
python trialgpt_retrieval/keyword_generation.py trec_2022 gpt-4-turbo
57+
```
58+
59+
After generating the keywords, one can run the code below for retrieving relevant clinical trials. The retrieved trials will be saved in the `./results/` directory. The code below will use our cached results of keyword generation that are located in `./dataset/{corpus}/id2queries.json`.
60+
61+
```bash
62+
# syntax: python trialgpt_retrieval/hybrid_fusion_retrieval.py ${corpus} ${q_type} ${k} ${bm25_weight} ${medcpt_weight}
63+
# ${corpus} can be sigir, trec_2021, and trec_2022
64+
# ${q_type} can be raw, gpt-35-turbo (our cached results), and gpt-4-turbo (our cached results), Clinician_A (for sigir only), Clinician_B (for sigir only), Clinician_C (for sigir only), and Clinician_D (for sigir only)
65+
# ${k} is the constant in the reciprocal rank fusion, and we recommend using 20
66+
# ${bm25_weight} is the weight for the BM25 retriever, it should be set as 1 unless in ablation experiments
67+
# ${medcpt_weight} is the weight for the MedCPT retriever, it should be set as 1 unless in ablation experiments
68+
# examples below
69+
python trialgpt_retrieval/hybrid_fusion_retrieval.py sigir gpt-4-turbo 20 1 1
70+
python trialgpt_retrieval/hybrid_fusion_retrieval.py trec_2021 gpt-4-turbo 20 1 1
71+
python trialgpt_retrieval/hybrid_fusion_retrieval.py trec_2022 gpt-4-turbo 20 1 1
72+
```
73+
74+
## TrialGPT-Matching
75+
76+
After retrieving the candidate clinical trials with TrialGPT-Retrieval, the next step is to use TrialGPT-Matching to perform fine-grained criterion-by-criterion analyses on each patient-trial pair (component b in the figure). We have also made the retrieved trials by GPT-4-based TrialGPT-Retrieval available at `./dataset/{corpus}/retrieved_trials.json`. One can run the following commands to use TrialGPT-Matching, and the results will be saved in `./results/`:
77+
78+
```bash
79+
# syntax: python trialgpt_matching/run_matching.py ${corpus} ${model}
80+
# ${corpus} can be sigir, trec_2021, and trec_2022
81+
# ${model} can be any model indices in OpenAI or AzureOpenAI API
82+
# examples below
83+
python trialgpt_matching/run_matching.py sigir gpt-4-turbo
84+
python trialgpt_matching/run_matching.py trec_2021 gpt-4-turbo
85+
python trialgpt_matching/run_matching.py trec_2022 gpt-4-turbo
86+
```
87+
88+
## TrialGPT-Ranking
89+
90+
The final step is to use TrialGPT-Ranking to aggregate the criterion-level predictions into trial-level scores for ranking (component c in the figure). To get the LLM-aggregation scores for TrialGPT-Ranking, one can run the following commands. The results will be saved in `./results/`:
91+
92+
```bash
93+
# syntax: python trialgpt_ranking/run_aggregation.py ${corpus} ${model} ${matching_results_path}
94+
# ${corpus} can be sigir, trec_2021, and trec_2022
95+
# ${model} can be any model indices in OpenAI or AzureOpenAI API
96+
# ${matching_results_path} is the path to the TrialGPT matching results
97+
# example below (please make sure to have the TrialGPT-Matching results for the SIGIR corpus with the gpt-4-turbo model before running this)
98+
python trialgpt_ranking/run_aggregation.py sigir gpt-4-turbo results/matching_results_sigir_gpt-4-turbo.json
99+
```
100+
101+
Once the matching results and the aggregation results are complete, one can run the following code to get the final ranking of clinical trials for each patient:
102+
103+
```bash
104+
# syntax: python trialgpt_ranking/rank_results.py ${matching_results_path} ${aggregation_results_path}
105+
# ${matching_results_path} is the path to the TrialGPT matching results
106+
# ${aggregation_results_path} is the path to the aggregation results generated above
107+
# example below (please make sure to have the TrialGPT-Matching results and the aggregation results for the SIGIR corpus with the gpt-4-turbo model before running this)
108+
python trialgpt_ranking/rank_results.py results/matching_results_sigir_gpt-4-turbo.json results/aggregation_results_sigir_gpt-4-turbo.json
109+
```
110+
111+
Example output:
112+
113+
```bash
114+
Patient ID: sigir-20141
115+
Clinical trial ranking:
116+
NCT00185120 2.8999999995
117+
NCT02144636 2.8999999995
118+
NCT02608255 2.84999999975
119+
NCT01724996 2.7999999998
120+
...
121+
```
122+
123+
## Acknowledgments
124+
125+
This work was supported by the Intramural Research Programs of the National Institutes of Health, National Library of Medicine.
126+
127+
## Disclaimer
128+
129+
This tool shows the results of research conducted in the Computational Biology Branch, NCBI/NLM. The information produced on this website is not intended for direct diagnostic use or medical decision-making without review and oversight by a clinical professional. Individuals should not change their health behavior solely on the basis of information produced on this website. NIH does not independently verify the validity or utility of the information produced by this tool. If you have questions about the information produced on this website, please see a health care professional. More information about NCBI's disclaimer policy is available.
130+
131+
## Citation
132+
133+
If you find this repo helpful, please cite TrialGPT by:
134+
```bibtex
135+
@article{jin2023matching,
136+
title={Matching patients to clinical trials with large language models},
137+
author={Jin, Qiao and Wang, Zifeng and Floudas, Charalampos S and Chen, Fangyuan and Gong, Changlin and Bracken-Clarke, Dara and Xue, Elisabetta and Yang, Yifan and Sun, Jimeng and Lu, Zhiyong},
138+
journal={ArXiv},
139+
year={2023},
140+
publisher={ArXiv}
141+
}
142+
```
143+
144+
If you use the SIGIR cohort, please cite the original dataset papers by:
145+
```bibtex
146+
@inproceedings{koopman2016test,
147+
title={A test collection for matching patients to clinical trials},
148+
author={Koopman, Bevan and Zuccon, Guido},
149+
booktitle={Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval},
150+
pages={669--672},
151+
year={2016}
152+
}
153+
@inproceedings{roberts2015overview,
154+
title={Overview of the TREC 2015 Clinical Decision Support Track},
155+
author={Roberts, Kirk and Simpson, Matthew S and Voorhees, Ellen M and Hersh, William R},
156+
booktitle={Proceedings of the Twenty-Fourth Text REtrieval Conference (TREC 2015)},
157+
year={2015}
158+
}
159+
@inproceedings{simpson2014overview,
160+
title={Overview of the TREC 2014 Clinical Decision Support Track},
161+
author={Simpson, Matthew S and Voorhees, Ellen M and Hersh, William R},
162+
booktitle={Proceedings of the Twenty-Third Text REtrieval Conference (TREC 2014)},
163+
year={2014}
164+
}
165+
```
166+
167+
If you use the TREC cohorts, please cite the original dataset papers by:
168+
```bibtex
169+
@inproceedings{roberts2021overview,
170+
title={Overview of the TREC 2021 clinical trials track},
171+
author={Roberts, Kirk and Demner-Fushman, Dina and Voorhees, Ellen M and Bedrick, Steven and Hersh, Willian R},
172+
booktitle={Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021)},
173+
year={2021}
174+
}
175+
@inproceedings{roberts2022overview,
176+
title={Overview of the TREC 2022 clinical trials track},
177+
author={Roberts, Kirk and Demner-Fushman, Dina and Voorhees, Ellen M and Bedrick, Steven and Hersh, Willian R},
178+
booktitle={Proceedings of the Thirty-first Text REtrieval Conference (TREC 2022)},
179+
year={2022}
180+
}
181+
```

0 commit comments

Comments
 (0)