hybrid_fusion_retrieval.py: how to run without qrels files

[kinal1999]

Hi, I was wondering if it was possible to run the hybrid_fusion_retrieval.py without the qrels. I would like to run my own data set but i do not have ground truths available for that.

[Andy-jqa]

Hi, I was wondering if it was possible to run the hybrid_fusion_retrieval.py without the qrels. I would like to run my own data set but i do not have ground truths available for that.

Yes. You can comment all qrels-related lines in the script and it will still run the retrieval, just without any evaluations.

[kinal1999]

I am trying to run the script with only 5 patients but I keep getting this error. I was able to run the script with qrels and full data set, but I run into this when commenting out qrels. I attached the file with qrels commented out (converted to txt for upload purposes)

(310) kinal@kinal-TensorBook-2022:~/Desktop/jan29_TrialGPT-main$ python trialgpt_retrieval/hybrid_fusion_retrieval.py kp gpt-4-turbo 20 1 1
/home/kinal/Desktop/jan29_TrialGPT-main/310/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
Encoding the corpus
100%|███████████████████████████████████████| 3621/3621 [15:43<00:00, 3.84it/s]
/home/kinal/Desktop/jan29_TrialGPT-main/310/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
0%| | 0/5 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/kinal/Desktop/jan29_TrialGPT-main/trialgpt_retrieval/hybrid_fusion_retrieval.py", line 163, in
conditions = id2queries[qid][q_type]["conditions"]
KeyError: 'gpt-4-turbo'

hybrid_fusion_retrieval (copy).txt

[Andy-jqa]

I am trying to run the script with only 5 patients but I keep getting this error. I was able to run the script with qrels and full data set, but I run into this when commenting out qrels. I attached the file with qrels commented out (converted to txt for upload purposes)

(310) kinal@kinal-TensorBook-2022:~/Desktop/jan29_TrialGPT-main$ python trialgpt_retrieval/hybrid_fusion_retrieval.py kp gpt-4-turbo 20 1 1 /home/kinal/Desktop/jan29_TrialGPT-main/310/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True. warnings.warn( Encoding the corpus 100%|███████████████████████████████████████| 3621/3621 [15:43<00:00, 3.84it/s] /home/kinal/Desktop/jan29_TrialGPT-main/310/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True. warnings.warn( 0%| | 0/5 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/kinal/Desktop/jan29_TrialGPT-main/trialgpt_retrieval/hybrid_fusion_retrieval.py", line 163, in conditions = id2queries[qid][q_type]["conditions"] KeyError: 'gpt-4-turbo'

hybrid_fusion_retrieval (copy).txt

Can you show me your id2queries file?

[kinal1999]

retrieval_keywords_gpt-4o_20pt.json
retrieval_keywords_gpt-4o_kp.json

same error with both of these

[Andy-jqa]

retrieval_keywords_gpt-4o_20pt.json retrieval_keywords_gpt-4o_kp.json

same error with both of these

Note that there is an additional model key in the original id2query files https://github.com/ncbi-nlp/TrialGPT/blob/main/dataset/trec_2021/id2queries.json#L4.

If you want to use the uploaded version you can just remove q_type in the error line.

[kinal1999]

my outputted retrieved_trials.json does not look like yours, so i am unable to run matching now. how can i fix this?

[Andy-jqa]

my outputted retrieved_trials.json does not look like yours, so i am unable to run matching now. how can i fix this?

what does your file look like?

[kinal1999]

qid2nctids_results_sys.argv[2]_20pt_k20_bm25wt1_medcptwt1_N2000.json

[Andy-jqa]

@kinal1999 do you still need help?

[stswee]

@Andy-jqa I've encountered a similar issue as I cannot reproduce the retrieved_trials.json file with the current code and I get results similar to @kinal1999 (a json file with patient ids with clinical trial ids)

[jchan58]

@stswee what is the format of your clinical trials - like did use this line "wget -O dataset/trial_info.json https://ftp.ncbi.nlm.nih.gov/pub/lu/TrialGPT/trial_info.json" to download the csv? Also did you modify any parts of the code - if so where specifically?

[stswee]

@jchan58 I've used git clone to download the repository. When modifying the code, I only changed the LLM I used (I used a Llama instruct model from Huggingface).

[jchan58]

@stswee I've added the missing metadata handling for retrieved_trials.json in hybrid_fusion_retrieval.py. This should resolve the issue where you were only seeing a list of NCTIDs for that file.
See the PR here: #9.

[Andy-jqa]

Hi @stswee, please let us know if the PR solves your issue.

Thanks