Parsing HPOA per row problematic for multiple entries for same OMIM-HPO association

[leokim-l]

@hansenp I finally managed to solve the issue we discussed today! TL;DR is: the problem is probably in pyboqa, but let us make sure.

I noticed it was not the 0/7 or similar to be problematic, but some lines of HPOA contain things like:

OMIM:191100 Tuberous sclerosis-1 HP:0012798 PMID:10852420 PCS 20/78 FEMALE P HPO:probinson[2019-11-28]
OMIM:191100 Tuberous sclerosis-1 HP:0012798 PMID:16485546 PCS HP:0040284 MALE P HPO:probinson[2019-11-28]
OMIM:191100 Tuberous sclerosis-1 HP:0012798 PMID:29196670 PCS 0/5 P ORCID:0000-0002-0736-9199[2024-06-06]

In DiseaseDataParseIngest each line is read on its own, so because of the first line the HPO term HP:0012798 will be included, and because of the third line it will also be in the excluded set. This means that we are correctly including the term. If I manually forcefully remove it when creating diseaseData, the counts match perfectly with pyboqa. I strongly suspect pyboqa considers this as excluded, and this is the origin of the mismatch.

We should make sure that the case in which multiple lines of HPOA contain the same HPO term but have different PMID as sources we create one single frequency term.

I think eventually we want to use phenol for parsing phenotype-diseases associations, but before deleting this issue I want to make sure

1. This problem does not turn up there, too.
2. How disease-phenotype data will look like in Exomiser.

[hansenp]

@leokim-l Very cool! But the explanation cannot be entirely correct.

pyboqa does take frequencies and included/excluded terms into account when reading data, but later in the calculations, the excluded terms are not used at all.

In fact, the difference is that pyboqa uses file genes_to_phenotype.txt instead of file phenotype.hpoa as input. This has historical reasons, because I initially thought it would be useful to have the disease-gene associations readily available. However, file genes_to_phenotype.txt only contains the line with a frequency of zero:

7248	TSC1	HP:0012798	Pulmonary lymphangiomyomatosis	0/5	OMIM:191100

The other two lines were probably discarded when file genes_to_phenotype.txt was created, which is what happens here: https://github.com/monarch-initiative/hpoannotqc/blob/master/src/main/java/org/monarchinitiative/hpoannotqc/cmd/SupplementalFilesCommand.java#L113

Code for verification

$ wget -O phenotype.hpoa https://github.com/obophenotype/human-phenotype-ontology/releases/latest/download/phenotype.hpoa
$ grep OMIM:191100 phenotype.hpoa | grep HP:0012798                                                                      
OMIM:191100	Tuberous sclerosis-1		HP:0012798	PMID:10852420	PCS		20/78	FEMALE		P	HPO:probinson[2019-11-28]
OMIM:191100	Tuberous sclerosis-1		HP:0012798	PMID:16485546	PCS		HP:0040284	MALE		P	HPO:probinson[2019-11-28]
OMIM:191100	Tuberous sclerosis-1		HP:0012798	PMID:29196670	PCS		0/5			P	ORCID:0000-0002-0736-9199[2024-06-06]

$ wget -O genes_to_phenotype.txt https://github.com/obophenotype/human-phenotype-ontology/releases/latest/download/genes_to_phenotype.txt
$ grep OMIM:191100 genes_to_phenotype.txt | grep HP:0012798                                                                                
7248	TSC1	HP:0012798	Pulmonary lymphangiomyomatosis	0/5	OMIM:191100

Can you provide some more examples where the counts differ between pyboqa and Java-BOQA?

[leokim-l]

I agree, the issue is not in pyboqa per se but in the genes_to_phenotype.txt . The only other example I have is

OMIM:614558	Epileptic encephalopathy, early infantile, 13		HP:0033258	PMID:22365152	PCS	HP:0003621	1/1			P	HPO:probinson[2022-06-04]
OMIM:614558	Epileptic encephalopathy, early infantile, 13		HP:0033258	PMID:24888894	PCS		0/7			P	HPO:probinson[2022-06-04]

where using your wget and then grepping

grep OMIM:614558 genes_to_phenotype.txt | grep HP:0033258
6334	SCN8A	HP:0033258	Sudden unexpected death in epilepsy	0/7	OMIM:614558

I think to spot all we could read phenotype.hpoa into a pandas dataframe, group by OMIM id, and take it from there. I think it's a bug in hpoannotqc that needs to be fixed.

[hansenp]

Two tests fail due to this bug. Since it's not a bug in BOQA, we could simply delete the corresponding lines in pyboqa_counts_for_top_ranked_diseases.csv.

These lines:

PMID_33731876_fam127.json,OMIM:614558,115,17,2,24
PMID_29196670_Patient9.json,OMIM:191100,212,25,0,37