Gwas Rest API -: Implement custom sorting for API v2.0

[sajo-ebi]

Implement custom sorting in API v2.0 to allow sorting on selected fields in API . For studies & associations we can use use Catalog as reference to display fields for sorting, for remaining endpoints curators to provide details

[eks-ebi]

Here are my suggested fields for sorting, for each endpoint. They are required, unless flagged as "nice to have". Any fields not mentioned do not need to be sortable. I have added the primary identifier for each endpoint (e.g. GCST for studies) as "nice to have", because this seems like good practice, but may not always be scientifically meaningful to sort.

associations - these are all already enabled for sorting in the UI

• p_value (should have)
• risk_frequency (should have)
• or_value (should have)
• beta_num (should have)

studies - the first two of these are already enabled for sorting in the UI, GCST is new. Some fields that are sortable in the UI are not available in the API (because they are actually not a property of the study entity), so I have left these out: publication date, association count. You can also sort the UI by full summary stats (yes/no) - but this is already implemented as a filter in the API and I think works better that way than as a sort feature.

• discovery_ancestry (should have) - only the integer (i.e. sample size) should be used for sorting, not the text component - DEPRIORITISED
• replication_ancestry (should have) - only the integer (i.e. sample size) should be used for sorting, not the text component - DEPRIORITISED
• accession_id (i.e. GCST) (nice to have)
• SNP_count (nice to have)

ancestries:

• number_of_individuals (should have)

publications:

• PMID (nice to have)
• publication_date (should have)

efo-traits:

• efo_id (nice to have)
• efo_trait (nice to have)

single-nucleotide-polymorphisms:

• locations - sort first by chromosome_name then chromosome_position (should have)
• maf (should have) - DEPRIORITISED
• rs_id (nice to have)

genomic contexts

• distance (nice to have)

genes: - ALL DEPRIORITISED

• gene_name (nice to have)
• ensembl_gene_ids (nice to have)
• entrez_gene_ids (nice to have)
• location (nice to have) - sort first by chromosome_name then chromosome_position if possible (note these are not listed as separate fields in this endpoint's response, but maybe this could be done to match the sort functionality of locations under single-nucleotide-polymorphisms above??

loci: no sorting required

body-of-works: no sorting required

unpublished-ancestries:

• sample_size (should have)
• cases (nice to have)
• controls (nice to have)

unpublished-studies:

• study_accession (nice to have)
• variant_count (nice to have)

[ljwh2]

@sajo-ebi please implement all of Elliot's proposals marked "should have". Any that are "nice to have", please implement if straightforward and will not impact performance.

[sajo-ebi]

Identified an issue with our current implementation w.r.t sorting where null values in DB are not handled properly & null values are coming on top when sorted in descending order. we have implement new sorting framework to handle null values , currenly working on it

[sajo-ebi]

Deprioritising these two sorting requirements

studies -: (This will some complex logic to implement which we will do later)

• discovery_ancestry
• replication_ancestry

genes -: (This is calling Search UI & code changes need to mbe made there)

• gene_name (nice to have)
• ensembl_gene_ids (nice to have)
• entrez_gene_ids (nice to have)
• location (nice to have)

[sajo-ebi]

single-nucleotide-polymorphisms: (The maf is not stored in our DB tables , the data is retrieved by calling Ensembl API , hence sorting can't be implemented in current structure in DB)

• maf

[ljwh2]

@eks-ebi did you mean maf or raf here?

[eks-ebi]

I meant maf - it is one of the fields that is shown in the API response for the SNP endpoint. There is no raf in that endpoint (because it is just for SNPs, not associations, there's no risk allele so no raf)

But as Sajo points out, I guess this is retrieved on the fly from Ensembl and therefore can't be used for sorting, which I think is okay.

[ljwh2]

OK, we can deprioritise it

[ljwh2]

@sajo-ebi it seems it is possible for the same rsID to appear on X and Y chromosomes. But this is only possible over a very short region of the chromosome, so I expect there would be very few examples in the Catalog. As discussed I suggest we sort by the first location listed. We can include a note about this in the documentation (I will deal with that).

[sajo-ebi]

@eks-ebi on the order of schema have placed schema for common objects like page.link & errors on top followed by all Api related response schema in alphabetical order

[eks-ebi]

@sajo-ebi It doesn't look like the schemas are exactly in alphabetical order right now, e.g. it goes ancestries, collection-ancestries, ancestry-groups, associations, page-associations, authors... So collection-ancestries and page-associations seem mixed in with the As.

Is there some other hidden factor in the sorting that I'm not seeing?

[eks-ebi]

For testing the sorting items above, I will tick the checkboxes above when I have confirmed that the sort works as expected. Any that are not working as expected I will comment here:

• associations sort by risk_frequency - it seems that risk_frequency is stored as a string which is sometimes formatted as a normal float (e.g. 0.0014) but sometimes in scientific notation (e.g. 9.9E-5). The string-based sorting is not dealing correctly with scientific numbers, so appears to sort just on the mantissa while ignoring the exponent. Example: when displaying associations for PMID 34662886 and sorting by descending order, the associations are being incorrectly returned in this order by risk_frequency (these are also very small numbers so should be at the other end of the list):

9.9E-5
9.6E-4
9.5E-4
9.4E-6
9.4E-5
9.2E-5
9.0E-6
9.0E-4
8.9E-4
8.7E-5

• associations sort by or_value looked okay in my tests, but it is also stored as a string, so worth checking if the same issue with scientific notation could affect this sort option too