Raw phenotypes and semantic similarity matrix for one entity (Aim 1)

[uyedaj]

For Aim I, @sergeitarasov and @uyedaj need:
An entity with good data coverage in phenoscape (e.g. dorsal fin) and all of the raw phenotypes and taxa.

We then need the raw phenotypes made into a semantic similarity matrix (Jaccard's) by @balhoff that gives us all pairwise similarities for all the phenotypes that we have.

[wdahdul]

Paula and I found that the term 'pectoral fin spine' for Siluriformes (catfishes) has broad categories of qualities associated with it, and the corresponding tree is available in OpenTree.

@balhoff can you send @sergeitarasov and @uyedaj the datasets mentioned above for 'pectoral fin spine' and 'Siluriformes'?

[wdahdul]

@sergeitarasov @uyedaj
For the tree, there's a ‘catfish’ collection in Open Tree with 13 trees that we uploaded a few years ago corresponding to taxa with data annotated in Phenoscape. We need to get a customized and synthetic tree for catfishes that includes the curated phylogenies in the tree collection. As I understand, this needs to be requested from OT. Have you done this before? I’ve asked Laura for more details, can update you next week.

[sergeitarasov]

I used OT but I am not familiar with all details yet

[wdahdul]

I chatted with Laura about requesting a custom tree from OT. She confirms that this needs to be requested (there’s been mention of adding a new feature allowing users to synthesize their own custom tree, but this isn’t implemented yet). Laura mentioned several options that she specified (e.g., include all incertae sedis taxa, remove monotypic branches (for reconstruction), remove OTT IDs from taxon names, and exclude subspecies. Having said that, if you get the tree from OT then there's the taxon reconciliation to work out, might be better to wait for the potential Catalog of Fishes update with OT.

[pmabee]

Yes. Details as per below I. The Jackson et al paper too. I’m scheduled to talk with catalog of fishes tomorrow Paula

…

Sent from my iPhone On Oct 3, 2018, at 2:10 PM, Wasila Dahdul <notifications@github.com<mailto:notifications@github.com>> wrote: I chatted with Laura about requesting a custom tree from OT. She confirms that this needs to be requested (there’s been mention of adding a new feature allowing users to synthesize their own custom tree, but this isn’t implemented yet). Laura mentioned several options that she specified (e.g., include all incertae sedis taxa, remove monotypic branches (for reconstruction), remove OTT IDs from taxon names, and exclude subspecies. Having said that, if you get the tree from OT then there's the taxon reconciliation to work out, might be better to wait for the potential Catalog of Fishes update with OT. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#7 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACObXlQh24cdLWdaAdMwu7PxooBxbPgMks5uhQuegaJpZM4W04fQ>.

[balhoff]

@uyedaj @sergeitarasov I've created a preliminary service for accessing similarity scores comparing pairs of character states. We don't currently have stable global identifiers for each character state, so for now I implemented the service to accept several parameters that specify two states ("left" and "right"): study, character number (1-indexed), and state symbol. If these don't work for you, we can discuss better designs for this endpoint. Here's how you can call it:

#!/bin/bash

STUDY='http%3A%2F%2Fdx.doi.org%2F10.1111%2Fj.1096-3642.2008.00407.x'
CHAR=158

curl "http://kb.phenoscape.org/api/similarity/states?leftStudy=$STUDY&leftCharacter=$CHAR&leftSymbol=0&rightStudy=$STUDY&rightCharacter=$CHAR&rightSymbol=1"

You can compare states from different studies; I just compared two from the same character here. This method will require you to make pairwise calls for all the states you want to compare. We can talk about better ways to specify the whole set at once, but this should allow you to get started.

[sergeitarasov]

I tried it and it works. It's definitely a good start for us to get the access to the similarity. But perhaps we may discuss on how to improve it by, for example, getting entire similarity matrix using only one query.

[hlapp]

I would argue that what we should be returning instead is a binary-encoded matrix of each character state against each ontology class (X(i,j) == 1 if ontology class i subsumes one of the terms annotated to state j and 0 otherwise). It is very easy from there in R to compute the Jaccard similarity through simple vector algebra.

[balhoff]

@sergeitarasov for getting a whole matrix back, what we would need to work out would be how to specify the set of states which you want to pairwise compare. For this use case would that simply be the identifier for a publication which has been annotated in the KB?

[balhoff]

I would argue that what we should be returning instead is a binary-encoded matrix of each character state against each ontology class X(i,j) == 1

@hlapp this would probably be a very large payload—currently the KB contains 490903 named classes, and this will probably grow. Would the result format need to identify each class (many of which fall more into the category of implementation detail than global concept)? Or could it simply guarantee that the same index is the same class for any given KB deployment?

[uyedaj]

@balhoff I don't think we want to limit it only to a single publication, but that might be a good start. An example use case is that I would want to look at a specific clade of fish and a specific entity, e.g. caudal fin, and get all annotated phenotypes from the KB. Then I would want to get a matrix of their semantic similarity using this service. This may combine multiple publications.