Write script to filter VCF

[eweitz]

Write a script to apply the user's filter selections to an annotated VCF.

Input parameters:

• Filters: command-line arguments describing the user's filter selections, e.g. "filters=clinsig:pathogenic+molcons:missense"
• Path to annotated VCF. an annotated VCF file where each row is a variant, and the INFO column has data on which filters the variant matches (e.g. clinsig=pathogenic;molcons=missense).

Output:

• Filtered VCF: A VCF file in which the variants are only those matching the user's filter selections. For example, if the user had a filter selection "clinsig:pathogenic", the output VCF would not include any variants with "clinsig:benign".

@nicovbing, this is the issue we discussed.

[eweitz]

@vlaufer, for a complete list of filters, see https://github.com/DCGenomics/NCBI_August_Hackathon_Push_Button_Genomics_Solution/blob/86be0185a497490e01997051d93281dfba7d4a5b/django/browser/templates/browser/upload.html#L34. The facets are in the 'legend' elements, e.g. "Clinical significance"; filters are in the input checkbox elements, e.g. "Pathogenic".

[eweitz]

@vlaufer, following up on #15 (comment), please refer to my Gitter comments from last night beginning at https://gitter.im/DCGenomics/NCBI_August_Hackathon_Push_Button_Genomics_Solution?at=55c8206b21801cd866ca7c22.

For convenience, I've copied them below:

Regarding "am i correct in assuming that currently the front-end only supports AND statements for variant filtering?" -- not quite.

Filters within a facet get combined with a Boolean OR; whereas filters across different facets get combined with a Boolean AND

Consider the facets and filters at left in http://www.ncbi.nlm.nih.gov/variation/view/#data
If one were to select "Variant type: single nucleotide variant" and "Variant type: insertion", then the result set would be variants that are of "Variant type: single nucleotide variant" OR "Variant type: insertion" -- e.g. http://www.ncbi.nlm.nih.gov/variation/view/?filters=vartype:single-nucleotide-variant,insertion#data

(Here "Variant type" is a facet, and "single nucleotide variant" and "insertion" are filters within that facet.)

Now consider the filter selection "Variant type: single nucleotide variant", "Variant type: insertion", "Clinical significance: Pathogenic"

In Boolean terms, that selects variants that have ("Variant type: single nucleotide variant" OR "Variant type: insertion") AND "Clinical significance: Pathogenic. For example, http://www.ncbi.nlm.nih.gov/variation/view/?filters=vartype:single-nucleotide-variant,insertion+clinsig:pathogenic#data

My recommendation: get the simple case of applying filters within a facet working (e.g. "Molecular consequence: missense" OR "Molecular consequence: nonsense"), then enhance that to work with filters in other facets (e.g. add on "Clinical significance: Pathogenic" to the molcons filter selections)
Example of how filters could be passed into your script: "molcons:missense,nonsense+clinsig:pathogenic".

Regarding the complete list of filters, please see #8 (comment)

In addition to the example linked in my previous comment, see also the mockup of the Upload page -- https://github.com/DCGenomics/NCBI_August_Hackathon_Push_Button_Genomics_Solution/blob/master/docs/mockup_Upload%20page.png.

Being able to view the filters in an actual UI from a web browser would be ideal, but we will be using a different instance (a copy of the machine as of tomorrow at 10 AM) shortly. I'll email the group about how to work with the new instance this week -- hopefully it will be as simple as changing the machine we SSH into.

In brief, if you can get your script working with input to a "filters" argument like "molcons:missense,nonsense+clinsig:pathogenic", using the logic described in my quoted comment above, then I think that will largely get this feature implemented.

[eweitz]

I've reassigned this from Nicolas (@nicovbing) to Vincent (@vlaufer).

Nicolas began developing a general VCF filtering feature in an awk script, but switched to working on the "Predicted impact" script (#13). The commit linked from there shows a script to filter specifiically for "Predicted impact".

Vincent is developing a generic VCF filtering module that will be called from Django.

[dauss75]

Just to follow up the current progress-- other than the filtering feature, all the functions in backend is complete and linked to the frontend?

[eweitz]

@dauss75, no, I think we still need to integrate the various back-end pipeline components, and I know there's still work to do in Django and the UI.

I'll try to comment here later today with a more detailed overview of what remains to be done. I should be in our Gitter chat room from 7:00 PM - 10:00 PM Eastern Time tonight if anyone wants discuss that way.

[ohjuarez]

Eric,
Please let me know if you need help.
Regards,

Octavio Juarez-Espinosa, PhD
Contractor –MSC, Inc. Scripting Developer
Computational Biology Section
Bioinformatics and Computational Biosciences Branch (BCBB)
OCICB/OSMO/OD/NIAID/NIH

5601 Fishers Lane, Room 4A72
Rockville, MD 20852
Mobile 2407628157
Office 12406692760

From: Eric Weitz <notifications@github.commailto:notifications@github.com>
Reply-To: DCGenomics/NCBI_August_Hackathon_Push_Button_Genomics_Solution <reply@reply.github.commailto:reply@reply.github.com>
Date: Monday, August 10, 2015 at 11:03 AM
To: DCGenomics/NCBI_August_Hackathon_Push_Button_Genomics_Solution <NCBI_August_Hackathon_Push_Button_Genomics_Solution@noreply.github.commailto:NCBI_August_Hackathon_Push_Button_Genomics_Solution@noreply.github.com>
Subject: Re: [NCBI_August_Hackathon_Push_Button_Genomics_Solution] Write script to filter VCF (#8)

@dauss75https://github.com/dauss75, no, I think we still need to integrate the various back-end pipeline components, and I know there's still work to do in Django and the UI.

I'll try to comment here later today with a more detailed overview of what remains to be done. I should be in our Gitter chat roomhttps://gitter.im/DCGenomics/NCBI_August_Hackathon_Push_Button_Genomics_Solution from 7:00 PM - 10:00 PM Eastern Time tonight if anyone wants discuss that way.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/8#issuecomment-129483138.

[eweitz]

Thanks Octavio! I did some work using the Solr output last night, and will indeed likely need help making some adjustments. I will open a separate issue for that soon, and ping you from there.

[eweitz]

Just to follow up the current progress-- other than the filtering feature, all the functions in backend is complete and linked to the frontend?

@dauss75, @vlaufer, please see #16, which describes what I think is the largest remaining chunk of integration work needed to link the Django front-end and your pipeline backend.