feat: add pgap annotation; closes #2

rabioinf · rabioinf · commit 2f5d6c849f6f · 2025-12-03T17:37:35.000+01:00
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -47,11 +47,11 @@ jobs:
         with:
           directory: .test
           snakefile: workflow/Snakefile
-          args: "--sdm conda --show-failed-logs --cores 3 --conda-cleanup-pkgs cache -n"
+          args: "--sdm conda --show-failed-logs --cores 1 --conda-cleanup-pkgs cache -n"
 
       - name: Test report
         uses: snakemake/snakemake-github-action@v2.0.0
         with:
           directory: .test
           snakefile: workflow/Snakefile
-          args: "--cores 3 --report report.zip -n"
+          args: "--cores 1 --report report.zip -n"
diff --git a/.test/config/config.yml b/.test/config/config.yml
@@ -1 +1,9 @@
-samplesheet: "config/samples.tsv"
+samplesheet: "config/samples.csv"
+outdir: "results"
+
+pgap:
+  bin: "path/to/pgap.py"
+  use_yaml_config: True
+  prepare_yaml_files:
+    generic: "config/generic.yaml"
+    submol: "config/submol.yaml"
diff --git a/.test/config/generic.yaml b/.test/config/generic.yaml
@@ -0,0 +1,6 @@
+fasta:
+  class: File
+  location: None
+submol:
+  class: File
+  location: None
diff --git a/.test/config/samples.csv b/.test/config/samples.csv
@@ -0,0 +1,2 @@
+sample,species,strain,id_prefix,file
+EC2224,"Streptococcus pyogenes",SF370,SPY,"data/assembly.fasta"
diff --git a/.test/config/submol.yaml b/.test/config/submol.yaml
@@ -0,0 +1,21 @@
+topology: "circular"
+location: "chromosome"
+organism:
+  genus_species: ""
+  strain: ""
+contact_info:
+  last_name: ""
+  first_name: ""
+  email: ""
+  organization: ""
+  department: ""
+  street: ""
+  city: ""
+  state: ""
+  postal_code: ""
+  country: ""
+authors:
+  - author:
+      last_name: "last_name"
+      first_name: "first_name"
+locus_tag_prefix: ""
diff --git a/.test/data/assembly.fasta b/.test/data/assembly.fasta
diff --git a/README.md b/README.md
@@ -13,6 +13,42 @@ The usage of this workflow is described in the [Snakemake Workflow Catalog](http
 
 If you use this workflow in a paper, don't forget to give credits to the authors by citing the URL of this repository.
 
+## Workflow overview
+
+1. Parse `samples.csv` table containing the samples's meta data (`python`)
+2. Annotate assemblies using NCBI's Prokaryotic Genome Annotation Pipeline ([PGAP](https://github.com/ncbi/pgap))
+
+## Requirements
+
+- [PGAP](https://github.com/ncbi/pgap)
+
+## Installation
+
+**Step 1: Clone this repository**
+
+```bash
+git clone https://github.com/MPUSP/snakemake-assembly-postprocessing.git
+cd snakemake-assembly-postprocessing
+```
+
+**Step 2: Install dependencies**
+
+It is recommended to install snakemake and run the workflow with `conda` or `mamba`. [Miniforge](https://conda-forge.org/download/) is the preferred conda-forge installer and includes `conda`, `mamba` and their dependencies.
+
+**Step 3: Create snakemake environment**
+
+This step creates a new conda environment called `snakemake-assembly-postprocessing`.
+
+```bash
+mamba create -c conda-forge -c bioconda -n snakemake-assembly-postprocessing snakemake pandas
+conda activate snakemake-assembly-postprocessing
+```
+
+**Step 4: Install PGAP**
+
+- PGAP can be downloaded from https://github.com/ncbi/pgap. Please follow the installation instructions there.
+- Define the path to the `pgap.py` script (located in the `scripts` folder) in the `config` file (recommended: `./resources`)
+
 ## Authors
 
 - Dr. Rina Ahmed-Begrich
@@ -25,4 +61,6 @@ If you use this workflow in a paper, don't forget to give credits to the authors
 
 ## References
 
+> Li W, O'Neill KR, Haft DH, DiCuccio M, Chetvernin V, Badretdin A, Coulouris G, Chitsaz F, Derbyshire MK, Durkin AS, Gonzales NR, Gwadz M, Lanczycki CJ, Song JS, Thanki N, Wang J, Yamashita RA, Yang M, Zheng C, Marchler-Bauer A, Thibaud-Nissen F. _RefSeq: Expanding the Prokaryotic Genome Annotation Pipeline reach with protein family model curation._ Nucleic Acids Res, **2021** Jan 8;49(D1):D1020-D1028. https://doi.org/10.1093/nar/gkaa1105
+
 > Köster, J., Mölder, F., Jablonski, K. P., Letcher, B., Hall, M. B., Tomkins-Tinch, C. H., Sochat, V., Forster, J., Lee, S., Twardziok, S. O., Kanitz, A., Wilm, A., Holtgrewe, M., Rahmann, S., & Nahnsen, S. _Sustainable data analysis with Snakemake_. F1000Research, 10:33, 10, 33, **2021**. https://doi.org/10.12688/f1000research.29032.2.
diff --git a/config/README.md b/config/README.md
@@ -5,8 +5,8 @@
 This workflow requires `fasta` input data.
 The samplesheet table has the following layout:
 
-| sample      | species  | strain                  | id_prefix   | file   |
-| ----------- | ------------ | ---------------------------------- | ------------- | ------------- |
+| sample | species | strain | id_prefix | file |
+| ----------- | ------------ | ------------- | ------------- | ------------- |
 | EC2224 | "Streptococcus pyogenes" | SF370 | Spy | assembly.fasta |
 
 ### Execution
@@ -22,11 +22,11 @@ Adjust options in the default config file `config/config.yml`.
 Before running the entire workflow, perform a dry run using:
 
 ```bash
-snakemake --cores 3 --sdm conda --directory .test --dry-run
+snakemake --cores 1 --sdm conda --directory .test --dry-run
 ```
 
 To run the workflow with test files using **conda**:
 
 ```bash
-snakemake --cores 3 --sdm conda --directory .test
+snakemake --cores 1 --sdm conda --directory .test
 ```
diff --git a/config/config.yml b/config/config.yml
@@ -1 +1,9 @@
-samplesheet: "config/samples.tsv"
+samplesheet: "config/samples.csv"
+outdir: "results"
+
+pgap:
+  bin: "path/to/pgap.py"
+  use_yaml_config: True
+  prepare_yaml_files:
+    generic: "config/generic.yaml"
+    submol: "config/submol.yaml"
diff --git a/config/generic.yaml b/config/generic.yaml
@@ -0,0 +1,6 @@
+fasta:
+  class: File
+  location: None
+submol:
+  class: File
+  location: None
diff --git a/config/samples.csv b/config/samples.csv
diff --git a/config/samples.tsv b/config/samples.tsv
diff --git a/config/schemas/config.schema.yml b/config/schemas/config.schema.yml
@@ -6,6 +6,35 @@ properties:
   samplesheet:
     type: string
     description: Path to the sample sheet file
+  outdir:
+    type: string
+    description: Output directory for results
+  pgap:
+    type: object
+    properties:
+      bin:
+        type: string
+        description: Path to the PGAP script
+      use_yaml_config:
+        type: boolean
+        description: Whether to use YAML configuration for PGAP
+      prepare_yaml_files:
+        type: object
+        properties:
+          generic:
+            type: string
+            description: Path to the generic YAML configuration file
+          submol:
+            type: string
+            description: Path to the submol YAML configuration file
+        required:
+          - generic
+          - submol
+    required:
+      - bin
+      - use_yaml_config
+      - prepare_yaml_files
 
 required:
   - samplesheet
+  - pgap
diff --git a/config/schemas/samples.schema.yml b/config/schemas/samples.schema.yml
@@ -1,3 +1,26 @@
 $schema: "https://json-schema.org/draft/2020-12/schema"
 title: "Sample Schema"
 description: "Sample schema for the assembly post-processing workflow"
+type: object
+properties:
+  sample:
+    type: string
+    description: Unique identifier for the sample
+  species:
+    type: string
+    description: Species name of the sample
+  strain:
+    type: string
+    description: Strain name of the sample
+  id_prefix:
+    type: string
+    description: Identifier prefix for the locus_tag of annotated genes of the sample
+  file:
+    type: string
+    description: Path to the assembly file for the sample
+required:
+  - sample
+  - species
+  - strain
+  - id_prefix
+  - file
diff --git a/config/submol.yaml b/config/submol.yaml
@@ -0,0 +1,21 @@
+topology: "circular"
+location: "chromosome"
+organism:
+  genus_species: ""
+  strain: ""
+contact_info:
+  last_name: ""
+  first_name: ""
+  email: ""
+  organization: ""
+  department: ""
+  street: ""
+  city: ""
+  state: ""
+  postal_code: ""
+  country: ""
+authors:
+  - author:
+      last_name: "last_name"
+      first_name: "first_name"
+locus_tag_prefix: ""
diff --git a/workflow/Snakefile b/workflow/Snakefile
@@ -3,11 +3,14 @@
 # microbial genome assemblies.                          #
 # ----------------------------------------------------- #
 
+import os
+from datetime import datetime
+
 
 # -----------------------------------------------------
 # container definition (optional)
 # -----------------------------------------------------
-container: "oras://ghcr.io/mpusp/snakemake-assembly-post-processing:latest"
+container: "oras://ghcr.io/mpusp/snakemake-assembly-postprocessing:latest"
 
 
 # -----------------------------------------------------
@@ -16,10 +19,18 @@ container: "oras://ghcr.io/mpusp/snakemake-assembly-post-processing:latest"
 configfile: "config/config.yml"
 
 
+# set outout directory
+OUTDIR = config.get("outdir", "")
+
+if not OUTDIR:
+    OUTDIR = f"{os.getcwd()}/results"
+
+
 # -----------------------------------------------------
 # load rules
 # -----------------------------------------------------
 include: "rules/common.smk"
+include: "rules/annotate.smk"
 
 
 # -----------------------------------------------------
@@ -33,11 +44,6 @@ end = "\033[0m"
 msg = f"""\nSnakemake-assembly-postprocessing: A Snakemake workflow 
 for the post-processing of microbial genome assemblies."""
 
-OUTDIR = config.get("outdir", "")
-
-if not OUTDIR:
-    OUTDIR = f"{os.getcwd()}/results"
-
 prolog = f"Output directory: {OUTDIR}"
 year = datetime.today().year
 
@@ -67,5 +73,8 @@ onerror:
 # -----------------------------------------------------
 rule all:
     input:
-        samples.index.to_list(),
+        expand(
+            os.path.join(OUTDIR, "annotation/pgap/{sample}/{sample}.gff"),
+            sample=samples.index,
+        ),
     default_target: True
diff --git a/workflow/envs/base.yml b/workflow/envs/base.yml
@@ -0,0 +1,9 @@
+name: base
+channels:
+  - conda-forge
+  - bioconda
+  - nodefaults
+dependencies:
+  - python==3.13.0
+  - pandas==2.3.3
+  - apptainer==1.4.5
diff --git a/workflow/rules/annotate.smk b/workflow/rules/annotate.smk
@@ -0,0 +1,86 @@
+rule get_fasta:
+    input:
+        get_fasta,
+    output:
+        fasta=os.path.join(
+            OUTDIR, "annotation/pgap/prepare_files/{sample}/genome.fasta"
+        ),
+    conda:
+        "../envs/base.yml"
+    log:
+        os.path.join(
+            OUTDIR, "annotation/pgap/prepare_files/logs/{sample}_get_fasta.log"
+        ),
+    shell:
+        "INPUT=$(realpath {input}); "
+        "ln -s ${{INPUT}} {output}; "
+        "echo -e 'created symbolic link to genome fasta: {output}' > {log}"
+
+
+rule prepare_yaml_files:
+    input:
+        fasta=rules.get_fasta.output.fasta,
+    output:
+        input_yaml=os.path.join(
+            OUTDIR, "annotation/pgap/prepare_files/{sample}/input.yaml"
+        ),
+        submol_yaml=os.path.join(
+            OUTDIR, "annotation/pgap/prepare_files/{sample}/submol.yaml"
+        ),
+    conda:
+        "../envs/base.yml"
+    params:
+        locus_tag=lambda wc: samples.loc[wc.sample]["id_prefix"],
+        organism=lambda wc: samples.loc[wc.sample]["species"],
+        generic=config["pgap"]["prepare_yaml_files"]["generic"],
+        submol=config["pgap"]["prepare_yaml_files"]["submol"],
+        sample="{sample}",
+        pd_samples=samples,
+    log:
+        os.path.join(
+            OUTDIR, "annotation/pgap/prepare_files/logs/{sample}_prepare_yaml_files.log"
+        ),
+    script:
+        "../scripts/prepare_yaml_files.py"
+
+
+rule annotate_pgap:
+    input:
+        branch(
+            lookup(dpath="pgap/use_yaml_config", within=config),
+            then=rules.prepare_yaml_files.output.input_yaml,
+            otherwise=rules.get_fasta.output.fasta,
+        ),
+    output:
+        os.path.join(OUTDIR, "annotation/pgap/{sample}/{sample}.gff"),
+    conda:
+        "../envs/base.yml"
+    message:
+        """--- Run PGAP annotation for sample {wildcards.sample} ---"""
+    params:
+        pgap=config["pgap"]["bin"],
+        use_yaml_config=config["pgap"]["use_yaml_config"],
+        species=lambda wc: samples.loc[wc.sample]["species"],
+        outdir=lambda wc, output: os.path.dirname(output[0]),
+    threads: 1
+    log:
+        os.path.join(OUTDIR, "annotation/pgap/logs/{sample}_pgap.log"),
+    shell:
+        "rm -rf {params.outdir}; "
+        "if [ {params.use_yaml_config} == 'True' ]; then "
+        "echo -e 'Using YAML configuration for PGAP annotation' > {log}; "
+        "{params.pgap} -n "
+        "-o {params.outdir} "
+        "--docker apptainer "
+        "--prefix {wildcards.sample} "
+        "--no-self-update "
+        "{input} &>> {log}; "
+        "else "
+        "echo -e 'Using FASTA file and species name for PGAP annotation' > {log}; "
+        "{params.pgap} -n "
+        "-o {params.outdir} "
+        "--docker apptainer "
+        "--prefix {wildcards.sample} "
+        "--no-self-update "
+        "-g {input} -s '{params.species}' &>> {log}; "
+        "fi; "
diff --git a/workflow/rules/common.smk b/workflow/rules/common.smk
diff --git a/workflow/scripts/prepare_yaml_files.py b/workflow/scripts/prepare_yaml_files.py

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+sample,species,strain,id_prefix,file`
	`2`	`+EC2224,"Streptococcus pyogenes",SF370,SPY,"data/assembly.fasta"`