VC evaluation changed to 1000 genes + local workflow added

docs/build/html/_sources/dataset.rst.txt

@@ -1,13 +1,16 @@
 Datasets
 ========
-The list of datasets integrated into geneRNIB is provided below with their perturbation signatures as well as the type of perturbation used in each dataset.
+The list of datasets integrated into geneRNIB is provided below:
 
 .. image:: images/datasets.png
-   :width: 80%
+   :width: 50%
    :align: center
 ----
 
-All datasets provide RNA data, while the `OPSCA` and `IBD` datasets also includes scATAC data. 
+.. image:: images/table_datasets_summary.png
+   :width: 100%
+   :align: center
+----
 
 You need `awscli` to download the datasets. 
 

docs/build/html/_sources/evaluation.rst.txt

@@ -9,13 +9,14 @@ The evaluation metrics used in geneRNIB are summarized below.
    :align: center
 ----
 
-.. image:: images/datasets_metrics.png
-   :width: 90%
+.. image:: images/metric_quality_evaluation.png
+   :width: 100%
    :align: center
 ----
 
 
-For a detailed description of each metric, refer to the geneRNIB paper.
+For a detailed description of each metric, refer to the geneRNIB paper. Not all the metrics were applicable to all datasets, as shown in the table. In addition, only those datasets with * passed the applicability criteria for a given metric, which includes minimal variability and performance threshold set for each metric.
+In addition, not all metrics passed the additional criteria for inclusion in the final score calculation, as explained in the paper, and marked with ** in the table. This includes context specificity and robustness in stability analysis. 
 
 The evaluation metrics expect the inferred network to be in the form of an AnnData object with specific format as explained here. 
 It should be noted that the metric currently evaluate only the **top TF-gene pairs**, currently limited to **50,000 edges**, ranked by their assigned weight.  
@@ -28,32 +29,42 @@ The inferred network should have a tabular format with the following columns:
 
 See `resources/grn_benchmark/prior/collectri.h5ad` for an example of the expected format.
 
-Running GRN evaluation using standard pipeline
-----------------------------------------
 
-To run the evalution for a given GRN and dataset, use the following command:
+
+Running GRN evaluation without docker
+----------------------------------------
+Considering that Docker is not supported by certtain systems, you can run the evaluation without Docker by following these steps:
 
 ```bash
-bash scripts/run_grn_evaluation.sh --prediction=<inferred GRN (e.g.collectri.h5ad)> --save_dir=<e.g.output/> --dataset=<e.g. replogle> --build_images=<true or false. true for the first time running> 
+bash src/metrics/all_metrics/run_local.sh --dataset <dataset_name> --prediction=<inferred GRN (e.g.collectri.h5ad)> --score <output_score_file.h5ad> --num_workers <number_of_workers>
 ```
 
 example command:
 
 ```bash
-bash scripts/run_grn_evaluation.sh --prediction=resources/grn_models/op/collectri.h5ad --save_dir=output/ --dataset=op --build_images=true 
+bash src/metrics/all_metrics/run_local.sh --dataset op --prediction=resources/grn_models/op/collectri.h5ad --score=output_score_file.h5ad --num_workers=20
 ```
 
+If you are evaluating a new GRN model, which is not part of geneRNIB, make you to generate the consensus prior file for the dataset you are evaluating on. 
 
-Running GRN evaluation without docker
+```bash
+bash scripts/prior/run_consensus.sh --dataset op --new_model {new_grn_model_file.h5ad}
+```
+
+This will add your model to the previous ones and create the new consensus prior file needed for evaluation.
+
+Running GRN evaluation using standard pipeline
 ----------------------------------------
-Considering that Docker is not supported by certtain systems, you can run the evaluation without Docker by following these steps:
+
+To run the evalution for a given GRN and dataset, use the following command:
 
 ```bash
-bash src/metrics/all_metrics/run_local.sh --dataset <dataset_name> --prediction=<inferred GRN (e.g.collectri.h5ad)> --score <output_score_file.h5ad> --num_workers <number_of_workers>
+bash scripts/run_grn_evaluation.sh --prediction=<inferred GRN (e.g.collectri.h5ad)> --save_dir=<e.g.output/> --dataset=<e.g. replogle> --build_images=<true or false. true for the first time running> 
 ```
 
 example command:
 
 ```bash
-bash src/metrics/all_metrics/run_local.sh --dataset op --prediction=resources/grn_models/op/collectri.h5ad --score=output_score_file.h5ad --num_workers=20
-```
+bash scripts/run_grn_evaluation.sh --prediction=resources/grn_models/op/collectri.h5ad --save_dir=output/ --dataset=op --build_images=true 
+```
+

docs/build/html/_sources/leaderboard.rst.txt

@@ -1,22 +1,22 @@
 
 Leaderboard
 =================
-.. The overal comparitive performance of the integrated GRN inference methods is summarized in the leaderboard below. 
+The overal comparitive performance of the integrated GRN inference methods is summarized in the leaderboard below. It should be noted that not all metrics count towards the final score, as some metrics did not pass the applicability criteria. See `here <evaluation.html>`_ for more details on the metrics applicability.
 
-.. .. image:: images/leaderboard.png
-..    :width: 90%
-..    :align: center
-.. ----
+.. image:: images/summary_figure.png
+   :width: 100%
+   :align: center
+----
 
-The individual performance of the methods on each dataset is summarized below.
+The individual performance of the methods on each dataset is summarized below. 
 
 .. image:: images/raw_scores_op.png
    :width: 70%
    :align: center
 ----
 
 .. image:: images/raw_scores_nakatake.png
-   :width: 60%
+   :width: 70%
    :align: center
 ----
 
@@ -25,10 +25,6 @@ The individual performance of the methods on each dataset is summarized below.
    :align: center
 ----
 
-.. image:: images/raw_scores_adamson.png
-   :width: 70%
-   :align: center
-----
 
 .. image:: images/raw_scores_replogle.png
    :width: 70%
@@ -45,8 +41,13 @@ The individual performance of the methods on each dataset is summarized below.
    :align: center
 ----
 
-.. image:: images/raw_scores_ibd.png
-   :width: 90%
+.. image:: images/raw_scores_ibd_uc.png
+   :width: 70%
+   :align: center
+----
+
+.. image:: images/raw_scores_ibd_cd.png
+   :width: 70%
    :align: center
 ----
 

docs/build/html/dataset.html

@@ -3,7 +3,7 @@
 <!DOCTYPE html>
 <html class="writer-html5" lang="en" data-content_root="./">
 <head>
-  <meta charset="utf-8" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+  <meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
 
   <meta name="viewport" content="width=device-width, initial-scale=1.0" />
   <title>Datasets &mdash; OpenProblems GRN benchmarking 0.1 documentation</title>
@@ -82,10 +82,13 @@
 
   <section id="datasets">
 <h1>Datasets<a class="headerlink" href="#datasets" title="Link to this heading"></a></h1>
-<p>The list of datasets integrated into geneRNIB is provided below with their perturbation signatures as well as the type of perturbation used in each dataset.</p>
-<a class="reference internal image-reference" href="_images/datasets.png"><img alt="_images/datasets.png" class="align-center" src="_images/datasets.png" style="width: 80%;" /></a>
+<p>The list of datasets integrated into geneRNIB is provided below:</p>
+<a class="reference internal image-reference" href="_images/datasets.png"><img alt="_images/datasets.png" class="align-center" src="_images/datasets.png" style="width: 50%;" />
+</a>
+<hr class="docutils" />
+<a class="reference internal image-reference" href="_images/table_datasets_summary.png"><img alt="_images/table_datasets_summary.png" class="align-center" src="_images/table_datasets_summary.png" style="width: 100%;" />
+</a>
 <hr class="docutils" />
-<p>All datasets provide RNA data, while the <cite>OPSCA</cite> and <cite>IBD</cite> datasets also includes scATAC data.</p>
 <p>You need <cite>awscli</cite> to download the datasets.</p>
 <section id="downloading-the-main-datasets">
 <h2>Downloading the main datasets<a class="headerlink" href="#downloading-the-main-datasets" title="Link to this heading"></a></h2>