Merge pull request #243 from daisybio/cli

Command Line Interfaces

Author: PascalIversen

Dockerfile

@@ -37,9 +37,7 @@ COPY --from=builder /usr/local/bin /usr/local/bin
 # Copy all relevant code
 
 COPY drevalpy ./drevalpy
-COPY create_report.py ./
 COPY README.md ./
-COPY run_suite.py ./
 COPY pyproject.toml ./
 COPY poetry.lock ./
 

README.md

@@ -72,15 +72,21 @@ pip install poetry-plugin-export
 poetry install
 ```
 
+Check your installation by running in your console:
+
+```bash
+drevalpy --help
+```
+
 ## Quickstart
 
 To run models from the catalog, you can run:
 
 ```bash
-python run_suite.py --run_id my_first_run --models NaiveTissueMeanPredictor NaiveDrugMeanPredictor --baselines NaiveMeanEffectsPredictor --dataset TOYv1 --test_mode LCO
+drevalpy --run_id my_first_run --models NaiveTissueMeanPredictor NaiveDrugMeanPredictor --dataset TOYv1 --test_mode LCO
 ```
 
-This will train our baseline models which just predict the drug or tissue means or the mean drug and cell line effects.
+This will download a small toy drug response dataset, train our baseline models which just predict the drug or tissue means or the mean drug and cell line effects.
 It will evaluate in "LCO" which is the leave-cell-line-out splitting strategy using 7 fold cross validation.
 The results will be stored in
 
@@ -91,10 +97,10 @@ results/my_first_run/TOYv1/LCO
 You can visualize them using
 
 ```bash
-python create_report.py --run_id my_first_run --dataset TOYv1
+drevalpy-report --run_id my_first_run --dataset TOYv1
 ```
 
-This will create an index.html file which you can open in your web browser.
+This will create an index.html file in the results directory which you can open in your web browser.
 
 You can also run a drug response experiment using Python:
 
@@ -103,56 +109,38 @@ from drevalpy.experiment import drug_response_experiment
 from drevalpy.models import MODEL_FACTORY
 from drevalpy.datasets import AVAILABLE_DATASETS
 
-naive_mean = MODEL_FACTORY["NaiveMeanEffectsPredictor"]
-rf = MODEL_FACTORY["RandomForest"]
-simple_nn = MODEL_FACTORY["SimpleNeuralNetwork"]
+from drevalpy.experiment import drug_response_experiment
+
+naive_mean = MODEL_FACTORY["NaivePredictor"] # a naive model that just predicts the training mean
+enet = MODEL_FACTORY["ElasticNet"] # An Elastic Net based on drug fingerprints and gene expression of 1000 landmark genes
+simple_nn = MODEL_FACTORY["SimpleNeuralNetwork"] # A neural network based on drug fingerprints and gene expression of 1000 landmark genes
 
-toyv2 = AVAILABLE_DATASETS["TOYv2"](path_data="data", measure="LN_IC50_curvecurator")
+toyv1 = AVAILABLE_DATASETS["TOYv1"](path_data="data")
 
 drug_response_experiment(
-            models=[rf, simple_nn],
-            baselines=[naive_mean],
-            response_data=toyv2,
-            metric="RMSE",
-            n_cv_splits=7,
-            test_mode="LCO",
-            run_id="my_second_run",
-            path_data="data",
-            hyperparameter_tuning=False,
-        )
+            models=[enet, simple_nn],
+            baselines=[naive_mean], # Ablation studies and robustness tests are not run for baselines.
+            response_data=toyv1,
+            n_cv_splits=2, # the number of cross validation splits. Should be higher in practice :)
+            test_mode="LCO", # LCO means Leave-Cell-Line out. This means that the test and validation splits only contain unseed cell lines.
+            run_id="my_first_run",
+            path_data="data", # where the downloaded drug response and feature data is stored
+            path_out="results", # results are stored here :)
+            hyperparameter_tuning=False) # if True (default), hyperparameters of the models and baselines are tuned.
 ```
 
 This will run the Random Forest and Simple Neural Network models on the CTRPv2 dataset, using the Naive Mean Effects Predictor as a baseline. The results will be stored in `results/my_second_run/CTRPv2/LCO`.
 To obtain evaluation metrics, you can use:
 
 ```python
-from drevalpy.visualization.utils import parse_results, prep_results, write_results
-import pathlib
-
-# load data, evaluate per CV run
-(
-        evaluation_results,
-        evaluation_results_per_drug,
-        evaluation_results_per_cell_line,
-        true_vs_pred,
-    ) = parse_results(path_to_results="results/my_second_run", dataset='TOYv2')
-# reformat, calculate normalized metrics
-(
-        evaluation_results,
-        evaluation_results_per_drug,
-        evaluation_results_per_cell_line,
-        true_vs_pred,
-    ) = prep_results(
-        evaluation_results, evaluation_results_per_drug, evaluation_results_per_cell_line, true_vs_pred, pathlib.Path("data")
-    )
-
-write_results(
-        path_out="results/my_second_run",
-        eval_results=evaluation_results,
-        eval_results_per_drug=evaluation_results_per_drug,
-        eval_results_per_cl=evaluation_results_per_cell_line,
-        t_vs_p=true_vs_pred,
-    )
+from drevalpy.visualization.create_report import create_report
+
+create_report(
+    run_id="my_first_run",
+    dataset=toyv1.dataset_name,
+    path_data= "data",
+    result_path="results",
+)
 ```
 
 We recommend the use of our Nextflow pipeline for computational demanding runs and for improved reproducibility.

create_report.py

@@ -40,13 +40,13 @@ How to set up your development environment
 
    .. code:: console
 
-      $ python run_suite.py --run_id my_first_run --models NaiveDrugMeanPredictor ElasticNet --dataset TOYv1 --test_mode LCO
+      $ drevalpy --run_id my_first_run --models NaiveDrugMeanPredictor ElasticNet --dataset TOYv1 --test_mode LCO
 
 6. Visualize the results by running the following command:
 
    .. code:: console
 
-      $ python create_report.py --run_id my_first_run --dataset TOYv1
+      $ drevalpy-report --run_id my_first_run --dataset TOYv1
 
 How to test the project
 -----------------------

docs/contributing.rst

@@ -78,4 +78,4 @@ To install DrEvalPy from source, clone the repository and install the package us
     pip install poetry-plugin-export
     poetry install
 
-Now, you can test the functionality by referring to the `Quickstart <./quickstart.html>`_ documentation.
+Now, you can test the functionality quickly via `drevalpy --help`. Or take a look at the `Quickstart <./quickstart.html>`_ documentation.

docs/installation.rst

@@ -8,7 +8,7 @@ dataset with the LCO test mode.
 
 .. code-block:: bash
 
-    python run_suite.py --run_id my_first_run --models NaiveTissueMeanPredictor NaiveDrugMeanPredictor --baselines NaiveMeanEffectsPredictor --dataset TOYv1 --test_mode LCO
+    drevalpy --run_id my_first_run --models NaiveTissueMeanPredictor NaiveDrugMeanPredictor --baselines NaiveMeanEffectsPredictor --dataset TOYv1 --test_mode LCO
 
 This will train the three baseline models to predict LN_IC50 values of our Toy dataset which is a subset of CTRPv2.
 It will evaluate in "LCO" which is the leave-cell-line-out splitting strategy
@@ -23,14 +23,15 @@ You can visualize them using
 
 .. code-block:: bash
 
-    python create_report.py --run_id my_first_run --dataset TOYv1
+    drevalpy-report --run_id my_first_run --dataset TOYv1
 
 This creates an index.html file which you can open in your browser to see the results of your run.
 
 We recommend the use of our nextflow pipeline for computational demanding runs and for improved reproducibility. No
 knowledge of nextflow is required to run it. The nextflow pipeline is available on the `nf-core GitHub
 <https://github.com/nf-core/drugresponseeval.git>`_, the documentation can be found `here <https://nf-co.re/drugresponseeval/dev/>`_.
 
+-  Want to test if your own model outperforms the baselines? See `Run Your Model <./runyourmodel.html>`_.
 -  Discuss usage, development and issues on `GitHub <https://github.com/daisybio/drevalpy>`_.
 -  Check the `Contributor Guide <./contributing.html>`_ if you want to participate in developing.
 -  If you use drevalpy for your work, `please cite us <./reference.html>`_.

docs/quickstart.rst

@@ -217,7 +217,7 @@ Update the ``MULTI_DRUG_MODEL_FACTORY`` if your model is a global model for mult
 Now you can run your model using the DrEvalPy pipeline. cd to the drevalpy root directory and run the following command:
 
 .. code-block:: shell
-    python -m run_suite.py --model YourModel --dataset CTRPv2 --data_path data
+    drevalpy --model YourModel --dataset CTRPv2 --data_path data
 
 
 To contribute the model, so that the community can build on it, please also write appropriate tests in ``tests/models`` and documentation in ``docs/``
@@ -543,4 +543,4 @@ Now you can run the model using the DrEvalPy pipeline.
 To run the model, navigate to the DrEvalPy root directory and execute the following command:
 .. code-block:: shell
 
-    python -m run_suite.py --model ProteomicsRandomForest --dataset CTRPv2 --data_path data
+    drevalpy --model ProteomicsRandomForest --dataset CTRPv2 --data_path data