Version 1.0

## General
- Runs are sorted now and only "visible" directories are shown.
- Mechanics to select runs changed.
    - Runs can be selected across different working directories now.
    - Another section to display selected runs.
    - Press on directory name changes the working directory directly. Makes it easier to navigate.
    There's also a button to go to parent directory.
    - Internally, a run has two hashes now
        - `id`: Based on prefix and path/name. This hash is used to select runs now. No need to
        distinguish between run and grouped run anymore.
        - `hash`: Changes based on content.
- If DeepCAVE was not started before, the path of execution is used as working directory.
- Shows the errors now if a run could not be converted.
- Increased stability for static plugins.
- Updated CLI immensively. Added --open, --n_workers and improved --config (both relative
and absolute paths are working now). DeepCAVE starts now using only `deepcave`.
- Improved run cache performance. Each input uses a single file for the output.
- Improved performance overall.
- Improved the API mode drastically. 
- Improved mapping of original and encoded data.
- Configurations are clickable now.
- API changes in `AbstractRun`.
- Jobs in sidebar are clickable and removeable now.

## Documentation
- Updated texts and images.
- Documentation of plugins are shown in the dashboard now (converted from rst to md).
- Added research questions.

## Plugins
- Overview: Redesigned with barplot and heatmap for statuses. Also includes configspace now.
- Added PDP again.
- Merged LPI and fANOVA.
- Added dynamic texts to overview and budget correlation.
- Added help buttons.
- Added configuration footprint.
- Configurations display code now.

## Code related
- Added dash-extensions so that multiple outputs, no outputs and trigger can be used.
- Added global notification.
- Cleaned-up run handler immensely.
- Logs are better readable.
- Improved config loading and add development config.
- Renamed and moved things (Groups, Status, Objectives, ...).
- Objective is a dataclass now.
- Added REFRESH_RATE and SAVE_IMAGES to profile.

## Bugfixes
- Files are no longer shown in run selection.
- Runs are reloaded from disk in worker now.
- Internal runs are updated now if they changed.
- Results from static plugins are saved under the right path now.

Co-authored-by: dwoiwode <[email protected]>
Co-authored-by: eddiebergman <[email protected]>

Author: 3 people

.github/workflows/pytest.yml

@@ -18,9 +18,8 @@ jobs:
 
       - name: Install dependencies
         run: |
-          conda env update --file environment.yml --name base
-          pip install .
-          pip install .[dev]
+          conda install -c anaconda swig
+          make install-dev
           
       - name: Run pytest 
         run: pytest tests

.gitignore

@@ -168,3 +168,6 @@ examples/others
 examples/record/_*
 datasets
 lightning_logs
+vendors
+
+file_system_store

CHANGELOG.md

@@ -1,3 +1,60 @@
+# Version 1.0
+
+## General
+- Runs are sorted now and only "visible" directories are shown.
+- Mechanics to select runs changed.
+    - Runs can be selected across different working directories now.
+    - Another section to display selected runs.
+    - Press on directory name changes the working directory directly. Makes it easier to navigate.
+    There's also a button to go to parent directory.
+    - Internally, a run has two hashes now
+        - `id`: Based on prefix and path/name. This hash is used to select runs now. No need to
+        distinguish between run and grouped run anymore.
+        - `hash`: Changes based on content.
+- If DeepCAVE was not started before, the path of execution is used as working directory.
+- Shows the errors now if a run could not be converted.
+- Increased stability for static plugins.
+- Updated CLI immensively. Added --open, --n_workers and improved --config (both relative
+and absolute paths are working now). DeepCAVE starts now using only `deepcave`.
+- Improved run cache performance. Each input uses a single file for the output.
+- Improved performance overall.
+- Improved the API mode drastically. 
+- Improved mapping of original and encoded data.
+- Configurations are clickable now.
+- API changes in `AbstractRun`.
+- Jobs in sidebar are clickable and removeable now.
+
+## Documentation
+- Updated texts and images.
+- Documentation of plugins are shown in the dashboard now (converted from rst to md).
+- Added research questions.
+
+## Plugins
+- Overview: Redesigned with barplot and heatmap for statuses. Also includes configspace now.
+- Added PDP again.
+- Merged LPI and fANOVA.
+- Added dynamic texts to overview and budget correlation.
+- Added help buttons.
+- Added configuration footprint.
+- Configurations display code now.
+
+## Code related
+- Added dash-extensions so that multiple outputs, no outputs and trigger can be used.
+- Added global notification.
+- Cleaned-up run handler immensely.
+- Logs are better readable.
+- Improved config loading and add development config.
+- Renamed and moved things (Groups, Status, Objectives, ...).
+- Objective is a dataclass now.
+- Added REFRESH_RATE and SAVE_IMAGES to profile.
+
+## Bugfixes
+- Files are no longer shown in run selection.
+- Runs are reloaded from disk in worker now.
+- Internal runs are updated now if they changed.
+- Results from static plugins are saved under the right path now.
+
+
 # Version 0.3
 
 ## General

MANIFEST.in

@@ -1,8 +1,15 @@
 include LICENSE
 include requirements.txt
+include deepcave/assets/custom.css
 include deepcave/py.typed
-include start.sh
 include deepcave/utils/logging.yml
+include deepcave/worker.py
+include deepcave/server.py
+include deepcave/open.py
+include deepcave/start.sh
+include docs/plugins/*
+
+recursive-include deepcave/docs *.rst
 
 prune tests
 prune examples

Makefile

@@ -30,7 +30,7 @@ help:
 
 PYTHON ?= python
 PYTEST ?= python -m pytest
-PIP ?= python -m pip
+PIP ?= pip
 MAKE ?= make
 BLACK ?= black
 ISORT ?= isort
@@ -119,15 +119,13 @@ publish: clean build
 	@echo "Test with the following:"
 	@echo "* Create a new virtual environment to install the uplaoded distribution into"
 	@echo "* Run the following:"
-	@echo
-	@echo "        pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ ${NAME}"
+	@echo "--- pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ ${NAME}"
 	@echo
 	@echo "* Run this to make sure it can import correctly, plus whatever else you'd like to test:"
-	@echo
-	@echo "        python -c 'import ${PACKAGE_NAME}'"
+	@echo "--- python -c 'import ${PACKAGE_NAME}'"
 	@echo
 	@echo "Once you have decided it works, publish to actual pypi with"
-	@echo
+	@echo "--- python -m twine upload dist/*"
 
 # Clean up any builds in ./dist as well as doc, if present
 clean: clean-build clean-doc

README.md

@@ -1,47 +1,64 @@
 # DeepCAVE
 
-DeepCAVE has two main contributions:
-- Recording runs and
-- Visualizing and evaluating trials of a run to get better insights into the AutoML process.
+DeepCAVE is a visualization and analysis tool for AutoML (especially for the sub-problem
+hyperparameter optimization) runs. The framework is programmed on top of Dash and therefore
+entirely interactive. Multiple and diverse plugins make it possible to efficiently generate insights
+and bring the human back in the loop. Moreover, the powerful run interface and the modularized
+plugin structure allow extending the tool at any time effortlessly.
+
+![Configuration Footprint](docs/images/plugins/configuration_footprint.png)
+
+Following features are provided:
+- Interactive Dashboard (completely written in Python) to self-analyze optimization runs/processes.
+- Analyzing while optimizing (run changes are automatically detected).
+- A large collection of plugins to explore multiple areas like performance, hyperparameter and
+budget analysis.
+- Save your runs using DeepCAVE's native recorder.
+- Support for many optimizers using converter (e.g., DeepCAVE, SMAC and BOHB).
+- Select runs directly from a working directory in the interface.
+- Select groups of runs for combined analysis.
+- Modularized plugin structure with access to selected runs/groups to provide maximal flexibility.
+- Asynchronous execution of expensive plugins and caching of their results.
+- Help buttons and integrated documentation in the interface helps you to understand the plugins.
+- Use the matplotlib mode to customize and save the plots for your publication.
+- The API mode gives you full access to the code, while you do not have to interact with the 
+interface. Otherwise, you can also make use of the raw data, provided by every plugin.
 
 
 ## Installation
 
-First, make sure you have
-[swig](https://www.dev2qa.com/how-to-install-swig-on-macos-linux-and-windows/) and
-[redis-server](https://flaviocopes.com/redis-installation/) installed on your
-computer.
+First, make sure you have [redis-server](https://flaviocopes.com/redis-installation/) installed on
+your computer.
 
-If you are on an Non-Intel Mac you have to add
-```
-export DISABLE_SPRING=true
-export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
+Afterwards, follow the instructions to install DeepCAVE:
+```bash
+conda create -n DeepCAVE python=3.9
+conda activate DeepCAVE
+conda install -c anaconda swig
+pip install DeepCAVE
 ```
-to your ```~/.bash_profile``` to enable multi-processing.
 
-Afterwards, follow the instructions:
-```
+If you want to contribute to DeepCAVE use the following steps instead:
+```bash
 git clone https://github.com/automl/DeepCAVE.git
-cd DeepCAVE
-conda env create -f environment.yml
+conda create -n DeepCAVE python=3.9
 conda activate DeepCAVE
-make install
-```
-to your ```~/.bash_profile``` to enable multi-processing.
-
-If you want to contribute to DeepCAVE also install the dev packages:
-```
+conda install -c anaconda swig
 make install-dev
 ```
 
+Please visit the [documentation](https://automl.github.io/DeepCAVE/main/installation.html) to get
+further help (e.g. if you can not install redis server or you are on a mac).
+
 
 ## Recording
 
-In the following, a minimal example is given to show the simplicity yet powerful API to record runs.
+A minimal example is given to show the simplicity yet powerful API to record runs.
+However, existing optimizers like BOHB, SMAC, Auto-Sklearn, Auto-PyTorch are supported natively.
 
-```
+```python
 import ConfigSpace as CS
-from deep_cave import Recorder, Objective
+from deepcave import Recorder, Objective
 
 
 configspace = CS.ConfigurationSpace(seed=0)
@@ -58,19 +75,21 @@ with Recorder(configspace, objectives=[accuracy, mse]) as r:
             r.start(config, budget)
             # Your code goes here
             r.end(costs=[0.5, 0.5])
-````
+```
 
 
 ## Visualizing and Evaluating
 
-The webserver as well as the queue/workers can be started by running
-```
-deepcave --start
+The webserver as well as the queue/workers can be started by simply running:
+```bash
+deepcave --open
 ```
 
-Visit `http://127.0.0.1:8050/` to get started. The following figures gives
-you a first impression of DeepCAVE. You can find more screenshots
-in the documentation.
+If you specify `--open` your webbrowser automatically opens at `http://127.0.0.1:8050/`.
+You can find more arguments and information (like using custom configurations) in the
+[documentation](https://automl.github.io/DeepCAVE/main/getting_started.html).
+
 
-![interface](docs/images/plugins/pareto_front.png)
+## Citation
 
+Currently, DeepCAVE is under review.

configs/local.py

@@ -0,0 +1,6 @@
+from deepcave.config import Config as C
+
+
+class Config(C):
+    DEBUG = True
+    REFRESH_RATE: int = 2000

configs/server.py

@@ -0,0 +1,11 @@
+from deepcave.config import Config as C
+
+
+class Config(C):
+    DEBUG = False
+
+    REDIS_PORT = 6379
+    REDIS_ADDRESS = "redis://localhost"
+
+    DASH_PORT = 8050
+    DASH_ADDRESS = "re"  # If you are connected to a remote server sass@se, the address is "re".