Merge remote-tracking branch 'origin/main' into electricitymarket2

Author: chrisyeh96

CONTRIBUTING.md

@@ -1,37 +1,38 @@
 # Contributing code
 
 1. Install [miniconda3](https://docs.conda.io/en/latest/miniconda.html).
-2. Create conda environment. Replace `XX` below with the name of the SustainGym environment you want to work on.
+2. (Optional, but recommended) If you are using a conda version `<=23.9.0`, set the conda solver to libmamba for faster dependency solving. Starting from conda version [`23.10.0`](https://github.com/conda/conda/releases/tag/23.10.0), libmamba is the default solver.
     ```bash
-    conda env update --file env_XX.yml --prune
+    conda config --set solver libmamba
     ```
-
-   If you are using RLLib with a GPU, you will also need to [configure TensorFlow for GPU](https://www.tensorflow.org/install/pip#4_gpu_setup):
+3. Clone the SustainGym repo, and enter the `sustaingym` directory.
     ```bash
-    mkdir -p $CONDA_PREFIX/etc/conda/activate.d
-    echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
-    echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
+    git clone https://github.com/chrisyeh96/sustaingym.git
+    cd sustaingym
     ```
-
-3. Make code modifications in a separate git branch
+4. Create conda environment. Replace `XX` below with the name of the SustainGym environment you want to work on. By default, the `env_XX.yml` environment files assume that you have a NVIDIA GPU. If you do not have a NVIDIA GPU, you may need to modify the `env_XX.yml` file.
+    ```bash
+    conda env update --file env_XX.yml --prune
+    ```
+5. Make code modifications in a separate git branch
     ```bash
     git checkout -b new_feature
     ```
-4. From repo root folder, run mypy type checker and fix any errors.
+6. From repo root folder, run mypy type checker and fix any errors.
     ```bash
     mypy sustaingym
     ```
-5. From repo root folder, run code linter and fix any linting errors.
+7. From repo root folder, run code linter and fix any linting errors.
     ```bash
     flake8 sustaingym
     ```
-6. Commit changes in git and push.
-7. Submit pull request on GitHub.
+8. Commit changes in git and push.
+9. Submit pull request on GitHub.
 
 
 ## Unit tests
 
-First, set your terminal directory to this repo's root directory. Next, make sure you have activated the appropriate conda environment for the SustainGym environment you want to test (e.g., `conda activate sustaingym_ev`). Finally, run the unit tests for the desired SustainGym environment:
+First, set your terminal directory to this repo's root directory. Next, make sure you have activated the appropriate conda environment for the SustainGym environment you want to test (_e.g._, `conda activate sustaingym_ev`). Finally, run the unit tests for the desired SustainGym environment:
 
 ```bash
 python -m unittest -v tests/test_evcharging.py

README.md

@@ -1,8 +1,8 @@
-# SustainGym: Reinforcement learning environments for sustainability applications
+# SustainGym: Reinforcement Learning Environments for Sustainable Energy Systems
 
 The lack of standardized benchmarks for reinforcement learning (RL) in sustainability applications has made it difficult to both track progress on specific domains and identify bottlenecks for researchers to focus their efforts on. We present SustainGym, a suite of environments designed to test the performance of RL algorithms on realistic sustainability tasks. These environments highlight challenges in introducing RL to real-world sustainability tasks, including physical constraints and distribution shift.
 
-[**Paper**](https://drive.google.com/file/d/1wrLGu2FCVOT_BvtDoudsz05zFG89r7dI/view?usp=drive_link)
+[**Paper**](https://openreview.net/forum?id=vZ9tA3o3hr)
 | [**Website**](https://chrisyeh96.github.io/sustaingym/)
 
 SustainGym contains both single-agent and multi-agent RL environments.

docs/buildingenv.md

@@ -3,13 +3,13 @@
 BuildingEnv considers the control of the heat flow in a multi-zone building so as to maintain a desired temperature setpoint.  Building temperature simulation uses first-principled physics models. Users can either choose from a pre-defined list of buildings (Office small, School primary, Apartment midrise, and Office large) and three climate types and cities (San Diego, Tucson, New York) provided by the Building Energy Codes Program or define a customized BuildingEnv environment by importing any self-defined EnergyPlus building models. Each episode runs for 1 day, with 5-minute time intervals ($H = 288$, $\tau = 5/60$ hours).
 
 ## Observation Space
-For a building with $M$ indoor zones, the state $s(t) \in \R^{3M+2}$ contains observable properties of the building environment at timestep $t$:
+For a building with $M$ indoor zones, the state $s(t) \in \R^{M+4}$ contains observable properties of the building environment at timestep $t$:
 
 $$
-s(t) = (T_1(t), ...,T_{M}(t), N_1(t), ..., N_{M}(t), Q_{1}^{GHI}(t), ..., Q_{M}^{GHI}(t), T_G(t), T_{E}(t)),
+    s(t) = (T_1(t), \dotsc, T_M(t), T_\mathrm{E}(t), T_\mathrm{G}(t), Q^\mathrm{GHI}(t), \bar{Q}^\mathrm{p}(t)),
 $$
 
-where $T_i(t)$ is zone $i$'s temperature at time step $t$, $N_i(t)$ is the number of occupants, $Q_{i}^{GHI}(t)$ is the heat gain from the solar irradiance, and $T_G(t)$ and $T_E(t)$ denote the ground and outdoor environment temperature. In practice, the agent may have access to all or part of the state variables for decision-making depending on the sensor setup. Note that the outdoor/ground temperature, room occupancy, and heat gain from solar radiance are time-varying uncontrolled variables from the environment.
+where $T_i(t)$ is zone $i$'s temperature at time step $t$, $\bar{Q}^\mathrm{p}(t)$ is the heat acquisition from occupant's activities, $Q^\mathrm{GHI}(t)$ is the heat gain from the solar irradiance, and $T_\mathrm{G}(t)$ and $T_\mathrm{E}(t)$ denote the ground and outdoor environment temperature. In practice, the agent may have access to all or part of the state variables for decision-making depending on the sensor setup. Note that the outdoor/ground temperature, room occupancy, and heat gain from solar radiance are time-varying uncontrolled variables from the environment.
 
 ## Action Space
 The action $a(t) \in [-1, 1]^M$ sets the controlled heating supplied to each of the $M$ zones, scaled to $[-1, 1]$.
@@ -18,13 +18,13 @@ The action $a(t) \in [-1, 1]^M$ sets the controlled heating supplied to each of
 The objective is to reduce energy consumption while keeping the temperature within a given comfort range. The default reward function is a weighted $\ell_2$ reward, defined as
 
 $$
-    r(t) = - (1-\beta) \|a(t)\|_2 - \beta \|T^{obj}(t)-T(t)\|_2
+    r(t) = - (1-\beta) \|a(t)\|_2 - \beta \|T^\mathrm{target}(t)-T(t)\|_2
 $$
 
-where $T^{obj}(t)=[T^{obj}_{1}(t),...,T^{obj}_{M}(t)]$ are the target temperatures and $T(t)=[T_{1}(t),...,T_{M}(t)]$ are the actual zonal temperature. BuildingEnv also allows users to customize reward functions using environment states $s(t)$, actions $a(t)$, target values $T^{obj}(t)$, and a weight term $\beta$. Users can also customize the reward function to take CO<sub>2</sub> emissions into consideration.
+where $T^\mathrm{target}(t)=[T^\mathrm{target}_{1}(t),\cdots,T^\mathrm{target}_{M}(t)]$ are the target temperatures and $T(t)=[T_1(t),\cdots,T_M(t)]$ are the actual zonal temperature. BuildingEnv also allows users to customize reward functions by changing the weight term $\beta$ or the parameter $p$ defining the $\ell_p$ norm. Users can also customize the reward function to take CO<sub>2</sub> emissions into consideration.
 
 ## Distribution Shift
-BuildingEnv features distribution shifts in the ambient outdoor temperature profile $T_E$ which varies with different seasons.
+BuildingEnv features distribution shifts in the ambient outdoor temperature profile $T_\mathrm{E}$ which varies with different seasons.
 
 ## Multiagent Setting
 In the multiagent setting for BuildingEnv, we treat each building as an independent agent whose action is the building's heat control decisions. It must coordinate with other building agents to maximize overall reward, which is the summation of each agent's reward. Each agent obtains either the global observation or individual building states.
@@ -33,11 +33,25 @@ In the multiagent setting for BuildingEnv, we treat each building as an independ
 
 ### Installation
 
-Coming soon!
+SustainGym is designed for Linux machines. SustainGym is hosted on [PyPI](https://pypi.org/project/sustaingym/) and can be installed with `pip`:
+
+```bash
+pip install sustaingym[building]
+```
 
 ### Using our training script
 
-Coming soon!
+1. Install [miniconda3](https://docs.conda.io/en/latest/miniconda-other-installer-links.html).
+2. (Optional, but recommended) If you are using a conda version `<=23.9.0`, set the conda solver to libmamba for faster dependency solving. Starting from conda version [`23.10.0`](https://github.com/conda/conda/releases/tag/23.10.0), libmamba is the default solver.
+    ```bash
+    conda config --set solver libmamba
+    ```
+3. Install the libraries necessary for runnning the BuildingEnv environment.
+    ```bash
+    conda env update --file env_building.yml --prune
+    ```
+
+More instructions coming soon!
 
 ### Custom RL Loop
 

docs/cogenenv.md

@@ -61,7 +61,7 @@ while not terminated:
 ### Using our training script
 
 1. Install [miniconda3](https://docs.conda.io/en/latest/miniconda-other-installer-links.html).
-2. (Optional) Set the conda solver to libmamba for faster dependency solving.
+2. (Optional, but recommended) If you are using a conda version `<=23.9.0`, set the conda solver to libmamba for faster dependency solving. Starting from conda version [`23.10.0`](https://github.com/conda/conda/releases/tag/23.10.0), libmamba is the default solver.
     ```bash
     conda config --set solver libmamba
     ```

docs/electricitymarketenv.md

@@ -34,7 +34,7 @@ ElectricityMarketEnv considers temporal distribution shifts, specifically in the
 ### Installation
 
 1. Install [miniconda3](https://docs.conda.io/en/latest/miniconda-other-installer-links.html).
-2. (Optional) Set the conda solver to libmamba for faster dependency solving.
+2. (Optional, but recommended) If you are using a conda version `<=23.9.0`, set the conda solver to libmamba for faster dependency solving. Starting from conda version [`23.10.0`](https://github.com/conda/conda/releases/tag/23.10.0), libmamba is the default solver.
     ```bash
     conda config --set solver libmamba
     ```

docs/evchargingenv.md

@@ -19,12 +19,14 @@ The reward function is a sum of three components: $r(t) = p(t) - c_V(t) - c_C(t)
 
 ### Installation
 
-SustainGym is hosted on [PyPI](https://pypi.org/project/sustaingym/) and can be installed with `pip`:
+SustainGym is designed for Linux machines. SustainGym is hosted on [PyPI](https://pypi.org/project/sustaingym/) and can be installed with `pip`:
 
 ```bash
 pip install sustaingym[ev]
 ```
 
+Specifically for `EVChargingEnv`, you also need to have a MOSEK license. You may either request a free [personal academic license](https://www.mosek.com/products/academic-licenses/), or a free 30-day [commercial trial license](https://www.mosek.com/products/trial/). The license file should be placed inside a folder called "mosek" under your home directory. Typically, that will be `~/mosek/mosek.lic`.
+
 ### Custom RL Loop
 
 ```python
@@ -48,7 +50,7 @@ while not terminated:
 ### Using our training script
 
 1. Install [miniconda3](https://docs.conda.io/en/latest/miniconda-other-installer-links.html).
-2. (Optional) Set the conda solver to libmamba for faster dependency solving.
+2. (Optional, but recommended) If you are using a conda version `<=23.9.0`, set the conda solver to libmamba for faster dependency solving. Starting from conda version [`23.10.0`](https://github.com/conda/conda/releases/tag/23.10.0), libmamba is the default solver.
     ```bash
     conda config --set solver libmamba
     ```
@@ -90,4 +92,4 @@ optional arguments:
 
 ## References
 
-[1] Z. J. Lee et al., "Adaptive Charging Networks: A Framework for Smart Electric Vehicle Charging," in _IEEE Transactions on Smart Grid_, vol. 12, no. 5, pp. 4339-4350, Sept. 2021, doi: 10.1109/TSG.2021.3074437. URL [https://ieeexplore.ieee.org/document/9409126](https://ieeexplore.ieee.org/document/9409126).
+[1] Z. J. Lee et al., "Adaptive Charging Networks: A Framework for Smart Electric Vehicle Charging," in _IEEE Transactions on Smart Grid_, vol. 12, no. 5, pp. 4339-4350, Sept. 2021, doi: 10.1109/TSG.2021.3074437. URL [https://ieeexplore.ieee.org/document/9409126](https://ieeexplore.ieee.org/document/9409126).

docs/index.md

@@ -11,7 +11,7 @@ A suite of environments designed to test the performance of RL algorithms on rea
 The lack of standardized benchmarks for reinforcement learning (RL) in sustainability applications has made it difficult to both track progress on specific domains and identify bottlenecks for researchers to focus their efforts on. We present **SustainGym**, a suite of environments designed to test the performance of RL algorithms on realistic sustainability tasks. These environments highlight challenges in introducing RL to real-world sustainability tasks, including physical constraints and distribution shift.
 
 <p>
-    <a href="https://drive.google.com/file/d/1wrLGu2FCVOT_BvtDoudsz05zFG89r7dI/view?usp=drive_link" class="btn btn-blue fs-5 mb-4 mb-md-0 mr-2">Read the Paper</a>
+    <a href="https://openreview.net/forum?id=vZ9tA3o3hr" class="btn btn-blue fs-5 mb-4 mb-md-0 mr-2">Read the Paper</a>
     <a href="https://github.com/chrisyeh96/sustaingym/" class="btn fs-5 mb-4 mb-md-0">View it on GitHub</a>
 </p>
 

env_building.yml

@@ -5,29 +5,41 @@
 # conda remove --name sustaingym_building --all
 #
 # Notes
-# - last updated: September 27, 2023
+# - Ray 2.8:
+#   - officially only supports up to Python 3.10 (see https://docs.ray.io/en/latest/ray-overview/installation.html)
+#   - only supports gymnasium 0.28.1 (see https://github.com/ray-project/ray/blob/ray-2.8.0/python/setup.py#L305)
+#   - officially seems to only supports pettingzoo 1.23.1 (see https://github.com/ray-project/ray/blob/ray-2.7.1/python/requirements/ml/rllib-test-requirements.txt),
+#     but empirically seems to work with pettingzoo 1.24.*
+#
+# last updated: November 13, 2023
 name: sustaingym_building
 channels:
+- pytorch
+- nvidia
 - conda-forge
 dependencies:
-- python=3.11
-- cvxpy       # for MPC controller
-- flake8      # Optional, for code linting
-- ipympl      # Optional, for Jupyter / VSCode notebooks
-- ipykernel   # Optional, for Jupyter / VSCode notebooks
+- python=3.10.*
+- cvxpy=1.4.*         # for MPC controller
+- flake8              # Optional, for code linting
+- ipympl              # Optional, for Jupyter / VSCode notebooks
+- ipykernel           # Optional, for Jupyter / VSCode notebooks
 - matplotlib
-- mypy        # Optional, for type checking
+- mypy                # Optional, for type checking
 - numpy
 - pandas
 - pip
 - pvlib
+- pytorch=2.1.*
 - scikit-learn
 - scipy
 - seaborn
-- tqdm        # Optional, for progress bars
+- tqdm                # Optional, for progress bars
+
+# for GPU. comment out for CPU-only.
+- pytorch-cuda=11.8   # for PyTorch 2
 
 - pip:
   - gymnasium==0.28.1
   - pettingzoo==1.24.1
-  - ray[rllib]==2.7.0
-  - stable_baselines3>=2.0.0
+  - ray[rllib]==2.8.*
+  - stable_baselines3>=2

env_cogen.yml

@@ -5,44 +5,49 @@
 # conda remove --name sustaingym_cogen --all
 #
 # Notes
-# - ray[rllib]==2.7 only supports gymnasium 0.28.1, pettingzoo 0.24.*
-# - last updated: October 30, 2023
+# - TensorFlow 2.14:
+#   - the GPU version only works with Python <=3.10 (see https://github.com/tensorflow/tensorflow/issues/61986)
+#   - TensorFlow 2.15 should fix this issue
+# - Ray 2.8:
+#   - officially only supports up to Python 3.10 (see https://docs.ray.io/en/latest/ray-overview/installation.html)
+#   - only supports gymnasium 0.28.1 (see https://github.com/ray-project/ray/blob/ray-2.8.0/python/setup.py#L305)
+#   - officially seems to only supports pettingzoo 1.23.1 (see https://github.com/ray-project/ray/blob/ray-2.8.0/python/requirements/ml/rllib-test-requirements.txt),
+#     but empirically seems to work with pettingzoo 1.24.*
+#
+# last updated: November 16, 2023
 name: sustaingym_cogen
 channels:
 - pytorch           # for pytorch
 - nvidia            # for pytorch-cuda
 - conda-forge
 dependencies:
-- python=3.11
+- python=3.10.*
 - flake8
 - ipympl              # for Jupyter / VSCode notebooks
 - ipykernel           # for Jupyter / VSCode notebooks
 - matplotlib
 - mypy
-- numpy
+- numpy=1.26.*
 - openpyxl            # for reading Excel files
 - pandas
 - pip
-- pytorch=2.0.1
+- pytorch=2.1.*
 - pytz=2023.3
 - seaborn
 - tqdm
 - xlrd                # for reading Excel files
 
-# for GPU
-- cudatoolkit=11.8.0  # for TensorFlow 2.12
-- pytorch-cuda=11.8   # for PyTorch 2.0
-
+# for GPU. comment out for CPU-only.
+- pytorch-cuda=11.8   # for PyTorch 2
 
 - pip:
   - gymnasium==0.28.1
   - pettingzoo==1.24.1
-  - "ray[rllib]==2.7.1"
-  - tensorflow==2.14.*
+  - ray[rllib]==2.8.*
+  - onnxruntime==1.16.*    # the ONNX model for CogenEnv is small and runs sufficiently fast on CPU
 
   # uncomment for CPU-only
-  # - onnxruntime
+  # - tensorflow==2.14.*
 
-  # for GPU
-  - nvidia-cudnn-cu11==8.6.0.163  # for TensorFlow 2.12
-  - onnxruntime-gpu
+  # for GPU. comment out for CPU-only.
+  - tensorflow[and-cuda]==2.14.*

env_ev.yml

@@ -5,45 +5,54 @@
 # conda remove --name sustaingym_ev --all
 #
 # Notes
-# - the 2 main bottlenecks are acnportal and ray
-#   1) acnportal v0.3.2 only supports up to Pandas 1.1,
-#      and Pandas 1.1 only supports up to Python 3.9
-#   2) ray[rllib]==2.7.0 only supports gymnasium 0.28.1, pettingzoo 0.24.*
-# - technically, ray(2.7.0) needs Pandas >= 1.3, but we can bypass this requirement by installing
-#   it through pip instead of conda
-# - last updated: September 27, 2023
+# - TensorFlow 2.14:
+#   - the GPU version only works with Python <=3.10 (see https://github.com/tensorflow/tensorflow/issues/61986)
+#   - TensorFlow 2.15 should fix this issue
+# - Ray 2.8:
+#   - officially only supports up to Python 3.10 (see https://docs.ray.io/en/latest/ray-overview/installation.html)
+#   - only supports gymnasium 0.28.1 (see https://github.com/ray-project/ray/blob/ray-2.8.0/python/setup.py#L305)
+#   - officially seems to only supports pettingzoo 1.23.1 (see https://github.com/ray-project/ray/blob/ray-2.8.0/python/requirements/ml/rllib-test-requirements.txt),
+#     but empirically seems to work with pettingzoo 1.24.*
+#
+# last updated: November 25, 2023
 name: sustaingym_ev
 channels:
 - pytorch           # for pytorch
 - nvidia            # for pytorch-cuda
 - mosek             # for mosek
 - conda-forge
 dependencies:
-- python=3.9.16
-- cudatoolkit=11.8.0      # for TensorFlow 2.12
-- cvxpy=1.3.1
-- flake8=6.0.0
+- python=3.10.*
+- cvxpy=1.4.*
+- flake8=6.1.*
 - ipympl=0.9.3            # for Jupyter / VSCode notebooks
 - ipykernel               # for Jupyter / VSCode notebooks
-- matplotlib=3.7.1
-- mosek=10.0.44
-- mypy=1.3.0
-- numpy=1.24.3
-- pandas=1.1.5            # acnportal 0.3.2 only works with Pandas 1.1
+- matplotlib=3.8.*
+- mosek=10.1.*
+- mypy=1.3.*
+- numpy=1.26.*
+- pandas=2.1.*
 - pip
-- pytorch=2.0.1
-- pytorch-cuda=11.8       # for PyTorch 2.0
+- pytorch=2.1.*
 - pytz=2023.3
-- requests=2.31.0
-- scikit-learn=1.1.1
-- scipy=1.10.1
-- seaborn=0.12.2
-- tqdm=4.65.0
+- requests=2.31.*
+- scikit-learn=1.1.*
+- scipy=1.11.*
+- seaborn=0.13.*
+- tqdm=4.66.*
+
+# for GPU. comment out for CPU-only.
+- pytorch-cuda=11.8       # for PyTorch 2
 
 - pip:
-  - acnportal==0.3.2
+  - acnportal>=0.3.3
   - gymnasium==0.28.1
   - pettingzoo==1.24.1
-  - "ray[rllib]==2.7.0"
-  - tensorflow==2.12.0
-  - nvidia-cudnn-cu11==8.6.0.163  # for TensorFlow 2.12
+  - ray[rllib]==2.8.*
+  - stable_baselines3>=2
+
+  # uncomment for CPU-only
+  # - tensorflow==2.14.*
+
+  # for GPU. comment out for CPU-only.
+  - tensorflow[and-cuda]==2.14.*