Docker implementation, pyproject.toml and frozen environment

.github/workflows/black.yml

@@ -0,0 +1,21 @@
+name: Code style (Black)
+
+on:
+  pull_request:
+
+jobs:
+  black:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install Black
+        run: |
+          python -m pip install --upgrade pip
+          pip install black==24.8.0
+      - name: Check formatting
+        run: black --check --diff .

.github/workflows/docs.yml

@@ -0,0 +1,40 @@
+name: 📖 Build & Deploy Docs
+
+on:
+  push:
+    branches:
+      - main
+      - dev
+      - 'yep/**'
+    tags:
+      - 'v*.*.*'
+
+jobs:
+  build_docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python Environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install ".[docs]"
+
+      - name: Build HTML docs
+        working-directory: docs
+        run: make html
+
+      - name: Deploy to GitHub Pages
+        uses: peaceiris/actions-gh-pages@v4
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: docs/build/html
+          publish_branch: gh-pages
+

.gitignore

@@ -3,4 +3,6 @@
 demos/outputs/simulation/**/*
 demos/outputs/figures
 .DS_Store
-.vscode
+.vscode
+build/
+*.egg-info

Dockerfile

@@ -1,17 +1,14 @@
-FROM python:3.8
+FROM continuumio/miniconda3:latest
 
-ADD . /base/
+# Create conda environment
+WORKDIR /tmp
+COPY environment.yml conda-linux-64.lock ./
+RUN conda create --name demos-env --file conda-linux-64.lock \
+	&& conda clean --all --yes \
+	&& rm conda-linux-64.lock environment.yml
 
-WORKDIR /base
+# Copy the code
+COPY ./demos /demos
+WORKDIR /demos
 
-RUN python setup.py develop
-
-WORKDIR /base/demos_urbansim
-
-RUN apt-get update && \
-	apt-get install -y gcc libhdf5-serial-dev
-
-RUN pip install -r requirements.txt
-
-WORKDIR /base/demos_urbansim
-ENTRYPOINT ["python", "-u", "simulate.py", "-c", "-cf", "custom", "-l"]
+ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "demos-env", "python", "-u", "simulate.py", "-cfg", "config.toml"]

README.md

@@ -1,81 +1,76 @@
 # Demographic Microsimulator(DEMOS)
 
->**TODO:** The previous description from Urbansim it outdated. Add description of DEMOS, aligning with the code (paper).
+[![Docs](https://github.com/NREL/DEMOS_NREL/actions/workflows/docs.yml/badge.svg)](https://nrel.github.io/DEMOS_NREL/)
 
 
-## i. setup guide
+## Usage
+### Docker Container
+The docker image for demos is stored in `registry/demos:latest`. The input data and configuration file are fed to the container through volumes. Alternatively, we provide a `docker-compose` workflow that can be used.
 
-This repository contains only code and configuration/setup files necessary 
+For running the `docker-compose` workflow:
+```bash
+DEMOS_CONFIG_PATH=<path-to-config> DEMOS_DATA_DIR=<path-to-data-dir> docker-compose up
+```
 
+By default `DEMOS_CONFIG_PATH` is set to `./demos_config.toml` and `DEMOS_DATA_DIR` is set to `./data`, so if `data` and `demos_config.toml` are part of the current directory, no additional input is needed.
 
-1. clone this repository into your local machine
+Alternatively,
+```bash
+docker run --volume <path-to-config>:/demos/config.toml:ro --volume <path-to-data-dir>:/demos/data --platform=linux/amd64 demos
+```
 
-2. create environment if needed
+#### IMPORTANT for MacOS and Windows users
+> Docker imposes a global limit on how much RAM containers can allocate. DEMOS easily surpases those limits, so in order to run DEMOS in Docker, users need to access the Docker Desktop GUI and `Preferences → Resources → Memory → Increase it (at least 16-20gb)`
 
-	```
-	conda create --name {myenv} python=3.8
-	```
+#### Building the docker image (development only)
+```bash
+docker build -t demos:0.0.1 --platform=linux/amd64 -f Dockerfile .
+```
 
-3. Enter into DEMOS environment
+### From Source
 
+1. Clone this repository
 	```
-	conda activate {myenv}
+	git clone https://github.com/NREL/DEMOS_NREL.git
 	```
 
-4. Install all packages if needed
+1. Create a virtual environment.
 
+	**If using conda**, prefer the provided `.lock` files
 	```
-	pip install -r requirements.txt
+	conda create --name demos-env --file conda-{system}.lock
 	```
 
-5. Download input data [data_nrel.zip](https://app.box.com/s/tox2nflumia2g4n6rk2i0navca9pskep). You may also refer to the data description [here](https://cloud.urbansim.com/docs/general/documentation/urbansim%20block%20model%20data.html)
+	Alternatively, create a `python 3.10` environment and install dependencies
+	```
+	conda create -n demos-env python=3.10
+	conda activate demos-env
+	pip install .
+	```
 
-6. Put all files of input data into `DEMOS_NREL/demos/data`
+## Running DEMOS
 
-7. Run DEMOS. First, enter into `DEMOS_NREL/demos`, then run:
+DEMOS requires a series of input tables. Example tables are provided [here](https://app.box.com/s/tox2nflumia2g4n6rk2i0navca9pskep) for internal NREL use. It is recommended to store all the input values in the folder `data` in root of the project, but absolute values can be used by specifying them in the configuration file. You may also refer to the data description [here](https://cloud.urbansim.com/docs/general/documentation/urbansim%20block%20model%20data.html)
 
-	```
-	python -u simulate.py -c -y 2011 -cf custom -l -r 06197001 -s 100
-	```
+To run demos:
+```
+python simulate.py -cfg {configuration_file}
+```
 
-    the following are the arguments used in the above command:
-	```
-	  -r region_code, --region_code region_code
-							region fips code
-	  -y year, --year year  forecast year to simulate to
-	  -c, --calibrated      whether to run with calibrated coefficients
-	  -cf calibrated_folder, --calibrated_folder calibrated_folder
-							name of the calibration folder to read configs from
-	  -sg, --segmented      run with segmented lcms
-	  -l, --all_local       no cloud access whatsoever
-	  -i input_year, --input_year input_year
-							input data (base) year
-	  -f freq_interval, --freq_interval freq_interval
-							intra-simulation frequency interval
-	  -o output_fname, --output_fname output_fname
-							output file name
-	  -ss skim_source, --skim_source skim_source
-							skims format, e.g. "beam", "polaris"
-	  -rm, --random_matching random matching in the single to x model to 
-							 reduce computational time due to 
-							 the matchmaking process
-	  -s --random_seed random seed settng
-	```
 
-8. simulation results
-the demos simulation will produce the following sets of data and results:
-  - a synthetic population file showing the evolution of the synthetic population throughout the simulation years. the file should be named `model_data_<scenario_name_output_year>.h5` in directory `DEMOS_NREL/demos/data`.
-  - series of aggregated statistics for the population size, number of households, household size distribution, gender distribution, number of births, number of mortalities, number of student enrollments, number of total marriages, number of total divorces, the age distribution of the synthetic population, and income distribution for each simulation year. The files are located at `DEMOS_NREL/demos/outputs/simulation`
 
-## ii. project structure
->**TODO:** this part still need to be complemented
+A default configuration file is provided in `configuration/demos_config.toml`. The `[[tables]]` entries outline tables to be loaded. For example, we can load the `persons` and `households` table from an H5 source:
 
-the main folder of this repository contains several python scripts that contain the different steps necessary to import, process, and run the demos framework. the following is a description of the different folder and scripts used to run the demos simulation
+```toml
+[[tables]]
+file_type = "h5"
+table_name = "persons"
+filepath = "../data/custom_mpo_06197001_model_data.h5"
+h5_key = "persons"
 
-1. the `configs\` directory: this folder contains the different `.yaml` configuration files to run each of the demos and urbansim models. the configuration files for each region are located in subdirectories with the name of the region
-2. the `data\` directory: contains all the data needed to run the simulation
-3. `variables.py`: this script defines all the temporary variables needed to run the models. each variable is created as an orca column.
-4. `datasources.py`: this script imports all the necessary data for the specified simulation region and create simulation output folders, if needed.
-5. `models.py`: this script defines all the models as orca steps and defines all pre-processing and post-processing steps needed for each of the models.
-6. `simulate.py`: this script defines all the simulation parameters and runs the rest of the scripts desribed above.
-7. the `outputs\` directory: contains the different results produced by the simulation. simulation results for each region are stored in their respective subdirectories.
+[[tables]]
+file_type = "h5"
+table_name = "households"
+filepath = "../data/custom_mpo_06197001_model_data.h5"
+h5_key = "households"
+```

conda-linux-64.lock

@@ -0,0 +1,80 @@
+# Generated by conda-lock.
+# platform: linux-64
+# input_hash: c8cb734b10aa888d5aa6e5f91c835f2ff8aefab478c1ff85765c00c7f5394087
+@EXPLICIT
+https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
+https://conda.anaconda.org/conda-forge/noarch/nomkl-1.0-h5ca1d4c_0.tar.bz2#9a66894dfd07c4510beb6b3f9672ccc0
+https://conda.anaconda.org/conda-forge/noarch/python_abi-3.10-8_cp310.conda#05e00f3b21e88bb3d658ac700b2ce58c
+https://conda.anaconda.org/conda-forge/noarch/tzdata-2025b-h78e105d_0.conda#4222072737ccff51314b5ece9c7d6f5a
+https://conda.anaconda.org/conda-forge/noarch/ca-certificates-2025.8.3-hbd8a1cb_0.conda#74784ee3d225fc3dca89edb635b4e5cc
+https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.44-h1423503_1.conda#0be7c6e070c19105f966d3758448d018
+https://conda.anaconda.org/conda-forge/linux-64/libgomp-15.1.0-h767d61c_4.conda#3baf8976c96134738bba224e9ef6b1e5
+https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
+https://conda.anaconda.org/conda-forge/linux-64/libgcc-15.1.0-h767d61c_4.conda#f406dcbb2e7bef90d793e50e79a2882b
+https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.34.5-hb9d3cd8_0.conda#f7f0d6cc2dc986d42ac2689ec88192be
+https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.7.1-hecca717_0.conda#4211416ecba1866fab0c6470986c22d6
+https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.6-h2dba641_1.conda#ede4673863426c0883c0063d853bbd85
+https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-15.1.0-h69a702a_4.conda#28771437ffcd9f3417c66012dc49a3be
+https://conda.anaconda.org/conda-forge/linux-64/libgfortran5-15.1.0-hcea5267_4.conda#8a4ab7ff06e4db0be22485332666da0f
+https://conda.anaconda.org/conda-forge/linux-64/liblzma-5.8.1-hb9d3cd8_2.conda#1a580f7796c7bf6393fddb8bbbde58dc
+https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hb9d3cd8_1.conda#d864d34357c3b65a4b731f78c0801dc4
+https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-15.1.0-h8f9b012_4.conda#3c376af8888c386b9d3d1c2701e2f3ab
+https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.1-hb9d3cd8_2.conda#edb0dca6bc32e4f4789199455a1dbeb8
+https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.5-h2d0b736_3.conda#47e340acb35de30501a76c7c799c41d7
+https://conda.anaconda.org/conda-forge/linux-64/openssl-3.5.2-h26f9b46_0.conda#ffffb341206dd0dab0c36053c048d621
+https://conda.anaconda.org/conda-forge/linux-64/yaml-0.2.5-h280c20c_3.conda#a77f85f77be52ff59391544bfe73390a
+https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h4bc722e_7.conda#62ee74e96c5ebb0af99386de58cf9553
+https://conda.anaconda.org/conda-forge/linux-64/keyutils-1.6.1-h166bdaf_0.tar.bz2#30186d27e2c9fa62b45fb1476b7200e3
+https://conda.anaconda.org/conda-forge/linux-64/libaec-1.1.4-h3f801dc_0.conda#01ba04e414e47f95c03d6ddd81fd37be
+https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20250104-pl5321h7949ede_0.conda#c277e0a4d549b03ac1e9d6cbbe3d017b
+https://conda.anaconda.org/conda-forge/linux-64/libev-4.33-hd590300_2.conda#172bf1cd1ff8629f2b1179945ed45055
+https://conda.anaconda.org/conda-forge/linux-64/libgfortran-15.1.0-h69a702a_4.conda#53e876bc2d2648319e94c33c57b9ec74
+https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.50.4-h0c1763c_0.conda#0b367fad34931cb79e0d6b7e5c06bb1c
+https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.1-hcf80075_0.conda#eecce068c7e4eddeb169591baac20ac4
+https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-15.1.0-h4852527_4.conda#2d34729cbc1da0ec988e57b13b712067
+https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda#40b61aab5c7ba9ff276c41cfffe6b80b
+https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.36-hd590300_1.conda#5aa797f8787fe7a17d1b0821485b5adc
+https://conda.anaconda.org/conda-forge/linux-64/lz4-c-1.10.0-h5888daf_1.conda#9de5350a85c4a20c685259b889aa6393
+https://conda.anaconda.org/conda-forge/linux-64/readline-8.2-h8c095d6_2.conda#283b96675859b20a825f8fa30f311446
+https://conda.anaconda.org/conda-forge/linux-64/snappy-1.2.2-h03e3b7b_0.conda#3d8da0248bdae970b4ade636a104b7f5
+https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_hd72426e_102.conda#a0116df4f4ed05c303811a837d5b39d8
+https://conda.anaconda.org/conda-forge/linux-64/zlib-ng-2.2.5-hde8ca8f_0.conda#1920c3502e7f6688d650ab81cd3775fd
+https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.7-hb8e6e7a_2.conda#6432cb5d4ac0046c3ac0a8a0f95842f9
+https://conda.anaconda.org/conda-forge/linux-64/blosc-1.21.6-he440d0b_1.conda#2c2fae981fd2afd00812c92ac47d023d
+https://conda.anaconda.org/conda-forge/linux-64/c-blosc2-2.15.2-h3122c55_1.conda#2bc8d76acd818d7e79229f5157d5c156
+https://conda.anaconda.org/conda-forge/linux-64/krb5-1.21.3-h659f571_0.conda#3f43953b7d3fb3aaa1d0d0723d91e368
+https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.64.0-h161d5f1_0.conda#19e57602824042dfd0446292ef90488b
+https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.30-pthreads_h94d23a6_1.conda#7e2ba4ca7e6ffebb7f7fc2da2744df61
+https://conda.anaconda.org/conda-forge/linux-64/python-3.10.18-hd6af730_0_cpython.conda#4ea0c77cdcb0b81813a0436b162d7316
+https://conda.anaconda.org/conda-forge/linux-64/libblas-3.9.0-33_h59b9bed_openblas.conda#b43d5de8fe73c2a5fb2b43f45301285b
+https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.14.1-h332b0f4_0.conda#45f6713cb00f124af300342512219182
+https://conda.anaconda.org/conda-forge/noarch/loguru-0.7.3-pyh707e725_0.conda#49647ac1de4d1e4b49124aedf3934e02
+https://conda.anaconda.org/conda-forge/noarch/packaging-25.0-pyh29332c3_1.conda#58335b26c38bf4a20f399384c33cbcf9
+https://conda.anaconda.org/conda-forge/noarch/py-cpuinfo-9.0.0-pyhd8ed1ab_1.conda#46830ee16925d5ed250850503b5dc3a8
+https://conda.anaconda.org/conda-forge/noarch/python-tzdata-2025.2-pyhd8ed1ab_0.conda#88476ae6ebd24f39261e0854ac244f33
+https://conda.anaconda.org/conda-forge/noarch/pytz-2025.2-pyhd8ed1ab_0.conda#bc8e3267d44011051f2eb14d22fb0960
+https://conda.anaconda.org/conda-forge/linux-64/pyyaml-6.0.2-py310h89163eb_2.conda#fd343408e64cf1e273ab7c710da374db
+https://conda.anaconda.org/conda-forge/noarch/setuptools-80.9.0-pyhff2d567_0.conda#4de79c071274a53dcaf2a8c749d1499e
+https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda#3339e3b65d58accf4ca4fb8748ab16b3
+https://conda.anaconda.org/conda-forge/noarch/toml-0.10.2-pyhd8ed1ab_1.conda#b0dd904de08b7db706167240bf37b164
+https://conda.anaconda.org/conda-forge/noarch/toolz-1.0.0-pyhd8ed1ab_1.conda#40d0ed782a8aaa16ef248e68c06c168d
+https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.14.1-pyhe01879c_0.conda#e523f4f1e980ed7a4240d7e27e9ec81f
+https://conda.anaconda.org/conda-forge/noarch/wheel-0.45.1-pyhd8ed1ab_1.conda#75cb7132eb58d97896e173ef12ac9986
+https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.14.4-nompi_h2d575fe_105.conda#d76fff0092b6389a12134ddebc0929bd
+https://conda.anaconda.org/conda-forge/linux-64/libcblas-3.9.0-33_he106b2a_openblas.conda#28052b5e6ea5bd283ac343c5c064b950
+https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.9.0-33_h7ac8fdf_openblas.conda#e598bb54c4a4b45c3d83c72984f79dbb
+https://conda.anaconda.org/conda-forge/noarch/pip-25.2-pyh8b19718_0.conda#dfce4b2af4bfe90cdcaf56ca0b28ddf5
+https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.9.0.post0-pyhe01879c_2.conda#5b8d21249ff20967101ffa321cab24e8
+https://conda.anaconda.org/conda-forge/noarch/typing-extensions-4.14.1-h4440ef1_0.conda#75be1a943e0a7f99fcf118309092c635
+https://conda.anaconda.org/conda-forge/noarch/typing-inspection-0.4.1-pyhd8ed1ab_0.conda#e0c3cd765dc15751ee2f0b03cd015712
+https://conda.anaconda.org/conda-forge/noarch/annotated-types-0.7.0-pyhd8ed1ab_1.conda#2934f256a8acfe48f6ebb4fce6cde29c
+https://conda.anaconda.org/conda-forge/linux-64/numpy-2.2.6-py310hefbff90_0.conda#b0cea2c364bf65cd19e023040eeab05d
+https://conda.anaconda.org/conda-forge/linux-64/pydantic-core-2.33.2-py310hbcd0ec0_0.conda#6b210a72e9e1b1cb6d30b266b84ca993
+https://conda.anaconda.org/conda-forge/linux-64/numexpr-2.10.2-py310hdb6e06b_100.conda#0d869e0096ccd43eeb3d387f2f65d1a7
+https://conda.anaconda.org/conda-forge/linux-64/pandas-2.3.1-py310h0158d43_0.conda#94eb2db0b8f769a1e554843e3586504d
+https://conda.anaconda.org/conda-forge/noarch/patsy-1.0.1-pyhd8ed1ab_1.conda#ee23fabfd0a8c6b8d6f3729b47b2859d
+https://conda.anaconda.org/conda-forge/noarch/pydantic-2.11.7-pyh3cfb1c2_0.conda#1b337e3d378cde62889bb735c024b7a2
+https://conda.anaconda.org/conda-forge/linux-64/scipy-1.15.2-py310h1d65ade_0.conda#8c29cd33b64b2eb78597fa28b5595c8d
+https://conda.anaconda.org/conda-forge/linux-64/pytables-3.10.1-py310h431dcdc_4.conda#4edbf66abeab03413c5be7b7fa0a366e
+https://conda.anaconda.org/conda-forge/linux-64/statsmodels-0.14.5-py310haaf2d95_0.conda#92b4b51b83f2cfded298f1b8c7a99e32
+https://conda.anaconda.org/conda-forge/noarch/orca-1.8-pyhd8ed1ab_0.tar.bz2#e0c47f71ede4fc985baf2ad94b8cd099