Skip to content

Commit d63c879

Browse files
yamilbknsuyamilbknsu
authored
Merge pull request #77 from NREL/dev
2 parents cc558e4 + 7890b3d commit d63c879

File tree

153 files changed

+10197
-927556
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

153 files changed

+10197
-927556
lines changed

‎.github/workflows/black.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
name: Code style (Black)
2+
3+
on:
4+
pull_request:
5+
6+
jobs:
7+
black:
8+
runs-on: ubuntu-latest
9+
timeout-minutes: 10
10+
steps:
11+
- uses: actions/checkout@v4
12+
- uses: actions/setup-python@v5
13+
with:
14+
python-version: "3.11"
15+
cache: pip
16+
- name: Install Black
17+
run: |
18+
python -m pip install --upgrade pip
19+
pip install black==24.8.0
20+
- name: Check formatting
21+
run: black --check --diff .

‎.github/workflows/docs.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
name: 📖 Build & Deploy Docs
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- dev
8+
- 'yep/**'
9+
tags:
10+
- 'v*.*.*'
11+
12+
jobs:
13+
build_docs:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
with:
18+
fetch-depth: 0
19+
20+
- name: Set up Python Environment
21+
uses: actions/setup-python@v4
22+
with:
23+
python-version: '3.10'
24+
25+
- name: Install dependencies
26+
run: |
27+
python -m pip install --upgrade pip
28+
pip install ".[docs]"
29+
30+
- name: Build HTML docs
31+
working-directory: docs
32+
run: make html
33+
34+
- name: Deploy to GitHub Pages
35+
uses: peaceiris/actions-gh-pages@v4
36+
with:
37+
github_token: ${{ secrets.GITHUB_TOKEN }}
38+
publish_dir: docs/build/html
39+
publish_branch: gh-pages
40+

‎.gitignore‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
11
**/__pycache__
22
.idea
33
demos/outputs/simulation/**/*
4-
demos/outputs/figures
4+
demos/outputs/figures
5+
.DS_Store
6+
.vscode
7+
build/
8+
*.egg-info

‎Dockerfile‎

Lines changed: 11 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,14 @@
1-
FROM python:3.8
1+
FROM continuumio/miniconda3:latest
22

3-
ADD . /base/
3+
# Create conda environment
4+
WORKDIR /tmp
5+
COPY environment.yml conda-linux-64.lock ./
6+
RUN conda create --name demos-env --file conda-linux-64.lock \
7+
&& conda clean --all --yes \
8+
&& rm conda-linux-64.lock environment.yml
49

5-
WORKDIR /base
10+
# Copy the code
11+
COPY ./demos /demos
12+
WORKDIR /demos
613

7-
RUN python setup.py develop
8-
9-
WORKDIR /base/demos_urbansim
10-
11-
RUN apt-get update && \
12-
apt-get install -y gcc libhdf5-serial-dev
13-
14-
RUN pip install -r requirements.txt
15-
16-
WORKDIR /base/demos_urbansim
17-
ENTRYPOINT ["python", "-u", "simulate.py", "-c", "-cf", "custom", "-l"]
14+
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "demos-env", "python", "-u", "simulate.py", "-cfg", "config.toml"]

‎README.md‎

Lines changed: 49 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,81 +1,75 @@
11
# Demographic Microsimulator(DEMOS)
22

3-
>**TODO:** The previous description from Urbansim it outdated. Add description of DEMOS, aligning with the code (paper).
3+
[![Docs](https://github.com/NREL/DEMOS_NREL/actions/workflows/docs.yml/badge.svg)](https://nrel.github.io/DEMOS_NREL/)
44

5+
## Usage
6+
### Docker Container
7+
The docker image for demos is stored in `registry/demos:latest`. The input data and configuration file are fed to the container through volumes. Alternatively, we provide a `docker-compose` workflow that can be used.
58

6-
## i. setup guide
9+
For running the `docker-compose` workflow:
10+
```bash
11+
DEMOS_CONFIG_PATH=<path-to-config> DEMOS_DATA_DIR=<path-to-data-dir> docker-compose up
12+
```
713

8-
This repository contains only code and configuration/setup files necessary
14+
By default `DEMOS_CONFIG_PATH` is set to `./demos_config.toml` and `DEMOS_DATA_DIR` is set to `./data`, so if `data` and `demos_config.toml` are part of the current directory, no additional input is needed.
915

16+
Alternatively,
17+
```bash
18+
docker run --volume <path-to-config>:/demos/config.toml:ro --volume <path-to-data-dir>:/demos/data --platform=linux/amd64 demos
19+
```
1020

11-
1. clone this repository into your local machine
21+
#### IMPORTANT for MacOS and Windows users
22+
> Docker imposes a global limit on how much RAM containers can allocate. DEMOS easily surpases those limits, so in order to run DEMOS in Docker, users need to access the Docker Desktop GUI and `Preferences → Resources → Memory → Increase it (at least 16-20gb)`
1223
13-
2. create environment if needed
24+
#### Building the docker image (development only)
25+
```bash
26+
docker build -t demos:0.0.1 --platform=linux/amd64 -f Dockerfile .
27+
```
1428

29+
### From Source
30+
31+
1. Clone this repository
1532
```
16-
conda create --name {myenv} python=3.8
33+
git clone https://github.com/NREL/DEMOS_NREL.git
1734
```
1835

19-
3. Enter into DEMOS environment
36+
1. Create a virtual environment.
2037

38+
**If using conda**, prefer the provided `.lock` files
2139
```
22-
conda activate {myenv}
40+
conda create --name demos-env --file conda-{system}.lock
2341
```
2442

25-
4. Install all packages if needed
26-
43+
Alternatively, create a `python 3.10` environment and install dependencies
2744
```
28-
pip install -r requirements.txt
45+
conda create -n demos-env python=3.10
46+
conda activate demos-env
47+
pip install .
2948
```
3049

31-
5. Download input data [data_nrel.zip](https://app.box.com/s/tox2nflumia2g4n6rk2i0navca9pskep). You may also refer to the data description [here](https://cloud.urbansim.com/docs/general/documentation/urbansim%20block%20model%20data.html)
50+
## Running DEMOS
3251

33-
6. Put all files of input data into `DEMOS_NREL/demos/data`
52+
DEMOS requires a series of input tables. Example tables are provided [here](https://app.box.com/s/tox2nflumia2g4n6rk2i0navca9pskep) for internal NREL use. It is recommended to store all the input values in the folder `data` in root of the project, but absolute values can be used by specifying them in the configuration file. You may also refer to the data description [here](https://cloud.urbansim.com/docs/general/documentation/urbansim%20block%20model%20data.html)
3453

35-
7. Run DEMOS. First, enter into `DEMOS_NREL/demos`, then run:
54+
To run demos:
55+
```
56+
python simulate.py -cfg {configuration_file}
57+
```
3658

37-
```
38-
python -u simulate.py -c -y 2011 -cf custom -l -r 06197001 -s 100
39-
```
40-
41-
the following are the arguments used in the above command:
42-
```
43-
-r region_code, --region_code region_code
44-
region fips code
45-
-y year, --year year forecast year to simulate to
46-
-c, --calibrated whether to run with calibrated coefficients
47-
-cf calibrated_folder, --calibrated_folder calibrated_folder
48-
name of the calibration folder to read configs from
49-
-sg, --segmented run with segmented lcms
50-
-l, --all_local no cloud access whatsoever
51-
-i input_year, --input_year input_year
52-
input data (base) year
53-
-f freq_interval, --freq_interval freq_interval
54-
intra-simulation frequency interval
55-
-o output_fname, --output_fname output_fname
56-
output file name
57-
-ss skim_source, --skim_source skim_source
58-
skims format, e.g. "beam", "polaris"
59-
-rm, --random_matching random matching in the single to x model to
60-
reduce computational time due to
61-
the matchmaking process
62-
-s --random_seed random seed settng
63-
```
6459

65-
8. simulation results
66-
the demos simulation will produce the following sets of data and results:
67-
- a synthetic population file showing the evolution of the synthetic population throughout the simulation years. the file should be named `model_data_<scenario_name_output_year>.h5` in directory `DEMOS_NREL/demos/data`.
68-
- series of aggregated statistics for the population size, number of households, household size distribution, gender distribution, number of births, number of mortalities, number of student enrollments, number of total marriages, number of total divorces, the age distribution of the synthetic population, and income distribution for each simulation year. The files are located at `DEMOS_NREL/demos/outputs/simulation`
6960

70-
## ii. project structure
71-
>**TODO:** this part still need to be complemented
61+
A default configuration file is provided in `configuration/demos_config.toml`. The `[[tables]]` entries outline tables to be loaded. For example, we can load the `persons` and `households` table from an H5 source:
7262

73-
the main folder of this repository contains several python scripts that contain the different steps necessary to import, process, and run the demos framework. the following is a description of the different folder and scripts used to run the demos simulation
63+
```toml
64+
[[tables]]
65+
file_type = "h5"
66+
table_name = "persons"
67+
filepath = "../data/custom_mpo_06197001_model_data.h5"
68+
h5_key = "persons"
7469

75-
1. the `configs\` directory: this folder contains the different `.yaml` configuration files to run each of the demos and urbansim models. the configuration files for each region are located in subdirectories with the name of the region
76-
2. the `data\` directory: contains all the data needed to run the simulation
77-
3. `variables.py`: this script defines all the temporary variables needed to run the models. each variable is created as an orca column.
78-
4. `datasources.py`: this script imports all the necessary data for the specified simulation region and create simulation output folders, if needed.
79-
5. `models.py`: this script defines all the models as orca steps and defines all pre-processing and post-processing steps needed for each of the models.
80-
6. `simulate.py`: this script defines all the simulation parameters and runs the rest of the scripts desribed above.
81-
7. the `outputs\` directory: contains the different results produced by the simulation. simulation results for each region are stored in their respective subdirectories.
70+
[[tables]]
71+
file_type = "h5"
72+
table_name = "households"
73+
filepath = "../data/custom_mpo_06197001_model_data.h5"
74+
h5_key = "households"
75+
```

‎conda-linux-64.lock‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Generated by conda-lock.
2+
# platform: linux-64
3+
# input_hash: c8cb734b10aa888d5aa6e5f91c835f2ff8aefab478c1ff85765c00c7f5394087
4+
@EXPLICIT
5+
https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
6+
https://conda.anaconda.org/conda-forge/noarch/nomkl-1.0-h5ca1d4c_0.tar.bz2#9a66894dfd07c4510beb6b3f9672ccc0
7+
https://conda.anaconda.org/conda-forge/noarch/python_abi-3.10-8_cp310.conda#05e00f3b21e88bb3d658ac700b2ce58c
8+
https://conda.anaconda.org/conda-forge/noarch/tzdata-2025b-h78e105d_0.conda#4222072737ccff51314b5ece9c7d6f5a
9+
https://conda.anaconda.org/conda-forge/noarch/ca-certificates-2025.8.3-hbd8a1cb_0.conda#74784ee3d225fc3dca89edb635b4e5cc
10+
https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.44-h1423503_1.conda#0be7c6e070c19105f966d3758448d018
11+
https://conda.anaconda.org/conda-forge/linux-64/libgomp-15.1.0-h767d61c_4.conda#3baf8976c96134738bba224e9ef6b1e5
12+
https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
13+
https://conda.anaconda.org/conda-forge/linux-64/libgcc-15.1.0-h767d61c_4.conda#f406dcbb2e7bef90d793e50e79a2882b
14+
https://conda.anaconda.org/conda-forge/linux-64/c-ares-1.34.5-hb9d3cd8_0.conda#f7f0d6cc2dc986d42ac2689ec88192be
15+
https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.7.1-hecca717_0.conda#4211416ecba1866fab0c6470986c22d6
16+
https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.6-h2dba641_1.conda#ede4673863426c0883c0063d853bbd85
17+
https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-15.1.0-h69a702a_4.conda#28771437ffcd9f3417c66012dc49a3be
18+
https://conda.anaconda.org/conda-forge/linux-64/libgfortran5-15.1.0-hcea5267_4.conda#8a4ab7ff06e4db0be22485332666da0f
19+
https://conda.anaconda.org/conda-forge/linux-64/liblzma-5.8.1-hb9d3cd8_2.conda#1a580f7796c7bf6393fddb8bbbde58dc
20+
https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hb9d3cd8_1.conda#d864d34357c3b65a4b731f78c0801dc4
21+
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-15.1.0-h8f9b012_4.conda#3c376af8888c386b9d3d1c2701e2f3ab
22+
https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.1-hb9d3cd8_2.conda#edb0dca6bc32e4f4789199455a1dbeb8
23+
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.5-h2d0b736_3.conda#47e340acb35de30501a76c7c799c41d7
24+
https://conda.anaconda.org/conda-forge/linux-64/openssl-3.5.2-h26f9b46_0.conda#ffffb341206dd0dab0c36053c048d621
25+
https://conda.anaconda.org/conda-forge/linux-64/yaml-0.2.5-h280c20c_3.conda#a77f85f77be52ff59391544bfe73390a
26+
https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h4bc722e_7.conda#62ee74e96c5ebb0af99386de58cf9553
27+
https://conda.anaconda.org/conda-forge/linux-64/keyutils-1.6.1-h166bdaf_0.tar.bz2#30186d27e2c9fa62b45fb1476b7200e3
28+
https://conda.anaconda.org/conda-forge/linux-64/libaec-1.1.4-h3f801dc_0.conda#01ba04e414e47f95c03d6ddd81fd37be
29+
https://conda.anaconda.org/conda-forge/linux-64/libedit-3.1.20250104-pl5321h7949ede_0.conda#c277e0a4d549b03ac1e9d6cbbe3d017b
30+
https://conda.anaconda.org/conda-forge/linux-64/libev-4.33-hd590300_2.conda#172bf1cd1ff8629f2b1179945ed45055
31+
https://conda.anaconda.org/conda-forge/linux-64/libgfortran-15.1.0-h69a702a_4.conda#53e876bc2d2648319e94c33c57b9ec74
32+
https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.50.4-h0c1763c_0.conda#0b367fad34931cb79e0d6b7e5c06bb1c
33+
https://conda.anaconda.org/conda-forge/linux-64/libssh2-1.11.1-hcf80075_0.conda#eecce068c7e4eddeb169591baac20ac4
34+
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-15.1.0-h4852527_4.conda#2d34729cbc1da0ec988e57b13b712067
35+
https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda#40b61aab5c7ba9ff276c41cfffe6b80b
36+
https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.36-hd590300_1.conda#5aa797f8787fe7a17d1b0821485b5adc
37+
https://conda.anaconda.org/conda-forge/linux-64/lz4-c-1.10.0-h5888daf_1.conda#9de5350a85c4a20c685259b889aa6393
38+
https://conda.anaconda.org/conda-forge/linux-64/readline-8.2-h8c095d6_2.conda#283b96675859b20a825f8fa30f311446
39+
https://conda.anaconda.org/conda-forge/linux-64/snappy-1.2.2-h03e3b7b_0.conda#3d8da0248bdae970b4ade636a104b7f5
40+
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_hd72426e_102.conda#a0116df4f4ed05c303811a837d5b39d8
41+
https://conda.anaconda.org/conda-forge/linux-64/zlib-ng-2.2.5-hde8ca8f_0.conda#1920c3502e7f6688d650ab81cd3775fd
42+
https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.7-hb8e6e7a_2.conda#6432cb5d4ac0046c3ac0a8a0f95842f9
43+
https://conda.anaconda.org/conda-forge/linux-64/blosc-1.21.6-he440d0b_1.conda#2c2fae981fd2afd00812c92ac47d023d
44+
https://conda.anaconda.org/conda-forge/linux-64/c-blosc2-2.15.2-h3122c55_1.conda#2bc8d76acd818d7e79229f5157d5c156
45+
https://conda.anaconda.org/conda-forge/linux-64/krb5-1.21.3-h659f571_0.conda#3f43953b7d3fb3aaa1d0d0723d91e368
46+
https://conda.anaconda.org/conda-forge/linux-64/libnghttp2-1.64.0-h161d5f1_0.conda#19e57602824042dfd0446292ef90488b
47+
https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.30-pthreads_h94d23a6_1.conda#7e2ba4ca7e6ffebb7f7fc2da2744df61
48+
https://conda.anaconda.org/conda-forge/linux-64/python-3.10.18-hd6af730_0_cpython.conda#4ea0c77cdcb0b81813a0436b162d7316
49+
https://conda.anaconda.org/conda-forge/linux-64/libblas-3.9.0-33_h59b9bed_openblas.conda#b43d5de8fe73c2a5fb2b43f45301285b
50+
https://conda.anaconda.org/conda-forge/linux-64/libcurl-8.14.1-h332b0f4_0.conda#45f6713cb00f124af300342512219182
51+
https://conda.anaconda.org/conda-forge/noarch/loguru-0.7.3-pyh707e725_0.conda#49647ac1de4d1e4b49124aedf3934e02
52+
https://conda.anaconda.org/conda-forge/noarch/packaging-25.0-pyh29332c3_1.conda#58335b26c38bf4a20f399384c33cbcf9
53+
https://conda.anaconda.org/conda-forge/noarch/py-cpuinfo-9.0.0-pyhd8ed1ab_1.conda#46830ee16925d5ed250850503b5dc3a8
54+
https://conda.anaconda.org/conda-forge/noarch/python-tzdata-2025.2-pyhd8ed1ab_0.conda#88476ae6ebd24f39261e0854ac244f33
55+
https://conda.anaconda.org/conda-forge/noarch/pytz-2025.2-pyhd8ed1ab_0.conda#bc8e3267d44011051f2eb14d22fb0960
56+
https://conda.anaconda.org/conda-forge/linux-64/pyyaml-6.0.2-py310h89163eb_2.conda#fd343408e64cf1e273ab7c710da374db
57+
https://conda.anaconda.org/conda-forge/noarch/setuptools-80.9.0-pyhff2d567_0.conda#4de79c071274a53dcaf2a8c749d1499e
58+
https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda#3339e3b65d58accf4ca4fb8748ab16b3
59+
https://conda.anaconda.org/conda-forge/noarch/toml-0.10.2-pyhd8ed1ab_1.conda#b0dd904de08b7db706167240bf37b164
60+
https://conda.anaconda.org/conda-forge/noarch/toolz-1.0.0-pyhd8ed1ab_1.conda#40d0ed782a8aaa16ef248e68c06c168d
61+
https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.14.1-pyhe01879c_0.conda#e523f4f1e980ed7a4240d7e27e9ec81f
62+
https://conda.anaconda.org/conda-forge/noarch/wheel-0.45.1-pyhd8ed1ab_1.conda#75cb7132eb58d97896e173ef12ac9986
63+
https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.14.4-nompi_h2d575fe_105.conda#d76fff0092b6389a12134ddebc0929bd
64+
https://conda.anaconda.org/conda-forge/linux-64/libcblas-3.9.0-33_he106b2a_openblas.conda#28052b5e6ea5bd283ac343c5c064b950
65+
https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.9.0-33_h7ac8fdf_openblas.conda#e598bb54c4a4b45c3d83c72984f79dbb
66+
https://conda.anaconda.org/conda-forge/noarch/pip-25.2-pyh8b19718_0.conda#dfce4b2af4bfe90cdcaf56ca0b28ddf5
67+
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.9.0.post0-pyhe01879c_2.conda#5b8d21249ff20967101ffa321cab24e8
68+
https://conda.anaconda.org/conda-forge/noarch/typing-extensions-4.14.1-h4440ef1_0.conda#75be1a943e0a7f99fcf118309092c635
69+
https://conda.anaconda.org/conda-forge/noarch/typing-inspection-0.4.1-pyhd8ed1ab_0.conda#e0c3cd765dc15751ee2f0b03cd015712
70+
https://conda.anaconda.org/conda-forge/noarch/annotated-types-0.7.0-pyhd8ed1ab_1.conda#2934f256a8acfe48f6ebb4fce6cde29c
71+
https://conda.anaconda.org/conda-forge/linux-64/numpy-2.2.6-py310hefbff90_0.conda#b0cea2c364bf65cd19e023040eeab05d
72+
https://conda.anaconda.org/conda-forge/linux-64/pydantic-core-2.33.2-py310hbcd0ec0_0.conda#6b210a72e9e1b1cb6d30b266b84ca993
73+
https://conda.anaconda.org/conda-forge/linux-64/numexpr-2.10.2-py310hdb6e06b_100.conda#0d869e0096ccd43eeb3d387f2f65d1a7
74+
https://conda.anaconda.org/conda-forge/linux-64/pandas-2.3.1-py310h0158d43_0.conda#94eb2db0b8f769a1e554843e3586504d
75+
https://conda.anaconda.org/conda-forge/noarch/patsy-1.0.1-pyhd8ed1ab_1.conda#ee23fabfd0a8c6b8d6f3729b47b2859d
76+
https://conda.anaconda.org/conda-forge/noarch/pydantic-2.11.7-pyh3cfb1c2_0.conda#1b337e3d378cde62889bb735c024b7a2
77+
https://conda.anaconda.org/conda-forge/linux-64/scipy-1.15.2-py310h1d65ade_0.conda#8c29cd33b64b2eb78597fa28b5595c8d
78+
https://conda.anaconda.org/conda-forge/linux-64/pytables-3.10.1-py310h431dcdc_4.conda#4edbf66abeab03413c5be7b7fa0a366e
79+
https://conda.anaconda.org/conda-forge/linux-64/statsmodels-0.14.5-py310haaf2d95_0.conda#92b4b51b83f2cfded298f1b8c7a99e32
80+
https://conda.anaconda.org/conda-forge/noarch/orca-1.8-pyhd8ed1ab_0.tar.bz2#e0c47f71ede4fc985baf2ad94b8cd099

0 commit comments

Comments
 (0)