diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..4c6b28c --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +Postprocessing/.Rproj.user +Postprocessing/.Rhistory +Postprocessing/.RData +Postprocessing/.Ruserdata +Postprocessing/renv/library +docs/_build diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d4bb2cb --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/about_colombia/index.rst b/docs/about_colombia/index.rst new file mode 100644 index 0000000..98a9827 --- /dev/null +++ b/docs/about_colombia/index.rst @@ -0,0 +1,32 @@ +About Colombia +============== + +(Source - Wikipedia) + +Colombia is a country in South America with an insular region in North America. +The Caribbean Sea borders it to the north, Venezuela to the east, +Brazil to the southeast, Ecuador and Peru to the south, the Pacific Ocean to the west and +Panama to the northwest. Colombia comprises 32 departments and the Capital District of Bogotá, +the country's largest city. It covers an area of 1,141,748 square kilometers (440,831 sq mi), with a population of 50 million. + +The geography of Colombia is characterized by its six main natural regions that +present their unique characteristics from the Andes mountain range region +shared with Ecuador and Venezuela; the Pacific Coastal region shared with Panama and Ecuador; +the Caribbean coastal region shared with Venezuela and Panama; the Llanos (plains) +shared with Venezuela; the Amazon rainforest region shared with Venezuela, Brazil, +Peru and Ecuador; to the insular area, comprising islands in the Atlantic and Pacific oceans. +The climate of Colombia is characterized as being tropical, presenting variations within six natural regions +and depending on the altitude, temperature, humidity, winds and rainfall. +Colombia has diverse climate zones, including tropical rainforests, savannas, steppes, deserts and mountain climates. + +Colombia is the second most biologically diverse country on Earth. This biodiversity results +from Colombia's varied ecosystems—from the rich tropical rainforest to the coastal cloud +forests to the open savannas + +.. figure:: ../assets/colombia.gif + :alt: Map of Colombia + :align: center + :width: 405px + :height: 486px + + Image source https://geology.com/world/belize-satellite-image.shtml diff --git a/docs/assets/colombia.gif b/docs/assets/colombia.gif new file mode 100644 index 0000000..70a68f8 Binary files /dev/null and b/docs/assets/colombia.gif differ diff --git a/docs/assets/install_tools.PNG b/docs/assets/install_tools.PNG new file mode 100644 index 0000000..b474a3f Binary files /dev/null and b/docs/assets/install_tools.PNG differ diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000..1f9368d --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,52 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = 'GCBM Colombia' +copyright = '2022, moja global' +author = 'moja global' + + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'furo' + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000..63e8766 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,16 @@ +GCBM Colombia +============= + +This is the official documentation of the GCBM Colombia project. +This project is based on the `Standalone Template for the GCBM `_ + +**Disclaimer : The calculations performed in this repository are not official and do not represent the Government of Colombia in any way.** + +.. toctree:: + :hidden: + + about_colombia/index.rst + run_simulation/index.rst + postprocessing/index.rst + + diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000..922152e --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +if "%1" == "" goto help + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.http://sphinx-doc.org/ + exit /b 1 +) + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/postprocessing/index.rst b/docs/postprocessing/index.rst new file mode 100644 index 0000000..c389066 --- /dev/null +++ b/docs/postprocessing/index.rst @@ -0,0 +1,15 @@ +Postprocessing +============== + +After the simulation is complete, either by running the **run_all.bat** script or following the step-by-step procedure for postprocessing, the following steps are to be followed: + +1. Install `R `_ based on the OS + +2. Get the `path `_ to the R executable, e.g. **c:\\"Program Files"\R\R-4.2.1\bin\R.exe**, on Windows + +3. Navigate to the directory `Standalone_GCBM\\Postprocessing` and execute **c:\\"Program Files"\\R\\R-4.2.1\\bin\\R.exe CMD BATCH Summarize_DOM_Stocks.R** . + +This generates the plots `Figures\\Colombia_Sensitivity_TropicalPremontaneWetTransitiontoBasal-Pacific.png`, +`Figures\\Colombia_Sensitivity_TropicalMoist.png` and `Figures\\Colombia_Sensitivity_TropicalDry.png` + +4. The logs are written into **Summarize_DOM_Stocks.Rout**, missing R packages, if any, can be installed by following the steps mentioned here: https://www.datacamp.com/tutorial/r-packages-guide \ No newline at end of file diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 0000000..9e279e7 --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1 @@ +furo \ No newline at end of file diff --git a/docs/run_simulation/docker.rst b/docs/run_simulation/docker.rst new file mode 100644 index 0000000..ea740f2 --- /dev/null +++ b/docs/run_simulation/docker.rst @@ -0,0 +1,59 @@ +On Docker +========= + +Install Docker and docker-compose based on your `operating system `_ . + +Steps to run the Colombia simulation in a docker environment + +1. Clone the `docker-setup` branch of `GCBM Colombia `_ using the command + +.. code-block:: bash + + git clone -b docker-setup https://github.com/moja-global/GCBM.Colombia + + +2. Navigate into directory `GCBM.Colombia` + +.. code-block:: bash + + cd GCBM.Colombia + + +3. Use docker-compose to build the image + +.. code-block:: bash + + docker-compose up -d + + +If an error with the following message appears : + +`ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the version key and place your service definitions at the root of the file to use version 1. +For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/` + +Change the docker version in the file `./docker-compose.yml` to an appropriate version and run the command again. + + +4. Run the container using the command + +.. code-block:: bash + + docker exec -it gcbm-colombia /bin/bash + + +5. Inside the running docker container, run + +.. code-block:: bash + + cd /server/gcbm_project + + +6. Start the simulation using + +.. code-block:: bash + + /opt/gcbm/moja.cli --config_file gcbm_config.cfg --config_provider provider_config.json + + +If there are existing images cached on your machine, you may need to docker pull ghcr.io/moja-global/rest_api_gcbm:master +and build the container using `docker-compose -d --force-recreate` \ No newline at end of file diff --git a/docs/run_simulation/index.rst b/docs/run_simulation/index.rst new file mode 100644 index 0000000..2421517 --- /dev/null +++ b/docs/run_simulation/index.rst @@ -0,0 +1,27 @@ +Running the Colombia simulation +=============================== + +A local simulation of the GCBM model is run for the Colombia region can be run locally. + +The repository is located at: https://github.com/moja-global/GCBM.Colombia + +.. list-table:: Contents of the repository + :widths: 50 50 + :header-rows: 1 + + * - Folder + - Description + * - Standalone_GCBM + - Contains the inputs and scripts required to run the simulation + * - Postprocessing + - Postprocessing scripts written in R, these operate on the outputs generated by the simulation + * - Science + - Contains information about the science behind the GCBM model + + +.. toctree:: + :hidden: + + windows + docker + diff --git a/docs/run_simulation/windows.rst b/docs/run_simulation/windows.rst new file mode 100644 index 0000000..8a00e52 --- /dev/null +++ b/docs/run_simulation/windows.rst @@ -0,0 +1,199 @@ +On Windows +========== + +Prerequisites ++++++++++++++ + +1. Clone the `GCBM Colombia `_ repository + +2. Download the installation tools from releases: `install_tools `_, and unzip them into `Standalone_GCBM\\tools` folder. On unzipping, `Standalone_GCBM\\tools` will contain a folder tools with the folders `python_3_installer` and `VC_redist` . +The configuration is **strictly for Python3.7**, it will not work with other versions + + +.. figure:: ../assets/install_tools.PNG + :alt: Unzipped contents of install tools + :align: center + :width: 600px + + +Running the simulation (short version) +-------------------------------------- + +Navigate to `Standalone_GCBM`, edit `run_all.bat` and set the `GCBM_PYTHON path` and `PLATFORM` variables in the `USER CONFIGURATION` section near the top to the correct values for your system. + +Run `run_all.bat`, which performs the following preprocessing and simulation. + +Running the project (long version) +---------------------------------- + +This action involves running each stage of the simulation step by step. +It helps in understanding the sequence of steps, inputs, outputs and dependencies to each step. + +Step 1 +------ +tiler +Working directory : **Standalone_GCBM\\layers\\tiled** + +.. code-block:: bash + :caption: Command + + c:\\python37\\python.exe ..\\..\\tools\\Tiler\\tiler.py + +Description : + 1. Define all spatial layers needed for the simulation - can be raster or shapefile: + - bounding box + - age + - classifiers + - disturbance events (optional) + 2. Crops all layers to a bounding box and reprojects to WGS84 processes layers into GCBM tile/block/cell format + 3. Output is a number of zip files containing GCBM-format data plus a json file containing metadata and an optional attribute table + +Output : Logs written into **Standalone_GCBM\\logs\\tiler_log.txt** + + +Step 2 +------ + +Working directory : **Standalone_GCBM** + +.. code-block:: bash + :caption: Command + + tools\\Recliner2GCBM-x64\\Recliner2GCBM.exe -c input_database\\recliner2gcbm_config.json + +Description : + +1. Runs the command-line version of Recliner2GCBM (`tools\Recliner2GCBM-[x86\x64]\Recliner2GCBM.exe`) on the saved project configuration made by running the GUI tool (Recliner2GCBM-GUI.exe) +2. Note: The paths in the saved `recliner2gcbm_config.json` file are relative to the location of the json file + +Output : An SQLite database **gcbm_input.db** which contains all of the non-spatial data required to run the project parameters taken from a CBM3 ArchiveIndex database - disturbance matrices, default climate data, etc. + +Step 3 +------ +Working directory : **Standalone_GCBM** + +.. code-block:: bash + :caption: Command + + c:\\python37\\python.exe input_database\\add_species_vol_to_bio.py input_database\\gcbm_input.db + +Description : Adding a generic tropical species and Vol to Bio parameters + +Step 4 +------ +Working directory : **Standalone_GCBM** + +.. code-block:: bash + :caption: Command + + c:\\python37\\python.exe input_database\\modify_root_parameters.py input_database\\gcbm_input.db + +Description : Modify root parameters + +Step 5 +------ +Working directory : **Standalone_GCBM** + +.. code-block:: bash + :caption: Command + + c:\\python37\\python.exe input_database\\modify_decay_parameters.py input_database\\gcbm_input.db + +Description : Modify decay parameters + +Step 6 +------ +Working directory : **Standalone_GCBM** + +.. code-block:: bash + :caption: Command + + c:\\python37\\python.exe input_database\\modify_turnover_parameters.py input_database\\gcbm_input.db + +Description : Modify turnover parameters + +Step 7 +------ +Working directory : **Standalone_GCBM** + +.. code-block:: bash + :caption: Command + + c:\\python37\\python.exe input_database\\modify_spinup_parameters.py input_database\\gcbm_input.db + +Description : Modify spinup parameters + + +Step 8 +------ + +Working directory : **Standalone_GCBM\\gcbm_project** + +.. code-block:: bash + :caption: Command + + update_gcbm_configuration.bat + +Description : + +1. Update the GCBM configuration +2. Automatically updates the GCBM configuration files based on the tiled layers: + - scans for all of the tiled layers and adds them to the provider configuration file + - sets the tile, block, and cell size in the config files so that the model knows the overall resolution of the simulation (the lowest common denominator of all the tiled layer resolutions) + - updates the list of disturbance layers in the simulation based on the DisturbanceLayer items in tiler.py + - updates the initial classifier set with the classifier layers tagged in tiler.py + +Output : Logs generated in `Standalone_GCBM\\logs\\update_gcbm_config.log` + +Step 9 +------ +Working directory : **Standalone_GCBM\\gcbm_project** + +.. code-block:: bash + :caption: Command + + run_gcbm.bat + +Description : + +1. Run the GCBM model +2. Project configuration is split between multiple files listed in `gcbm_project\\gcbm_config.cfg` +3. Data source configuration (spatial layers + SQLite) is in `gcbm_project\\provider_config.json` + +Output : Logs generated in `Standalone_GCBM\\logs\\Moja_Debug.log` + +Step 10 +------- + +Working directory : **Standalone_GCBM\\tools\\CompileGCBMSpatialOutput** + +.. code-block:: bash + :caption: Command + + create_tiffs.bat + +Description : + +1. Compile the spatial output +2. Generates tiff layers from raw GCBM spatial output + +Output : + +1. A tiff layer per indicator and timestep in `processed_output\\spatial` +2. Logs written into `Standalone_GCBM\\logs\\create_tiffs.log` + +Step 11 +------- + +Working directory : **Standalone_GCBM\\tools\\CompileGCBMResults** + +.. code-block:: bash + :caption: Command + + compileGCBMResults.bat + +Description : + +1. Compile the GCBM results +2. Turns the raw GCBM output database into a more user-friendly format containing most of the familiar indicators from the CBM3 Toolbox +3. Produces `processed_output\\compiled_gcbm_output.db`