Merge pull request #1371 from vkarak/feat/immediate-install

[feat] Add bootstrap script to install dependencies of ReFrame

Author: Vasileios Karakasis

README.md

@@ -29,25 +29,37 @@ Users can create their own test hierarchies, create test factories for generatin
 
 ## Getting ReFrame
 
-You may install ReFrame directly from [PyPI](https://pypi.org/project/ReFrame-HPC/) through `pip`:
+ReFrame is almost ready to run just after you clone it from Github.
+All you need is Python 3.6 or above and to run its bootstrap script:
 
 ```bash
-pip install reframe-hpc
+git clone https://github.com/eth-cscs/reframe.git
+cd reframe
+./bootstrap.sh
+./bin/reframe -V
 ```
 
-ReFrame will be available in your PATH:
+### Other installation ways
 
-```bash
-reframe -V
-```
+You can also install ReFrame through the following channels:
 
-Alternatively, and especially if you want to contribute back to the framework, you may clone this repository:
+- Through [PyPI](https://pypi.org/project/ReFrame-HPC/):
 
-```bash
-git clone https://github.com/eth-cscs/reframe.git
-cd reframe
-./bin/reframe -V
-```
+  ```
+  pip install reframe-hpc
+  ```
+
+- Through [Spack](https://spack.io/):
+
+  ```
+  spack install reframe
+  ```
+
+- Through [EasyBuild](https://easybuild.readthedocs.io/):
+
+  ```
+  eb easybuild/easyconfigs/r/ReFrame/ReFrame-VERSION.eb -r
+  ```
 
 Finally, you may access all previous versions of ReFrame [here](https://github.com/eth-cscs/reframe/releases).
 
@@ -57,24 +69,22 @@ Finally, you may access all previous versions of ReFrame [here](https://github.c
 You may find the official documentation of the latest release and the current master in the following links:
 
 - [Latest release](https://reframe-hpc.readthedocs.io/en/stable)
-- [Current master](https://reframe-hpc.readthedocs.io)
+- [Current master](https://reframe-hpc.readthedocs.io/en/latest)
 
 
 ### Building the documentation locally
 
-You may build the documentation of the master locally either with Python 2 or Python 3.
-Here is how to do it:
+You may build the documentation of the master manually as follows:
 
 ```
-pip install -r docs/requirements.txt
-make -C docs latest
+./bootstrap.sh +docs
 ```
 
 For viewing it, you may do the following:
 
 ```
 cd docs/html
-python -m http.server # or python -m SimpleHTTPServer for Python 2
+python3 -m http.server
 ```
 
 The documentation is now up on [localhost:8000](http://localhost:8000), where you can navigate with your browser.

bin/reframe

@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+#
+# Copyright 2016-2020 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
+# ReFrame Project Developers. See the top-level LICENSE file for details.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+import os
+import sys
+
+prefix = os.path.join(os.path.abspath(os.path.dirname(__file__)), '..')
+external = os.path.join(prefix, 'external')
+sys.path = [prefix, external] + sys.path
+
+
+import reframe.frontend.cli as cli  # noqa: F401, F403
+
+if __name__ == '__main__':
+    cli.main()

bin/reframe

@@ -0,0 +1,62 @@
+#!/bin/bash
+#
+# Copyright 2016-2020 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
+# ReFrame Project Developers. See the top-level LICENSE file for details.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+#
+# Bootstrap script for running ReFrame from source
+#
+
+BLUE='\033[0;34m'
+GREEN='\033[0;32m'
+YELLOW='\033[0;33m'
+NC='\033[0m'
+
+CMD()
+{
+    echo -e "${BLUE}==>${NC}" ${YELLOW}$*${NC} && $*
+}
+
+usage()
+{
+    echo "Usage: $0 [-h] [+docs]"
+    echo "Bootstrap ReFrame by pulling all its dependencies"
+    echo "  -P EXEC  Use EXEC as Python interpreter"
+    echo "  -h       Print this help message and exit"
+    echo "  +docs    Build also the documentation"
+}
+
+
+while getopts "hP:" opt; do
+    case $opt in
+        "P") python=$OPTARG ;;
+        "h") usage && exit 0 ;;
+        "?") usage && exit 0 ;;
+    esac
+done
+
+shift $((OPTIND - 1))
+if [ -z $python ]; then
+    python=python3
+fi
+
+pyver=$($python -V | sed -n 's/Python \([0-9]\+\)\.\([0-9]\+\)\..*/\1.\2/p')
+
+# Install pip for Python 3
+CMD $python -m ensurepip --root external/ --default-pip
+
+# ensurepip installs pip in `external/usr/` whereas the --target option installs
+# everything under `external/`. That's why include both in the PYTHONPATH
+
+export PATH=$(pwd)/external/usr/bin:$PATH
+export PYTHONPATH=$(pwd)/external:$(pwd)/external/usr/lib/python$pyver/site-packages:$PYTHONPATH
+
+CMD $python -m pip install -q --upgrade pip --target=external/
+CMD $python -m pip install -q -r requirements.txt --target=external/ --upgrade
+
+if [ x"$1" == x"+docs" ]; then
+    CMD $python -m pip install -q -r docs/requirements.txt --target=external/ --upgrade
+    make -C docs
+fi

bootstrap.sh

@@ -132,14 +132,8 @@ else
     module load reframe
 fi
 
-# Always install our requirements
-python3 -m venv venv.unittests
-source venv.unittests/bin/activate
-pip install --upgrade pip
-pip install -r requirements.txt
-
-# FIXME: XALT is causing linking problems (see UES-823)
-module unload xalt
+# Bootstrap ReFrame
+./bootstrap.sh
 
 echo "[INFO] Loaded Modules"
 module list
@@ -216,5 +210,4 @@ else
         done
     fi
 fi
-deactivate
 exit $CI_EXITCODE

ci-scripts/ci-runner.bash

@@ -37,21 +37,26 @@ echo "Working directory: $tmpdir ..."
 cd $tmpdir
 git clone https://github.com/eth-cscs/reframe.git
 cd reframe
-found_version=$(./reframe.py -V | sed -e 's/ (.*)//g')
+./bootstrap.sh
+found_version=$(./bin/reframe -V | sed -e 's/ (.*)//g')
 if [ $found_version != $version ]; then
     echo "$0: version mismatch: found $found_version, but required $version" >&2
     exit 1
 fi
 
-python3 -m venv venv.deployment
-source venv.deployment/bin/activate
-pip install --upgrade pip setuptools wheel twine
-pip install -r requirements.txt
 ./test_reframe.py
 git tag -a v$version -m "ReFrame $version"
 git push origin --tags
-python setup.py sdist bdist_wheel
-twine upload dist/*
+
+# We need this for running the setup.py of ReFrame
+export PYTHONPATH=$(pwd)/external:$PYTHONPATH
+
+# We create a virtual environment here just for the deployment
+python3 -m venv venv.deployment
+source venv.deployment/bin/activate
+python3 -m pip install --upgrade pip setuptools wheel twine
+python3 setup.py sdist bdist_wheel
+python3 -m twine upload dist/*
 deactivate
 cd $oldpwd
 echo "Deployment was successful!"

ci-scripts/deploy.sh

@@ -29,7 +29,11 @@
 
 import sphinx_rtd_theme
 
-sys.path.insert(0, os.path.abspath('..'))
+prefix = os.path.join(os.path.abspath(os.path.dirname(__file__)), '..')
+pymajver = sys.version_info.major
+pyminver = sys.version_info.minor
+external = os.path.join(prefix, 'external')
+sys.path = [prefix, external] + sys.path
 
 import reframe                           # noqa: F401, F403
 import reframe.utility.os_ext as os_ext  # noqa: F401, F403

docs/conf.py

@@ -7,11 +7,8 @@ Requirements
 
 * Python 3.6 or higher.
   Python 2 is not supported.
-* Required Python packages can be found in the ``requirements.txt`` file, which you can install as follows:
-
-  .. code:: bash
-
-     pip3 install -r requirements.txt
+* Required Python packages can be found in the ``requirements.txt`` file.
+  See :ref:`install-from-source` for more information on how to install ReFrame from source.
 
 
 ---------------------
@@ -63,6 +60,8 @@ ReFrame's latest stable version is available through different channels:
      eb easybuild/easyconfigs/r/ReFrame/ReFrame-VERSION.eb -r
 
 
+.. _install-from-source:
+
 -------------------------------
 Getting the Latest and Greatest
 -------------------------------
@@ -71,10 +70,23 @@ If you want the latest development version or any pre-release, you can clone ReF
 
 .. code:: bash
 
-    git clone https://github.com/eth-cscs/reframe.git
+   git clone https://github.com/eth-cscs/reframe.git
 
 
 Pre-release versions are denoted with the ``devX`` suffix and are `tagged <https://github.com/eth-cscs/reframe/releases>`__ in the repository.
+Preparing and running ReFrame from source is pretty straightforward:
+
+.. code:: bash
+
+   git clone https://github.com/eth-cscs/reframe.git
+   cd reframe
+   ./bootstrap.sh
+   ./bin/reframe -V
+
+.. note::
+   .. versionadded:: 3.1
+      The bootstrap script for ReFrame was added.
+      For previous ReFrame versions you should install its requirements using ``pip install -r requirements.txt`` in a Python virtual environment.
 
 
 Running the Unit Tests

docs/started.rst

@@ -5,12 +5,16 @@
 #
 # SPDX-License-Identifier: BSD-3-Clause
 
-import argparse
 import os
-import pytest
 import sys
 
-import unittests.fixtures as fixtures
+prefix = os.path.abspath(os.path.dirname(__file__))
+external = os.path.join(prefix, 'external')
+sys.path = [prefix, external] + sys.path
+
+import argparse                         # noqa: F401, F403
+import pytest                           # noqa: F401, F403
+import unittests.fixtures as fixtures   # noqa: F401, F403
 
 
 if __name__ == '__main__':
@@ -38,5 +42,12 @@
     fixtures.USER_CONFIG_FILE = options.rfm_user_config
     fixtures.USER_SYSTEM = options.rfm_user_system
     fixtures.init_runtime()
+
+    # If no positional argument is specified, use the `unittests` directory,
+    # so as to avoid any automatic discovery of random unit tests from the
+    # external dependencies.
+    if all(arg.startswith('-') for arg in rem_args):
+        rem_args.append('unittests')
+
     sys.argv = [sys.argv[0], *rem_args]
     sys.exit(pytest.main())