Merge branch 'development' into feature/boris-linearized-cloud

* development:
  docs: Update install instructions.
  config: Add abs/rel residual tolerances.
  examples: Update option initialization.
  config: Tidy and set defaults.
  config: Get rid of redundant `add_option`.
  config: Add two stage config/log init to `pfasst::init`.
  config: Revamp adding options.
  tests: Add RelativeAdaptiveErrorTest to test_advection_diffusion.cpp.
  mpi: Minor tidying.
  tests: Re-organise the adaptive MLSDC tests a bit.
  tests: Add AdaptiveErrorTest for AD example to MPI PFASST.
  encap: Compute relative residuals as well.
  status: Bug fix.
  status: Move to blocking send/recv.
  status: Break iterations in PFASST when converged.
  status: Begin adding converged status MPI communication.

Conflicts:
	examples/advection_diffusion/vanilla_sdc.cpp
	examples/boris/boris_sdc.cpp
	examples/boris/boris_sweeper.hpp
	include/pfasst.hpp

Author: torbjoernk

doc/source/installing.md

@@ -2,34 +2,63 @@
 
 ## Prerequesites
 
-* A recent C++ compiler that supports (most of) the C++11 standard.
-  This either implies at least GCC 4.7 (we recommend 4.8 or later) or Clang 3.2 (we recommend 3.3 or later).
+If you use [CMake] to compile, most of the prerequesites are downloaded automatically if they aren't
+already found on your system.
 
-* The [Boost](https://boost.org/) libraries at least of version 1.53.0.
-  Especially `boost::program_options` is required.
+### Required
 
-* The [CMake] build tool (at least version 2.8).
+* A recent C++ compiler that supports (most of) the C++11 standard.  This implies either: GCC 4.7
+  (we recommend 4.8 or later) or Clang 3.2 (we recommend 3.3 or later).
 
-* The [Eigen3] library is required for everything using the \em PFASST++ library.
-  It will be automatically downloaded if it is not found on the system.
+* The [Boost] `boost::program_options` library (at least version 1.53.0).  The [Boost]
+  multiprecision component may also be of interest to \em PFASST++ users.
 
-* The [FFTW3] library is required for (some) of the examples.
-  It will be automatically downloaded and built as well if it is not found on the system.
+* The [Eigen3] library.
 
-* The [Google Testing] and [Mocking Framework] are required for the unit tests, and are automatically downloaded and 
-  built.
+### Optional
+
+* The [CMake] build tool (at least version 2.8).  This is especially useful for developers.
+
+* The [FFTW3] library is required for some of the examples.
+
+* The [Google Testing] and [Google Mocking] are required for the unit tests.
+
+* An MPI implementation.
+
+### Using [HashDist] to obtain prerequesites
+
+The [HashDist] environment management system provides an easy, robust, and version controlled way
+for our developers (and users) to download and install all prerequesites *locally* (ie, you don't
+need special privileges on your system to use [HashDist], and it won't modify your system).
+
+To use [HashDist]
+
+1. Install [HashDist]
+
+        git clone [email protected]:hashdist/hashdist.git
+        export PATH=$PWD/hashdist/bin:$PATH
+
+2. Install prerequesites
+
+        cd PFASST
+        hit build -v tools/hashdist/pfasst-stack.debian.yaml
+        mv pfasst-stack.debian stack
+
+This may take a while (ie, do this before going for lunch) the first time through.  However, once
+complete, subsequent builds (using [CMake] or otherwise) will bypass prerequesite installation, even
+if you change [CMake] build options or start a fresh out-of-source build directory.
 
 
 ## Obtaining the Sources
 
-Either use _Git_ to clone the repository (https://github.com/Parallel-in-Time/PFASST.git) or download the latest 
-release from [GitHub][github_releases].
+Use _Git_ to clone the repository (https://github.com/Parallel-in-Time/PFASST.git) or download the
+latest release from [GitHub][github_releases].
 
 
-## Building
+## Building with CMake
 
-1. Create a directory for out-of-source builds (usually, people are creative and name it `build`) in the source tree 
-   (or elsewhere).
+1. Create a directory for out-of-source builds (usually, people are creative and name it `build`) in
+   the source tree (or elsewhere).
 
 2. Decide on what and how you want to build:
 
@@ -58,7 +87,7 @@ release from [GitHub][github_releases].
 
    * __Test Suite__
 
-     * Deactivate building of the test suite by passing `-Dpfasst_BUILD_TESTS=OFF` to the _CMake_ command line (not 
+     * Deactivate building of the test suite by passing `-Dpfasst_BUILD_TESTS=OFF` to the _CMake_ command line (not
        recommended).
 
    * __Examples__
@@ -81,7 +110,7 @@ release from [GitHub][github_releases].
      * In case you also want to install the compiled example programs on your system, specify
        `-Dpfasst_INSTALL_EXAMPLES=ON`.
 
-   * For example, to build a release version with _Clang_ on a Linux system without the examples, the call to 
+   * For example, to build a release version with _Clang_ on a Linux system without the examples, the call to
      _CMake_ looks like:
 
          cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -Dpfasst_BUILD_EXAMPLES=OFF ..
@@ -99,18 +128,25 @@ release from [GitHub][github_releases].
 
        make test
 
-   In case any of the tests do not pass, please [open an issue on GitHub][github_new_issue] and provide the full log of 
+   In case any of the tests do not pass, please [open an issue on GitHub][github_new_issue] and provide the full log of
    your _CMake_ and _make_ invocations.
 
 5. (optional) Install \em PFASST++ headers (and compiled examples if specified so):
 
        make install
 
+## Building with vanilla make
+
+A sample `Makefile` is included in the `advection_diffusion` example.  This may be of particular
+interested to advanced users wishing to incorporate \rm PFASST++ into their existing code bases.
+
 
+[Boost]: https://boost.org/
 [CMake]: http://cmake.org/
 [Eigen3]: http://eigen.tuxfamily.org/
 [Google Testing]: https://code.google.com/p/googletest/
-[Mocking Framework]: https://code.google.com/p/googlemock/
+[Google Mocking]: https://code.google.com/p/googlemock/
 [FFTW3]: http://fftw.org/
 [github_releases]: https://github.com/Parallel-in-Time/PFASST/releases
 [github_new_issue]: https://github.com/Parallel-in-Time/PFASST/issues/new
+[HashDist]: http://hashdist.readthedocs.org/en/latest/

examples/advection_diffusion/advection_diffusion_sweeper.hpp

@@ -47,21 +47,15 @@ namespace pfasst
       class AdvectionDiffusionSweeper
         : public encap::IMEXSweeper<time>
       {
-        private:
-          static void init_config_options(po::options_description& opts)
+        public:
+          static void init_opts()
           {
-            opts.add_options()
-              ("spatial_dofs", po::value<size_t>(), "number of spatial degrees of freedom")
-              ;
+            pfasst::config::options::add_option<size_t>("Adv/Diff Sweeper", "spatial_dofs", "Number of spatial degrees of freedom");
           }
 
-        public:
-          static void enable_config_options(size_t index = -1)
+          static void init_logs()
           {
-            config::Options::get_instance()
-              .register_init_function("Advection-Diffusion Sweeper",
-                                      std::function<void(po::options_description&)>(init_config_options),
-                                      index);
+            pfasst::log::add_custom_logger("Advec");
           }
 
         private:
@@ -141,8 +135,8 @@ namespace pfasst
 
             auto n = this->get_controller()->get_step();
             auto k = this->get_controller()->get_iteration();
-            CLOG(INFO, "Advec") << "err: " << n << " " << k << " " << max << " (" << qend.size() << "," << predict << ")";
 
+            CLOG(INFO, "Advec") << "err: " << n << " " << k << " " << max << " (" << qend.size() << "," << predict << ")";
             this->errors.insert(vtype(ktype(n, k), max));
           }
 

examples/advection_diffusion/mpi_pfasst.cpp

@@ -21,21 +21,20 @@ using namespace std;
 
 #include "advection_diffusion_sweeper.hpp"
 #include "spectral_transfer_1d.hpp"
+
 using namespace pfasst::encap;
 using namespace pfasst::mpi;
 
-
 namespace pfasst
 {
   namespace examples
   {
     namespace advection_diffusion
     {
-      error_map run_mpi_pfasst()
+      error_map run_mpi_pfasst(double abs_residual_tol, size_t niters=4)
       {
         const size_t nsteps = 4;
         const double dt     = 0.01;
-        const size_t niters = 4;
 
         vector<pair<size_t, quadrature::QuadratureType>> nodes = {
           { 3, quadrature::QuadratureType::GaussLobatto },
@@ -67,6 +66,7 @@ namespace pfasst
         pf.set_comm(&comm);
         pf.set_duration(0.0, nsteps * dt, dt, niters);
         pf.set_nsweeps({2, 1});
+        pf.get_finest<AdvectionDiffusionSweeper<>>()->set_residual_tolerances(abs_residual_tol, 0.0);
         pf.run();
 
         auto fine = pf.get_finest<AdvectionDiffusionSweeper<>>();
@@ -81,7 +81,7 @@ namespace pfasst
 int main(int argc, char** argv)
 {
   MPI_Init(&argc, &argv);
-  pfasst::examples::advection_diffusion::run_mpi_pfasst();
+  pfasst::examples::advection_diffusion::run_mpi_pfasst(0.0);
   fftw_cleanup();
   MPI_Finalize();
 }

examples/advection_diffusion/serial_mlsdc.cpp

@@ -101,10 +101,9 @@ namespace pfasst
 #ifndef PFASST_UNIT_TESTING
 int main(int argc, char** argv)
 {
-  pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::enable_config_options();
-  pfasst::init(argc, argv);
-  pfasst::log::add_custom_logger("Advec");
-
+  pfasst::init(argc, argv,
+               pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::init_opts,
+               pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::init_logs);
   pfasst::examples::advection_diffusion::run_serial_mlsdc(3);
 }
 #endif

examples/advection_diffusion/serial_mlsdc_autobuild.cpp

@@ -97,10 +97,9 @@ namespace pfasst
 #ifndef PFASST_UNIT_TESTING
 int main(int argc, char** argv)
 {
-  pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::enable_config_options();
-  pfasst::init(argc, argv);
-  pfasst::log::add_custom_logger("Advec");
-
+  pfasst::init(argc, argv,
+               pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::init_opts,
+               pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::init_logs);
   pfasst::examples::advection_diffusion::run_serial_mlsdc_autobuild();
 }
 #endif

examples/advection_diffusion/vanilla_sdc.cpp

@@ -23,16 +23,13 @@ namespace pfasst
   {
     namespace advection_diffusion
     {
-      error_map run_vanilla_sdc(double abs_residual_tol)
+      error_map run_vanilla_sdc(double abs_residual_tol, double rel_residual_tol=0.0)
       {
         SDC<> sdc;
 
-        const size_t nsteps = config::get_value<size_t>("num_steps", 4);
-        const double dt     = config::get_value<double>("delta_step", 0.01);
-        const size_t nnodes = config::get_value<size_t>("num_nodes", 3);
-        const size_t ndofs  = config::get_value<size_t>("spatial_dofs", 64);
-        const size_t niters = config::get_value<size_t>("num_iter", 4);
-        const quadrature::QuadratureType quad_type = \
+        auto const nnodes = config::get_value<size_t>("num_nodes", 3);
+        auto const ndofs  = config::get_value<size_t>("spatial_dofs", 64);
+        auto const quad_type = \
           config::get_value<quadrature::QuadratureType>("nodes_type", quadrature::QuadratureType::GaussLegendre);
 
         auto quad    = quadrature::quadrature_factory(nnodes, quad_type);
@@ -41,10 +38,11 @@ namespace pfasst
 
         sweeper->set_quadrature(quad);
         sweeper->set_factory(factory);
-        sweeper->set_residual_tolerances(abs_residual_tol, pfasst::config::get_value<double>("rel_res_tol", 0.0));
+        sweeper->set_residual_tolerances(abs_residual_tol, rel_residual_tol);
 
         sdc.add_level(sweeper);
-        sdc.set_duration(0.0, nsteps*dt, dt, niters);
+        sdc.set_duration(0.0, 4*0.01, 0.01, 4);
+        sdc.set_options();
         sdc.setup();
 
         auto q0 = sweeper->get_start_state();
@@ -64,12 +62,9 @@ namespace pfasst
 #ifndef PFASST_UNIT_TESTING
 int main(int argc, char** argv)
 {
-  pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::enable_config_options();
-  pfasst::init(argc, argv);
-  pfasst::log::add_custom_logger("Advec");
-
-  const double abs_res_tol = pfasst::config::get_value<double>("abs_res_tol", 0.0);
-
-  pfasst::examples::advection_diffusion::run_vanilla_sdc(abs_res_tol);
+  pfasst::init(argc, argv,
+               pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::init_opts,
+               pfasst::examples::advection_diffusion::AdvectionDiffusionSweeper<>::init_logs);
+  pfasst::examples::advection_diffusion::run_vanilla_sdc(0.0);
 }
 #endif

examples/boris/boris_mlsdc.cpp

@@ -82,7 +82,7 @@ namespace pfasst
 #ifndef PFASST_UNIT_TESTING
 int main(int argc, char** argv)
 {
-  pfasst::examples::boris::enable_config_options<double>();
+  pfasst::init(argc, argv, pfasst::examples::boris::init_opts<double>);
   pfasst::init(argc, argv);
   pfasst::log::add_custom_logger("Boris");
   pfasst::log::add_custom_logger("BorisTransfer");

examples/boris/boris_sdc.cpp

@@ -67,8 +67,7 @@ namespace pfasst
 #ifndef PFASST_UNIT_TESTING
 int main(int argc, char** argv)
 {
-  pfasst::examples::boris::enable_config_options<double>();
-  pfasst::init(argc, argv);
+  pfasst::init(argc, argv, pfasst::examples::boris::init_opts<double>);
   pfasst::log::add_custom_logger("Boris");
   pfasst::log::add_custom_logger("Solver");
 

examples/boris/boris_sweeper.hpp

@@ -75,10 +75,7 @@ namespace pfasst
 
 
       template<typename scalar>
-      static void init_config_options(po::options_description& opts);
-
-      template<typename scalar>
-      static void enable_config_options(size_t index = -1);
+      static void init_opts();
 
 
       class LogIndent

examples/boris/boris_sweeper_impl.hpp

@@ -56,23 +56,12 @@ namespace pfasst
 
 
       template<typename scalar>
-      static void init_config_options(po::options_description& opts)
+      static void init_opts()
       {
-        opts.add_options()
-          ("num_particles", po::value<size_t>(), "number of particles in the cloud")
-          ("epsilon", po::value<scalar>(), "Boris' epsilon")
-          ("omega_e", po::value<scalar>(), "E-field constant")
-          ("omega_b", po::value<scalar>(), "B-field constant")
-          ;
-      }
-
-      template<typename scalar>
-      static void enable_config_options(size_t index)
-      {
-        config::Options::get_instance()
-          .register_init_function("Boris-SDC",
-                                  std::function<void(po::options_description&)>(pfasst::examples::boris::init_config_options<scalar>),
-                                  index);
+        pfasst::config::options::add_option<size_t>("Boris-SDC", "num_particles", "number of particles in the cloud");
+        pfasst::config::options::add_option<scalar>("Boris-SDC", "epsilon", "Boris' epsilon");
+        pfasst::config::options::add_option<scalar>("Boris-SDC", "omega_e", "E-field constant");
+        pfasst::config::options::add_option<scalar>("Boris-SDC", "omega_b", "B-field constant");
       }