pluto: Add documentation to pluto/README.md

Author: wdeconinck

pluto/README.md

@@ -10,6 +10,262 @@ What is it?
 ===========
 
 Pluto contains high-level abstractions for memory resource management and offloading data.
-It relies on the low-level hic (HIP/CUDA abstraction) library to support GPUs.
+The memory resource management is based on and compatible with C++17 `std::pmr::memory_resource` and `std::pmr::polymorphic_allocator`,
+and is extended with asynchronous (de)allocation methods.
+
+GPU specific memory resources are available, delegating to the low-level hic (HIP/CUDA abstraction) library to support GPUs.
+
+Pluto can be used and configured both from C++ and Fortran.
+
+The concepts
+============
+
+### pluto::memory_resource
+
+The `pluto::memory_resource` abstract class is an alias for
+[`std::pmr::memory_resource`](https://en.cppreference.com/w/cpp/memory/memory_resource)
+and provides following noteworthy member functions:
+```c++
+void* allocate(std::size_t bytes, std::size_t alignment) {
+    return do_allocate(bytes, alignment);
+}
+void deallocate(void* ptr, std::size_t bytes, std::size_t alignment) {
+    do_deallocate(ptr, bytes, alignment);
+}
+```
+Concrete implementations deriving from `pluto::memory_resource` must implement these functions:
+```c++
+void* do_allocate(std::size_t bytes, std::size_t alignment) override;
+void do_deallocate(void* ptr, std::size_t bytes, std::size_t alignment) override;
+```
+
+Pluto provides 8 predefined concrete implementations:
+| memory_resource         | memory_pool_resource         |
+|-------------------------|------------------------------|
+| pluto::host_resource    | pluto::host_pool_resource    |
+| pluto::device_resource  | pluto::device_pool_resource  |
+| pluto::pinned_resource  | pluto::pinned_pool_resource  |
+| pluto::managed_resource | pluto::managed_pool_resource |
+
+These predefined memory resources have unlimited lifetime and have memory tracking and tracing capability.  \
+See [Predefined pluto::memory\_resources](#predefined-plutomemory_resources) below for details on each memory resource.
+
+For convenience, pluto also provides aliases to two predefined standard library `std::pmr::memory_resources`:
+- pluto::new_delete_resource -> [std::pmr::new_delete_resource](https://en.cppreference.com/w/cpp/memory/new_delete_resource)
+- pluto::null_memory_resource -> [std::pmr::null_memory_resource](https://en.cppreference.com/w/cpp/memory/null_memory_resource)
+
+#### Example:
+```C++
+double* data;
+std::size_t bytes     = 10 * sizeof(double);
+std::size_t alignment = 64;
+pluto::memory_resource* mr = pluto::host_resource();
+double* data = (double*) mr->allocate(bytes, alignment);
+mr->deallocate(data, bytes, alignment);
+```
+
+### pluto::async_memory_resource
+
+The `pluto::async_memory_resource` extends `pluto::memory_resource` with asynchronous allocation and deallocation features. The asynchronous argument is a `pluto::stream_view`, which implements a `cudaStream` or `hipStream`. \
+The extra member functions are:
+```c++
+void* allocate_async(std::size_t bytes, std::size_t alignment, pluto::stream_view stream) {
+    return do_allocate_async(bytes, alignment, stream);
+}
+void deallocate_async(void* ptr, std::size_t bytes, std::size_t alignment, pluto::stream_view stream) {
+    do_deallocate_async(ptr, bytes, alignment, stream);
+}
+```
+Concrete implementations deriving from `pluto::async_memory_resource` then further implement these functions:
+```c++
+void* do_allocate_async(std::size_t bytes, std::size_t alignment, pluto::stream_view stream) override;
+void do_deallocate_async(void* ptr, std::size_t bytes, std::size_t alignment, pluto::stream_view stream) override;
+```
+
+### pluto::allocator
+
+The `pluto::allocator<T>` extends [`std::pmr::polymorphic_allocater`](https://en.cppreference.com/w/cpp/memory/polymorphic_allocator)
+which implements all functions required of a C++ [Allocator](https://en.cppreference.com/w/cpp/named_req/Allocator) to be given to [AllocatorAwareContainers](https://en.cppreference.com/w/cpp/named_req/AllocatorAwareContainer) such as e.g. `std::vector`, `std::map`, `std::set`, `std::list`, `std::string`.
+It internally uses a `pluto::memory_resource*` for allocation and deallocation.
+The noteworthy functions are:
+```c++
+/// Constructor without arguments; a configurable default pluto::memory_resource will be used
+/// This default can be set with `std::pmr::set_default_resource()` or `pluto::set_default_resource()`
+pluto::allocator<T>();
+
+/// Constructor using a given memory_resource. Note this is compatible with any third-party `std::pmr::memory_resource`
+pluto::allocator<T>(pluto::memory_resource*);
+
+/// Return an new allocated array with `size` number of elements. (Not bytes unlike pluto::memory_resource)
+T* allocate(std::size_t size);
+
+/// Deallocate a given array with `size` number of elements. (Not bytes unlike pluto::memory_resource)
+void deallocate(T* ptr, std::size_t size);
+
+/// Return an new allocated array with `size` number of elements. (Not bytes unlike pluto::memory_resource)
+T* allocate_async(std::size_t size, pluto::stream_view stream);
+
+/// Deallocate a given array with `size` number of elements. (Not bytes unlike pluto::memory_resource)
+void deallocate_async(T* ptr, std::size_t size, pluto::stream_view stream);
+```
+
+When `(de)allocate_async` is used with a `memory_resource` that does not derive from `pluto::async_memory_resource`, then `(de)allocate` will be used instead.
+The functions `(de)allocate`, `(de)allocate_async` also have overrides with a first argument `std::string_view label`, that can be used for tracing memory (de)allocations if the used concrete memory resources supports it.
+
+#### Examples:
+
+- Use allocator to allocate array of 10 elements using pre-defined `pluto::host_resource()` \
+  C++:
+  ```c++
+  pluto::allocator<double> alloc(pluto::host_resource());
+  double* data = alloc.allocate(10);
+  alloc.deallocate(data, 10);
+  ```
+  Fortran:
+  ```fortran
+  real(8), pointer :: array1d(:)
+  type(pluto_allocator) :: alloc
+  alloc = pluto%make_allocator(pluto%host_resource())
+  call alloc%allocate(array1d, [10])
+  call alloc%deallocate(array1d)
+  ```
+- Use memory pool in allocator-aware type using pre-defined `pluto::host_pool_resource()`
+  ```c++
+  std::vector<double, pluto::allocator<double>> vector(pluto::host_pool_resource());
+  vector.resize(10);
+  ```
+- The latter can be done via the `std::pmr::vector` as well due to the `std::pmr::memory_resource` compatibility:
+   ```c++
+   std::pmr::vector<double> vector(pluto::host_pool_resource());
+   vector.resize(10);
+   ```
+- We don't need to explicitely add the `memory_resource` in the `std::pmr::vector` constructor,
+  when setting the default beforehand:
+   ```c++
+   std::pmr::set_default_resource(pluto::host_pool_resource());
+   std::pmr::vector<double> vector;
+   vector.resize(10);
+   ```
+
+### pluto::{host,device} namespace
+
+In namespaces `pluto::host` and `pluto::device`, pluto manages defaults per memory space, independently from `std::pmr::{get,set}_default_resource()`.
+
+Following functions exist for C++
+```c++
+pluto::host::set_default_resource(pluto::memory_resource*);
+pluto::host::get_default_resource() -> pluto::memory_resource*;
+pluto::device::set_default_resource(pluto::memory_resource*);
+pluto::device::get_default_resource() -> pluto::memory_resource*;
+```
+Following routines exist for Fortran (pseudocode):
+```
+pluto%host%set_default_resource( type(pluto_memory_resource) )
+pluto%host%get_default_resource() -> type(pluto_memory_resource)
+pluto%device%set_default_resource( type(pluto_memory_resource) )
+pluto%device%get_default_resource() -> type(pluto_memory_resource)
+```
+
+Following C++ classes exist that extend `pluto::allocator<T>`:
+```c++
+pluto::host::allocator<T>
+pluto::device::allocator<T>
+```
+The only difference with `pluto::allocator` is that the default constructor won't use `std::pmr::get_default_resource()`, \
+but rather `pluto::{host,device}::get_default_resource()`.
+
+In Fortran you would create allocators that use the memory space specific defaults via:
+```fortran
+type(pluto_allocator) :: host_alloc, device_alloc
+host_alloc   = pluto%host%make_allocator()
+device_alloc = pluto%device%make_allocator()
+```
+
+The initial value returned by `pluto::{host,device}::get_default_resource()` is respectively `pluto::host_resource()` and `pluto::device_resource()` unless specified otherwise via environment variables:
+```sh
+export PLUTO_HOST_MEMORY_RESOURCE=pluto::pinned_pool_resource
+export PLUTO_DEVICE_MEMORY_RESOURCE=pluto::device_pool_resource
+```
+
+Predefined pluto::memory_resources
+----------------------------------
+
+Pluto provides a number of predefined concrete `pluto::memory_resources` via accessor
+ returning `pluto::memory_resource*` in C++ or `type(pluto_memory_resource)` in Fortran.
+They have C++ and Fortran accessor functions and are as well registered by name:
+- **pluto::new_delete_resource** \
+   Alias to `std::pmr::new_delete_resource`, using C++ new and delete.
+
+- **pluto::null_memory_resource()** \
+   Alias to `std::pmr::null_memory_resource`, throwing exception when used.
+
+- **pluto::host_resource** \
+   Allocates host CPU memory aligned to 256 bytes.
+
+- **pluto::host_pool_resource** \
+   A memory pool based on pluto::host_resource
+
+- **pluto::pinned_resource** \
+   Allocates host-pinned (a.k.a. page-locked) CPU memory aligned to 256 bytes.
+
+- **pluto::pinned_pool_resource** \
+   A memory pool based on pluto::pinned_resource
+
+- **pluto::device_resource** \
+   A `pluto::async_memory_resource` that allocates device resident memory. \
+   Internally this uses `cudaMalloc` or `hipMalloc` for allocate and
+   `cudaMallocAsync` or `hipMallocAsync` for allocate_async
+
+- **pluto::device_pool_resource** \
+   A memory pool based on pluto::device_resource
+
+- **pluto::managed_resource** \
+   Allocates UVM a.k.a. managed memory accessible from both host and device.\
+   Internally this uses `cudaMallocManaged` or `hipMallocManaged`
+
+- **pluto::managed_pool_resource** \
+   A memory pool based on pluto::managed_resource
+
+## Data transfer
+
+### pluto::memcpy\_{host,device}\_to\_{device,host}
+
+These functions work with void* pointers and bytes arguments. \
+An optional `pluto::stream_view` provides async data transfers
+
+### pluto::copy\_{host,device}\_to\_{device,host}
+
+These functions work with templated T* pointers, and size (number of elements) arguments. \
+An optional `pluto::stream_view` provides async data transfers
+
+
+### memcpy\_{host,device}\_to\_{device,host}\_2D and  copy\_{host,device}\_to\_{device,host}\_2D
+
+Like the above but for discontiguous slices, this is useful for the atlas::MultiField
+
+
+## Tracing and tracking memory
+
+The pluto predefined memory resources have tracking and tracing capability.
+To enable tracing, e.g. for debugging, set environment variable `PLUTO_TRACE=1`.
+Fine control is also possible programaticaly:
+```c++
+bool previous_status = pluto::trace::enable(true);
+// ... do stuff ...
+pluto::trace::enable(previous_status);
+```
+The trace output gets written to `pluto::trace::out` stream, which defaults to `std::cout`. This can be modified, e.g.
+```c++
+std::stringstream pluto_trace_stream;
+pluto::trace::set(pluto_trace_stream);
+```
+
+A memory usage report can be obtained, which reports on the use of each of the pluto predefined memory resources.
+Other user-defined memory resources are not taken into consideration.
+```
+pluto::memory::report() -> std::string
+```
+
+# Real Examples
 
 See examples subdirectory on how to use Pluto.