Merge #111: doc: Add internal design section

1fa2ca7 doc: Add internal design section (TheCharlatan)

Pull request description:

  I am not sure if the descriptions here are correct, or even all that useful, but I think some of the finer points are useful to document to serve as an anchor for future developers reading the code. Maybe I also missed some documentation in the headers and the other docs that already describe these things better.

  ~What could potentially also be useful are some diagrams to show the wrapping hierarchy as done below (again, not sure if actually correct, and given the polymorphism and templating going on it is also questionable if such a hierarchy can accurately be depicted as a diagram).~

  Given the points made by ryanofsky below, I don't think anymore than an object diagram will be particularly helpful.

   A sequence diagram that follows a call from its c++ callsite to the client invocation to the server receiving and processing it, the client resolving it and finally returning a c++ type again would probably be more useful.

ACKs for top commit:
  ryanofsky:
    ACK 1fa2ca7. Looks good, and nice job incorporating the extra information, especially about the waiter object. I'll go ahead and merge this now but feel free to make more changes or suggest improvements.

Tree-SHA512: cc247c5ef8fa787f4cdf151958c21c1428c848979fb46765bb9dd104bb144438aec8a3b38c714d05fbac013c0aba0dc4e23f2fd84800efddc1af9c934f0372ce

Author: ryanofsky

doc/design.md

@@ -11,6 +11,20 @@ If the wrapped C++ object inherits from an abstract base class declaring virtual
 
 There is also optional support for thread mapping, so each thread making interprocess calls can have a dedicated thread processing requests from it, and callbacks from processing threads are executed on corresponding request threads (so recursive mutexes and thread names function as expected in callbacks).
 
+Libmultiprocess acts as a pure wrapper or layer over the underlying protocol. Clients and servers written in other languages, but using a shared capnproto schema can communicate with interprocess counterparties using libmultiprocess without having to use libmultiprocess themselves or having to know about the implementation details of libmultiprocess.
+
+### Internals
+
+The `ProxyClient` and `ProxyServer` generated classes are not directly exposed to the user, as described in [usage.md](usage.md). Instead, they wrap c++ interfaces and appear to the user as pointers to an interface. They are first instantiated when calling `ConnectStream` and `ServeStream` respectively for creating the `InitInterface`. These methods establish connections through sockets, internally creating `Connection` objects wrapping a `capnp::RpcSystem` configured for client and server mode respectively.
+
+The `InitInterface` interface will typically have methods which return other interfaces, giving the connecting process the ability to call other functions in the serving process. Interfaces can also have methods accepting other interfaces as parameters, giving serving processes the ability to call back and invoke functions in connecting processes. Creating new interfaces does not create new connections, and typically many interface objects will share the same connection.
+
+Both `ConnectStream` and `ServeStream` also require an instantiation of the `EventLoop`. The `EventLoop` owns pending requests, notifies on request dispatch, allows clients from multiple threads to make synchronous calls, and handles some cleanup routines on exit. It must be run in a separate thread so it is always active and can process incoming requests from local clients and remote connections.
+
+When a generated method on the `ProxyClient` is called, it calls `clientInvoke` with the capnp-translated types. `clientInvoke` creates a self-executing promise (`kj::TaskSet`) that drives the execution of the request and gives ownership of it to the `EventLoop`. `clientInvoke` blocks until a response is received, or until there is a call from the server that needs to run on the same client thread, using a `Waiter` object.
+
+On the server side, the `capnp::RpcSystem` receives the capnp request and invokes the corresponding c++ method through the corresponding `ProxyServer` and the heavily templated `serverInvoke` triggering a `ServerCall`. Its return values from the actual c++ methods are copied into capnp responses by `ServerRet` and exceptions are caught and copied by `ServerExcept`. The two are connected through `ServerField`. The main method driving execution of a request is `PassField`, which is invoked through `ServerField`. Instantiated interfaces, or capabilities in capnp speak, are tracked and owned by the server's `capnp::RpcSystem`.
+
 ## Interface descriptions
 
 As explained in the [usage](usage.md) document, interface descriptions need to be consumed both by the _libmultiprocess_ code generator, and by C++ code that calls and implements the interfaces. The C++ code only needs to know about C++ arguments and return types, while the code generator only needs to know about capnp arguments and return types, but both need to know class and method names, so the corresponding `.h` and `.capnp` source files contain some of the same information, and have to be kept in sync manually when methods or parameters change. Despite the redundancy, reconciling the interface definitions is designed to be _straightforward_ and _safe_. _Straightforward_ because there is no need to write manual serialization code or use awkward intermediate types like [`UniValue`](https://github.com/bitcoin/bitcoin/blob/master/src/univalue/include/univalue.h) instead of native types. _Safe_ because if there are any inconsistencies between API and data definitions (even minor ones like using a narrow int data type for a wider int API input), there are errors at build time instead of errors or bugs at runtime.