Difficulties of re-entering wasm in presence of stack switching

[frank-emrich]

This issue is intended to document, discuss and track some difficulties that occur when performing host calls in the presence of stack switching that in turn re-enter wasm.

For example, consider the following logical call stack (here, "logical" means that this is not a single, contingent stack):

$f -resume-> $g -call-> host_function -call-> $h

where $f, $g, and $h are wasm functions. Here, $g is not executing on the main stack, but inside a continuation, when performing the host call. The latter then re-enters wasm by calling $h.

There are multiple issues that would need to be resolved in order for this to work correctly:

1. Most likely, we want to consider host calls to act as barriers for the purposes of stack switching, meaning that a suspend in $h cannot get handled in $f or $g. This means that if any of these functions run in instances of the same Store, we need to set aside the continuation chain when entering h and restore it once it returns.

2. The first point illustrates that when starting execution of $h, we would want to indicate that it is not subject to any handlers. Currently, this would be achieved by setting its Store's stack chain to MainStack (independently from whether it actually shares a Store with the other involved wasm functions or not).
   However, $h is not actually running on the main stack, since it is running on same FiberStack as $g. For the purposes of correctly handling stack limits, we would need some way to indicate that $h is running inside a FiberStack, but without actually being subject to any other handlers. Of course, we may also eventually decide that all host calls switch to the actual main stack for execution.

I think that all of these issues can be resolved, but that these solutions possibly interact with future design decisions. Therefore, I suggest that we simply disallow this for now by enforcing the current rule in the implementation: Re-entering wasm from a host call is only permitted when not already running inside a continuation. Or in other words, the host call from which we want to re-enter wasm must have been running on the main stack. We can then ease these restrictions at a later point.

[dhil]

As we discussed the yesterday on the call, I think for now it is sufficient if we assume all host calls are leaves in the call graph, and if that assumption is violated then we hard fail. Later, we can look into relaxing this assumption.

[frank-emrich]

I'm afraid that restriction is too strong because it would break existing (i.e., non-wasmfx) test cases in wasmtime.

The restriction I was proposing is slightly weaker: We require that host calls are leaf calls (in the sense of not re-entering wasm) if we were running inside a continuation when performing the host call.