Detecting and reacting to network failures

[erlingrj]

Since we are targeting distributed systems, we must have a strategy for handling node failures and network failures. Initially, we made it very easy for ourselves and said that the NetworkChannels must provide reliable, in-order delivery. But we know this is unrealistic for many systems. In the same way that we let users handle timing and ordering failures with a deadline violation handler and STP violation handler, I think we must let the user handle network failures at the application level.

I propose that we let the user provide a handler, which is triggered when the connection status of the associated port changes. I am trying to find the right keyword for it, I am using status for now, to indicate that it is triggered by a change in the port status.

reactor Fed {
  input in:int
  output out:int
  
  reaction(in) {=
    // Handle events on in
  =} status {=
    // Handle changes to the status of the input port.
    if (in->is_connected()) {
    } else {
   }
   =}

  reaction(in) -> out {=
    // Handle events on in
  =} status {=
    // Handle status change on either `in` or `out`
   =}

Some open questions:

• At what logical tag should the status handler be invoked. What if it is simultaneous to the reaction triggering also.
• Another alternative is to specify a handler for each port, and not tie it to a reaction declaration.

I am curious to hear your thoughts here. Personally, I think this is the only reasonable way forward. We are thinking about distributed real-time systems with unreliable networks, we cannot hide the fact that the connection status of ports will change throughout the lifecycle of the federate.

[edwardalee]

Perhaps a more generic catch instead of status? Good question about what the logical time should be...

[erlingrj]

I like catch.

On the question of the logical tag. I think we want this handler to execute like a normal reaction and be able to schedule actions and write to ports, like the STP and deadline violation handler. This means that it must execute at a logical tag. The STP and deadline violation handlers are naturally mutually exclusive with the associated reaction body, but this is not the case for the catch handler. It seems always to be the possibility that the associated reaction is triggered simultaneously. I can think of a couple of approaches:

1. Execute the catch handler at a "reserved" microstep, e.g., UINT32_MAX, but this might open a slippery slope of reserving microsteps for various things.
2. Do not associate catch with a reaction. Instead, we register it for a port (or set of ports) independently. This means that the user decides which effects it can have and the priority wrt reactions through the lexicographic order:

reactor Fed {
  input in:int
  output out:int 
  
  reaction(in) -> out {==}
  
  catch(in) -> out {=
    // Triggered at earliest possible tag when a network failure on port `in` is detected
    // Can write to port `out`
  =}
  catch (out) {=
  =}
  catch(in,out)  {=
    // Triggered at earliest possible tag when a network failure on either port is detected
  =}

3. Allow logically simultaneous catch-handler and reaction body with a predefined ordering. E.g. catch handler is executed after. This has the subtlety that the catch handler might overwrite port values set by the reaction body.

I don't like (1) because we start changing the model-of-computation, so I think (2) or (3) is better.

[erlingrj]

I have done some more thinking and I have come to the following conclusion:

If the connection status on an input or output port changes, an event should be created and put on the event queue with the earliest possible unique tag, this event will trigger the catch-handler. A unique tag is a tag where no other reactions execute.

If a status change occurs at time T, and produces an event with time t. Then any input events arriving on the associated connection with tag less than or equal to t will be discarded. Any writes to an output port at tags strictly less than t will also be discarded. In other words. The status change happens at a well-defined logical tag, which is the tag the catch handler is executed at, and the catch handler can write to a newly connected output port (since only outputs strictly less than t are discarded). However events on an input port can not trigger any reaction on t.

Why do we need a unique tag? We need a unique tag because we want two contradictory things:

1. In the case of a status change, we want catch handler to execute BEFORE any reaction executes that sees the updated status. I.e. if they were to execute on the same tag, catch handler should have precedence.
2. If the catch handler produces an output to a port, we don't want that to possibly be overwritten by a reaction executing at the same tag. This implies that the catch handler should execute AFTER any reaction.

So these two requirements cant be reconciled. Thus I propose to reserve a special microstep index for catch handlers. E.g. UINT32_MAX. This will ensure that both 2 requirements above are satisfied.

[edwardalee]

I see some problems with this. If the catch handler produces an output, that output has to be seen at the same tag as when the catch handler is invoked. The output will not be present at a future tag. Hence, any downstream reaction has to execute at the same tag.

I think a simpler solution is to set the precedence of the catch handler to be equal to that of the reaction it is attached to. This will be obvious to the programmer, who can then avoid overwriting any output produced.

For item 1. above, I don't really understand what you mean by "any reaction that sees the updated status." If the catch handler is associated with a reaction, then it seems to me that it should run instead of the reaction, rather than before the reaction. If the reaction is expecting network inputs that will not arrive, I don't really see how running the reaction makes any sense.

[erlingrj]

I see some problems with this. If the catch handler produces an output, that output has to be seen at the same tag as when the catch handler is invoked. The output will not be present at a future tag. Hence, any downstream reaction has to execute at the same tag.

This should be fine. But I see now that there is an edge case. If the catch handler produces a message for another federate, then this might trigger a reaction in the other federate at the UINT32_MAX microstep, which might coincide with the execution of an associated catch-handler.

I think a simpler solution is to set the precedence of the catch handler to be equal to that of the reaction it is attached to. This will be obvious to the programmer, who can then avoid overwriting any output produced.

So the source of this complication is that the catch handler and reaction body cannot have the same precedence because they might both execute at the same tag. Imagine:

reaction(in1, in2) -> out {=
  // Do something
  lf_set(out, "OK");
=} catch {= 
  // Do something else
  lf_set(out, "NOT OK");
=}

It is possible (although maybe unlikely) that we detect that our connection to in1 is down and receive a message from in2 very close int time. In this scenario we want to execute the catch handler and also the reaction body. And if we just pick the earliest possible tag for the catch handler it might be simultaneous to the reaction body.

For item 1. above, I don't really understand what you mean by "any reaction that sees the updated status." If the catch handler is associated with a reaction, then it seems to me that it should run instead of the reaction, rather than before the reaction. If the reaction is expecting network inputs that will not arrive, I don't really see how running the reaction makes any sense.

If we consider the code sample above again. Consider that the connection to in1 is NOT_CONNECTED and that at some point we establish connection again. This change of status should trigger the catch handler again. Lets say it is trigger at t1. I think it is important that a reaction triggering at any t < t1 should not observe the change from NOT_CONNECTED to CONNECTED. In other words, the catch handler is the very first thing that happens after the connection status changes.

Consider that instead it was out which was NOT_CONNECTED. I am arguing that when we establish a connection over out, we should not let any reaction send messages over out until AFTER the catch handler has executed. In the same way, this makes it so that the catch handler is the very first thing that is executed after a status change. We enforce this by delaying the effect of the status change, and for the edge cases, possibly drop an outgoing messages even if we have a connection.

Consider this example:

reaction(t) -> out {=
  // Do stuff,
  lf_set(out, "Do things");
=} catch {=
  // Handle status changes in `out`
  if(out->connected) {
    lf_set(out, "Handshake");
  }
=}

out is currently NOT_CONNECTED. I think our semantics should ensure that when out gets connected that the message with "Handshake" is always the very first thing to be sent.

In the last example one could argue for a different semantics where the catch handler is invoked not in response to changes of the status of out, but instead every time the timer triggers and the status of out is not CONNECTED. I think maybe this is what you are thinking of Edward? Can we reconcile this with reactions triggered by multiple triggers? We can discuss this in our meeting later today

[erlingrj]

Had an interesting discussion with Edward yesterday where we concluded that:

1. The catch handler as proposed should not react to connection failures on the output port because it contradicts the signature of the associated reaction (Triggered by input ports, writes to output port)
2. A way to resolve the problem of simultaneous catch handlers and reaction bodies is to only execute the catch handler in this scenario.
3. We concluded that it would be useful to produce an event if we fail to send something over an output port, but the event must enter the program at a non-deterministic tag (like a physical action) and carry to tag of the failed transmission as a payload.
4. We might not need catch event for connection status changing from NOT_CONNECTED to CONNECTED. This can be done at the application level by just sending a message.

Some more questions:

1. Should we produce a fault event for each failed call to lf_set? Or only the first one? If we produce for every failed lf_set, then we have a kind of asymmetry between fault events for inputs and outputs.
2. What about the syntax for catching faults on output ports?
   We could do it through a reaction, but it is a little weird since the reaction body will never execute.

output out: int
reaction(out) {= =} catch {= =}

I still think it might be reasonable to trigger the catch handler associated with a reaction if lf_set fails:

input in:int
output out:int
reaction(in) -> out {=
=} catch {=
   // Handle connection failure on input
   if (in->failed) {
   }
   // Handle connection failure on output
   if (out->failed) {
   
   }
   
   // Handle simultaneous failure event and reaction event
   if (in->present) {

  }
=}

[edwardalee]

On question (1), I think there should be just one call, but then the question comes up, how would the reactor know when an output has been repaired?

On question (2), I think an output as a trigger could have another use:

output out: int
reaction(out) {=
  // Execute if a previous reaction has written to the output port.
=} catch {=
  // Execute when the output connection breaks.
=}

A problem with triggering the catch handler associated with the reaction where lf_set is called is that the user will assume that if the catch handler is not invoked, then the lf_set succeeded. This can't be known at this point because the send doesn't occur right away. Even if it did, then the lf_set call would need to block until it got an ack.

Another alternative might be to introduce another event for each output port, e.g. out_connection. Then you could write

reaction(out_connection) {=
  if (out_connection->disconnected) {
    do something;
  } else if (out_connection->connected) {
    do something else;
  }
=}

In fact, we could also use this strategy for inputs... This would make it clear that the exception handler is not invoked at the same tag as the associated reaction.

Alternatively, we could define just one exception event, and then define reactions like:

reaction(exception) {=
  if (exception->type == DISCONNECT) { ...}
=}

[tanneberger]

Just to throw another idea into the ring:

What stops us from introducing a new event type, that behaves like a physical action but is triggered by the runtime?

reactor Node {
    output x: int;
    
    reaction(connection_state_update) {=
      // handle the network failure
      
      printf("Connection %i has now state %i", connection_state_update->conn_id, connection_state_update->new_state);
    =}

}

[edwardalee]

We can introduce new exception types, but we need to avoid a proliferation of keywords. This is why I suggested an "exception" event type. It too would be like a physical action.

There is a subtlety with decentralized coordination: advancing to the tag of an exception event would have to subject to STA, which means that an intact connection could block the handling of a broken connection. This might be ok, but it's another subtlety among many with decentralized coordination.

[erlingrj]

On question (1), I think there should be just one call, but then the question comes up, how would the reactor know when an output has been repaired?

Initially I was thinking that a repaired connection (input or output) should also produce such an event.

On question (2), I think an output as a trigger could have another use:

output out: int
reaction(out) {=
  // Execute if a previous reaction has written to the output port.
=} catch {=
  // Execute when the output connection breaks.
=}

A problem with triggering the catch handler associated with the reaction where lf_set is called is that the user will assume that if the catch handler is not invoked, then the lf_set succeeded. This can't be known at this point because the send doesn't occur right away. Even if it did, then the lf_set call would need to block until it got an ack.

Good point, might make sense to decouple it from the reaction yes.

Another alternative might be to introduce another event for each output port, e.g. out_connection. Then you could write

reaction(out_connection) {=
  if (out_connection->disconnected) {
    do something;
  } else if (out_connection->connected) {
    do something else;
  }
=}

In fact, we could also use this strategy for inputs... This would make it clear that the exception handler is not invoked at the same tag as the associated reaction.

Alternatively, we could define just one exception event, and then define reactions like:

reaction(exception) {=
  if (exception->type == DISCONNECT) { ...}
=}

A single event and then checking might be a good idea. But if we should encompass repaired connections, and I think we should, then exception might not be the right word. How about network or communication?