Design and Implement active coordinator.

[satyamjay-iitd]

Why?

Some (maybe all) distributed applications require some form of runtime coordination. Just starting the application is not enough, which is what generic_jctrl does.

Scenarios where coordination might be required

1. Fault Tolerance:- An active coordinator(AC) can monitor failures, and in response take some actions to heal the cluster.
2. Runtime Reconfiguration:- An AC is required to do this correctly.
3. Implementation of distributed protocols.

What?

AC cannot be just another actor. Why?
An Actor doesn't have a view of the entire cluster.
AC will require few interfaces that the actor cannot have (without sacrificing the simplicity of the actor). Few that come to mind are:-

1. Asking the node to start/stop an actor.
2. Asking the node current status of the actor.
3. Having the state of the entire cluster of the job that it is managing.

How?

Implementing AC requires careful considerations. Few designs that are at-least worth discussing

Design 1

AC is just another actor with extra privileges.
From a normal actor's POV it is receiving/sending messages from just another actor.

Modification Required for this Design

1. Actor: None
2. Node: None
3. SpecialActor(Framework's Responsibilities):
   3.1. New special actor will have to be written.
   3.2. Logic to send start/stop actor can be provided by the framework itself. User can simply call start(ActorAddr)
   3.3. State of the Cluster ClusterState<GS, NS, AS>

struct ClusterState<GS, NS>{
      global_state: GS,
      node_state: Map<NodeId, NS>,
      actor_state: Map<ActorId, AS>
}

    3.4. Handle messages coming from other Actors. Framework will have to spawn a receiver server for the other actors to send message to AC.
    3.5. Manage timers which when expire trigger some user provided logic.

4. SpecialActor (User's Responsibility)
   4.1 Decide what GS, NS, AS would be.
   4.2 Provide handler that will be called when the AC receives a message from other actors.
   4.3 Provide handler that will be called when the various timer in the AC expires.

Design 2

AC is a new kind of entity. Actor and nodes are now aware of this new type of entity

Design 3

Modify Actor such that they can now spawn other actors.

Modification Required for this Design

1. Actors
   1. Actors can now be related. An actor that spawns another actor becomes parent of that actor.
   2. Parent of the actor should be notified of the children's failure.

[codenet]

IMHO, a controller actor would be the best way to keep things most general and flexible. I don't agree that we will have to give up the simplicity of actors if we make controller as just an actor. Letting any actor start other actors is the most general approach. For our distributed protocols, we may spawn a "controller actor" that spawns all other actors.

The only general thing (which might complicate actors) we would need is to make every actor heartbeat with its parent actor at regular intervals. Parent actors can get a message when it misses heartbeats from its children and decide what to do, like spawn another actor.

The complexity comes from exposing node interfaces and getting a view of the entire cluster, not from letting anyone call those interfaces.

[Aman-Hassan]

With respect to Design 3, when considering a self-reliant protocol like the chainpaxos protocol, since each server sends heartbeats to the previous server (with head sending to tail), a parent-child actor relationship would form a cycle of actor dependency (?). So, if one actor fails and has to be respawned, we would have to somehow link the actors that depended on the failed actor with the newly respawned actor?

[satyamjay-iitd]

Ok since there is a consensus that Design 3 seems to be more elegant from User's POV (even though it requires more work upfront). I will flesh out some its details.

What changes from User's POV:-

1. User is now able to spawn other actors from the actor. First question is how?

Possible Interface 1:- 
Processor instead of returning OMsg now returns Enum(OMsg, ActorControlMsg).
enum ActorControlMsg{
    Start(node_addr, actor_addr),
    Stop(actor_addr),
}

Possible Interface 2:-
Router(Sender) instead of returning ActorAddr now returns Enum(ActorAddr, ActorControlMsg).

Possible Interface 3:- User provides a `fn(&ActorContext, &actor_addr) -> NodeAddr`, say child_spawner.

Note that when the actor wants to send message to another actor for the first time, it has to consult
the Node for the  receipent's physical addr (IP:Port). And if the receipeint doesn't exist the actor
simply crashes. Instead of crashing, the node can run the child_spawner to spawn the child actor.

struct ActorContext{
   children: Map(ActorAddr -> (NodeAddr, Status))
}

ActorContext might be required in all the above designs.

2. User now needs to provide lifecycle hooks

fn onChildStart(&mut ActorContext) -> {}

fn onChildStop(&mut ActorContext) -> {}

Changes in Framework:-

1. How to report actor's state to its parents.

Design 1:- Pull Based
A node will know the state of all the actor it manages.
A node will also need to know the state of actor other nodes manages (not all just
the ones that manages the children of my actors).
It can ping other nodes to get that state.
If the state changes it will call the user provided hook.

Design 2:- Push Based
Node of the children will notify the node of the parent of any change.
The node now needs to know who the parent of an actor is (for all the actors it manages).
There also needs to be a timeout, and if within that timeout a node hasn't sent any 
message assume the node died, and consequently all the actor it managed died.

Design 3:- Let user figure it out.
Its user's responsibility to figure out a way to monitor the lifecycle of its children actors.
Framework will not provide any help.

[rsrs2013]

Can we have facility, where the user of the framework can decide, to which actor do I want to send my heartbeat to ? This will be more flexible, rather than just having it to send to its parent which has spawned me.

[satyamjay-iitd]

Cons:- User has to provide extra parameter (extra headache)
Pros:- ?

[satyamjay-iitd]

Can someone research how Akka does it? And post here their findings.

[rsrs2013]

In the approach you suggested, can you elucidate how will the head of a chain send the heartbeat to the tail ?

Pros: Keep the default behavior as sending to the parent, but by giving a provision to specify to whom my heart beat has to be sent, you are giving more flexibility to design any kind of protocol that I want to.

[satyamjay-iitd]

I am assuming you are talking about chain paxos.
Can you tell me what is the purpose of the heartbeat in this protocol?

[codenet]

Design 3 seems best. We can implement a few "built-ins" and let the user choose from built-ins or implement their own. For instance, there can be a built-in generator_out that just outputs HeartBeatMsg at a configurable timeout. But user is free to route it to any actor. They shouldn't be tied to routing it to the parent actor.

Some suggestions:

• router should return Vec<&Addr>. An Addr can be an ActorAddr or a NodeAddr. Let the user figure out what kind of message they want to send and to whom.
• Node has its own well-defined NodeIn and NodeOut messages. SpawnActor is in NodeIn.
• The resolution of actors/nodes should not be the responsibility of the user.
• There are no special lifecycle hooks. Actor should handle heartbeats just like any other message.

[satyamjay-iitd]

But then what is the semantics of Parent -> Child relationship then.
It seems to me you just want a way for an actor to spawn other actors.
Framework will not provide any guarantee that the actor was started successfully or not, its the starter actor's responsibility to check if the other actor started (using perhaps HBs).
Purpose of returning NodeAddr is also not obvious to me.

[satyamjay-iitd]

For Design 3:-
I was thinking to add new variant RouteTo::NewActors

pub enum RouteTo<'a> {
    /// Drop the message; it will not be sent to any actor.
    Blackhole,
    /// Send the message back to the sender.
    Reply,
    /// Route to a single actor by address (either borrowed or owned).
    Single(Cow<'a, str>),
    /// Route to multiple actors by addresses (either borrowed or owned).
    Multiple(Cow<'a, [String]>),
    /// Spawn these actors and send messages to them
    NewActors(Vec<(NodeAddr, ActorAddr, LibName, OpName, Payload)>),
}

Issues that need be solved for this approach

1. How to return the status of spawning the actor. For example how will the node tell the actor that actor was spawned successfully or not.
2. How will actor know the topology of the nodes.