Clustering

[volfco]

Curious if there is any thoughts about clustering right now. The docs make references to Kubernetes, but from what I can tell the current code is purely focused on running AuraeScript against a local instance of aurae.

Is the current idea to fully offload this to an external system, such as Kubernetes, or have a cluster aware layer that is accessible from AuraeScript?

I ask because this very closely mirrors ideas I've been playing with and would like to contribute.

[krisnova]

So we will need to develop a "higher order" mechanism that serves as a clustering service.

We are building out the raw functionality (likely with more generic names) to enable the Kubernetes control plane to schedule work directly on Aurae. (Note: we will likely need to build a kubelet shim or virtual kubelet implementation)

Another option is to build an Aurae specific / bespoke clustering service that enables similar functionality to the Kubernetes control plane.

There are pros/cons to either approach and I believe we should build towards support both.

Are either of these approaches what you had in mind?

[volfco]

The idea I had in mind is having auraed or creating aurae-clusterd (that would talk to auraed), which would publish resource information to some shared metadata store. Clients would then connect to a known endpoint to interface with the cluster.

Then to interface with the cluster, you could do something like the following:

let cluster = cluster_connect();
let runtime = cluster.runtime();

let selector = #{
  required: #! {
    cpu: 1,
    memory: 512
  },
  requested: #! {
    cpu: 2,
    memory: 1024
  }
};

let example = cmd("cat /etc/resolv.conf");
let job = runtime.exec(example, selector);

# job.target() would be an object with information on where the job is being run

For Kubernetes integration, the two options I see are

1. Just running aurae inside pods and using the same approach as above. The end result being just using Kubernetes for running aurae and aurae doing everything else
2. Run Aurae via Jobs, where user submits a job like kubectl aurae submit hello.aurae --cpu-requests 100m --memory-requests 128mb. Internally, the job would be submitted to Kubernetes that would run a pod launching aurae, and passing the hello.aurae to the daemon in a one off fashion

For the 2nd approach, the only thing that would need to be done to aurae would be to add arguments that let it read a file and execute it on launch (and then exit after)

[krisnova]

So I would prefer to never "run Aurae in Kubernetes". I think that is how we conflate the layers of the stack. Check out my post on System Awareness which basically says we should "run Kubernetes ON TOP OF Aurae".

I think Aurae should sit below Kube. Maybe we run some VMs with Kubernetes that have Auraed as a pid1 or something but that is kind of beside the point. I think the "meat" of the conversation is what runs on the nodes directly.

I think your code snippet is spot on. Your idea is EXACTLY the direction we want to move towards.

I think we have some architecture and considerations in front of us. For example does the Runtime subsystem work at the cluster level? Or do we introduce a cluster level subsystem called "ClusterSchedule" or similiar that then leverages lower level aurae functionality on the node as needed.