Job entity

[jprachir]

• Add memory to job req
• Add priority to job req

Current flow:

job request -> bidding -> immediate allocation

But,

job request -> bidding -> {immediate allocation/ queue for future execution/ forward to better suited peer (if it predicts failure or better efficiency)/ reject if impossible}

• estimated_runtime object has to use for scheduling logic
• local queue?

[frs69wq]

• Agreed to add memory to job requirement, but have to decide how to express this requirement. the --mem flag of sbatch is per node. Finer specification can be per CPU or GPU, but let's start per node.
• Not sure about dealing with priority for the moment
• I'm not satisfied with the current flow either. The current code of the demo just pretends to be a distributed system but it's not ... at all.
  - A job could be submitted to any agent in the network (and not to "the network" i.e., all agents at once in the demo), so a first step in a truly distributed system would be to broadcast the job description to all the agents, so that each of them can compute its bid.
  - This implies to create and maintain an P2P overlay of agents so that they can talk to each other. A first simple version is to ensure that all agents know each other and can communicate before doing anything and assume a static network of agents. A more realistic and fault tolerant implementation would be to have a true P2P overlay (e.g. a distributed hash table) with a broadcast algorithm. With a number of agents in the tens, and not a big churn, that should be okay performance wise.
  - this overlay can be used to reach consensus on whose agent is the bid winner.
  - and yes we should have a way to simulate the execution of jobs on the selected HPC system. The demo doesn't allow that as there is no discrete event simulation in it, so no way to handle the job runtime. Again the complexity can vary from immediate allocation to having a queue/batch but only with the submitted job and a "fake load" (artificially changing the amount of available resources over time), or a real load that may cause jobs to wait in the queue and would imply to take local estimated wait times in the bidding process.

My main problems at the moment are that 1) the problem and objectives are not well defined; 2) the demo is too simplistic and do thing that would be impossible in a distributed system; and 3) there is no simulation of the job execution anywhere.

I don't think that trying to build something from the demo (or the rest of the code base in the repo) will be easy. I have the feeling that starting from a discrete event simulator of distributed systems and scheduling on batch-managed clusters and then code what is needed to do the network of agents, the competitive bidding and all would be a more reasonable approach.

I have a good candidate tool to do that (WRENCH) but that doesn't solve the issue of the definition of the problem and objectives. The overleaf doesn't help at all for that either. I started to draft some working hypotheses at the end of the document. It needs to be augmented and that we all agree on these hypotheses if we want to get a paper. I'm afraid it will take more than a week though...