Stop polling for pending jobs

[Aradhya-Tripathi]

Problem
Once a job is triggered by press on the build/other servers (Archive, New Bench etc.), a job is created on the press worker that polls for job status every 5 seconds (via hooks).

This problem causes obvious load on the servers.

Solution
Using sockets in agent (out of the box with flask-socketIo).
We also need to use gunicorn with eventlet. To allow for async connections

Steps

• Once a job is triggered by press on the server create a on_[success, failure] callback.
• These callbacks will trigger events / emit which let the client know the status of the job.
• This also requires a pub-sub mechanism as socketIo will have to emit these callback events from rq-worker processes.

Problems caused by the solution

• Job's have a certain order of being processed once completed this might break for which also needs to be address.

[ankush]

This description is not comprehensive enough.

Once a job is triggered by press on the server create a on_[success, failure] callback.

Why can't agent callback just be a simple HTTP request like job creation calls? 🤔

---

To answer the question of socket-async issues: you'll most likely have to run a separate thread that consumes messages from Redis pub-sub channels and sends it back to press. Don't mix any async and sync code, that's madness waiting to happen.

Something like this

web worker -> |              | 
              | Redis PubSub | -> socketio ----network---> press
bg worker  -> |              |

[ankush]

First expand this document:

• List down all the bidirectional communication that is required and what type of communication is that.
• Callbacks? Just information? continuous stream processing?
• List down all current usage + possible expansion that might be useful.

@adityahase or @tanmoysrt can help clarify the requirements.

[Aradhya-Tripathi]

First expand this document:

• List down all the bidirectional communication that is required and what type of communication is that.
• Callbacks? Just information? continuous stream processing?
• List down all current usage + possible expansion that might be useful.

@adityahase or @tanmoysrt can help clarify the requirements.

Spoke to @adityahase, and half duplex seems fine for this for now