workers management

[comboy]

Currently every worker finds job for itself on his own (based on which instances are free to take, and based on test suites that are left to run for differnt commits). This works suprisingly well, but:

• some queue is needed to let user push task
  ** like rerun test suites for given commit
  ** or handling webhooks from github to avoid polling (or at least speed things up)
• workers should be easy to manage, and web interface should display info about what are they doing currently

And probably resque will come to the rescue. It's not perfect solution, and it may be replaced later, but it's something stable and solid that should do the job (not necessarily in an optimal way). Issues with resque:

• no numeric priorities, which could be handy to make sure commits are run in proper order
• once worker fetches job from the queue, it should handle it, but it may turn out that there is no currently available instance, so it will have to wait until some instance is free, because we cannot easily manipulate queue, and order matters
  ** so basically you will have to start many workers (for example 9 if you have 3 project instances for project with 3 test suites), and it's available project instances number that will decide how many test suites can be run in parallel, not number of workers - not nice
• workers will have to test commit dependencies (commits that should be run before) on their own
  ** which in some rare edge case may lead to 2 workers running the same test suite (in separate instances) at the same time, which is ok, except for the waste of time before getting result about given commit
• it makes it unable to assign priorities to projects (unless each one have different queues)