Feature Request: Dynamically update load balancer configuration

[jamesdbaker]

I don't know how difficult this would be, but it would be a useful feature to be able to dynamically update/reload the load balancer configuration so that new servers can be added or removed.

The use case to this is an auto-scaling translation server farm, where new servers are added and removed as required to handle a variable volume of data. At the moment, as far as I can see, this would require the load balancer to be restarted with an updated configuration file which obviously isn't ideal and may result in queries being lost.

[ivan-zapreev]

Dear James,

Thank you for you feedback and request. I agree that such a dynamic loading functionality would be useful. It should not be hard adding it either, I estimate about 40 work hours for this feature.
(I assume that the updated configuration is to be stored into the config file and that one shall be able to add/remove servers to the configuration from the load ballancer's server console).

What is the priority of this feature? I got a request for Secure Web Sockets support which I was planning to finish by the first of June. I did not start on this yet, so the feature above is more desired I could start with that one first. It might however delay the delivery of the secure web sockets.

Kind regards,

Ivan

[jamesdbaker]

Hi Ivan,

I've moved on from the organisation funding this work, so no longer have any say over the priorities; that decision will have to come from those continuing to fund your work.

This is for a separate project for a different organisation where we're hoping to leverage your good work, so more of a suggestion for future work than a tasking!

(That said, to clarify how I would see it working, ideally there would be some way of programmatically updating the configuration rather than via the console, so that this can be done automatically as new servers are added rather than manually).

Thanks,
James

[ivan-zapreev]

Dear James,

Thank you for the clarification! Adding such a remote configuration via some remove administration client would require some more work indeed. However the initial basic configuration update through the ssh/screen option would be rather easy to implement for the start. Once I have the other REMEDI requests finished, I can look into this as well. For now I myself in a bit of a transfer phase. I currently finish up my research at Technical University of Delft and then I will be moving to Philips Medical from the 1st of june. So after that I can again look into your, definitely very useful and interesting, suggestions. So let us keep in touch.

Kind regards,

Ivan

[ivan-zapreev]

Additional ideas on the subject from Nigel, originated from (#21 (comment)):

As far as the 'hot' configuration changes idea goes, I was thinking that
one could go one of two ways (or even both). On starting a server, could it
notify/register a specified server that it is now ready? I'm guessing the
JSON based msg protocol could be simply extended for this purpose?... The
point(s) to send notification to could be specified in start-up command
line or in the associated config file. Balancers could then propagate msg
to other balancers (much like DNS process I guess). The other way is for
balancers to query some form of register at regular intervals. Downside of
the second is the introduction of another bit of 'infrastructure'. However,
the balancer doing a regular polling is possibly worth introducing. One
thing we've noticed is that if a server goes down, balancers will still
report they can access those servers. A regular 'health-check' would be
useful so that we get an indication if a translation service is
unavailable, so not to try that service and do an auto reboot (which
coupled with hot configuration would give us potential for a degree of
system self-maintenance).

As for writing back out to config files - I guess the 'register' option
would give the current. I think modifying initial configs is probably a bad
idea, but an output of current (running) config would potentially be
useful. Again I can see tow useful modes, one a run-time query through
extending the JSON msg protocol and/or two a runtime server command to dump
config to a file. I guess the former is the most flexible as it divorces
what is done with the configuration from the system running it.

Does that make sense? In part I guess it is a question of what is least
effort / most easily achievable! It's a "would be nice to have" rather than
show stopper.

[ivan-zapreev]

A note on dynamic balancing of the servers:

I remember I built in and tested the load balancer feature that would detect if a server goes down and back online. So once a translation server is down and the websocket connection with it is closed the load balancer automatically stops sending translation requests to it and re-weights the balance weights among the remaining servers. The balancer will periodically check if the offline translation server is back online and try to re-establish the connection. Once the connection is re-established the balancer again re-estimates the weights and starts using the server.

Allowing servers to register by the balancer does sound a good idea but that would require some infrastructural re-design. My only major concern here is however with security. This way one allows any server to be connected to a balancer thus intercepting messages. What if this will be a malicious server? However given that we have SSL based security in place this may be still a safe option.

I roughly estimate the development effort for this feature from 2 to 4 work weeks. If there is budget for it I would be glad to work on it.