Synchronize locked resources between multiple jenkins instances

[mPokornyETM]

What feature do you want to see added?

Our jenkins environment is bigger and bigger. And possibly maintaining time is lower and lower . We will to use more jenkins instances to eliminate this problem. But we need to synchronize some (not all) resources between. In other words yue has jenkins1 which use resource r1 in that case shall jenkins2 wait till resource r1 are unlocked on jenkins1. When I tell jenkins instance I mean really 2 different jenkins built-in nodes.

This will also help to make jenkins redundancy. So we can run 24/7 without distribution of build process. But there are many other thinks in jenkins which break this idea. But split into smaller environments will help us too.

Upstream changes

No response

[jimklimov]

I believe currently opensource Jenkins does not directly support clustering like this. You can have same resources or agents known to both instances, but they would be used independently.
Probably you can make a PR for a backend of lockable resources manager to check not only data state in memory (which was not trivial to synchronize as is, to avoid races) but also to consult with a remote instance - covering timeouts, network outages, other-server-crashes, etc.

Generally, from context heard over the years, I think Cloudbees equivalent of Jenkins has clustering as an enterprise feature.

Also you can google for "Gearman" setups that distribute jobs between several independent Jenkins controllers.

[mPokornyETM]

this might helps

https://github.com/jenkinsci/node-sharing-plugin/tree/master/nodesharing-lib/src/main/java/com/redhat/jenkins/nodesharing
https://plugins.jenkins.io/node-sharing-orchestrator/

[mPokornyETM]

I have much more ease idea. It can be don in this plugin and also in groovy as well

We can you existing API.
You need an extra instance, which handle all resources. This is mostly fine, because so small Jenkins instance does not need so many HW resources. Let we call it here lockable-master. Then your current instance will be called lockable-slave

Just add property 'masters' like here https://github.com/Signiant/super-jenkins#2-configure
This will indicate, that lockable-slave shall interact with other Jenkins instances (multiple lockable-master). (Yeas multiple, because we can also provide hot-by redundancy, but this is an other issue)
Then the implementation looks like:

• do the lock on lockable-slave, because when the resource is currently in use in lockable-slave, it will be also locked in the lockable-master, and you can spare html post request.
• send post request to lockable-master. In case it is not free, send it in the loop (f.e every second). In case the first one does not answer send it to next one.
• after confirmation just run given Groovy closure in lockable-slave.
• 'on unlock' must send unlock request to lockable-master and then free resource on lockable-slave.

It must be also possible to unlock the resource on lockable-master. That means the lockable-master must send request to lockable-slave.

It shall be also possible, that we mirror the state between instance:

• manually. Just add some button only for Jenkins-administrators.
• periodically. Like a chron job.
• on every lock() call, but in this case only for 'affected' resources. This might be challenge in case of 'labels' request.

Limitations:

• We shall disable user-actions on lockable-slave. It will be enough to do it in lockable-master. No body from end-users will handle about our Jenkins-instance-farm, and we can spare many of possible mistakes. Just caring about the lock() step.
• I thing we can spare the implementation free-style projects

Inspired by:

• https://github.com/jenkinsci/vrealize-automation-8-plugin/blob/2774798ae62db9a25857886a2898ca7d021e7c17/src/main/java/com/vmware/vra/jenkinsplugin/vra/VraClient.java#L135
• https://github.com/jenkinsci/node-sharing-plugin/tree/master/nodesharing-lib/src/main/java/com/redhat/jenkins/nodesharing
  https://plugins.jenkins.io/node-sharing-orchestrator/