Build service update

[BeckerFrank]

On 30/04/2025 03:10, Denis Roy via eclipse.org-committers wrote:
Folks,

I was overly optimistic about the recovery process of the storage cluster that is hosting Jenkins data (job configurations, etc). Recovery is likely, but not guaranteed, and the timeline is not certain.

Unfortunately, we’ve been relying on this technology to not fail in the manner it has, and the end result is that the availability of Jenkins backup data is not consistent for each hosted instance. While we may be able to recover some (and hopefully most), there are instances for which we don’t have a recent backup.

To avoid prolonging the outage any further, we have two options:

We can provision a Jenkins instance with the last known version of the Jenkins data. This data is the last snapshot prior to migration to the storage cluster, and, depending on when your instance was migrated, may be weeks, or up to several months old.

We can provision a new, bare Jenkins instance for your project. If your build is defined in pipelines alongside your code, recovery may be more straightforward.

You can also choose to wait for the storage cluster to recover, so that we can either continue its usage, or extract data from it to provision a new instance. We do not, however, have an ETA for this recovery, nor do we know with any level of certainty that it actually will recover.

We recognise that none of these options are ideal, and we will work with your project to restore service as best we can. We’ve compiled a list[1] of Jenkins instances as well as the date of the last-known-good data, to help with your decision. When you’re ready to proceed, please file an issue on HelpDesk[2] with Project Lead +1 and the team will assist as soon as possible.

Once service is restored, we will perform a postmortem and assess the path forward and share that information with you.

Regardless of what we find in the postmortem, we obviously missed in not ensuring backups of these jobs were captured regularly in case of a catastrophic failure such as this one. We sincerely apologize and thank you for your understanding.

[1] https://docs.google.com/spreadsheets/d/13zNUbwAHkYacC193HFwgZOd1fqn-N9cyzXf_lHke07s
[2] https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/new

--
Denis Roy
Director, IT Services | Eclipse Foundation
Eclipse Foundation: The Community for Open Innovation and Collaboration

[BeckerFrank]

Sorry for We can provision a new, bare Jenkins instance for your project. we have the Jenkins Jobs in /org.eclipse.mylyn.releng/multipass/roles/jenkins/files/jenkinsci

[merks]

I know of at least a dozen instances that have done "We can provision a Jenkins instance with the last known version of the Jenkins data." and while there is some loss of data, at least most of the stuff is still there, minus the changes you've done since the backup. For most of these instances, those changes have been minimal and are easily recovered. I think the bare instance is the worse of all possible choices; after all, you can delete things that are restored and arrive at a bare instance via the route.

[ruspl-afed]

Since I didn't expect others to care about Mylyn releng problems, I've already created helpdesk ticket
Sorry if it was impolite from my side.

[merks]

Getting into the queue as early as possible is in my opinion the best approach! We can't wait for something that might never happen, and starting from scratch would only make sense if we had a few jobs that were purely pipeline jobs in git, and even then restoring seems more convenient.

[ruspl-afed]

Thank you for creating this discussion @BeckerFrank , I think we can close it since restoration is in progress already