Tests stuck for new contributors of Contributor Spotlight repo

[krisstern]

Service(s)

GitHub, infra.ci.jenkins.io

Summary

The jenkins/infra.ci.jenkins.io/website-jobs/contributor-spotlight/pr-head CI/CD check gets stuck every time a new contributor submits a pull request to the https://github.com/jenkins-infra/contributor-spotlight/ repo, and there is no way for the maintainers to either bypass it or trigger it manually.

For an example, see this PR:
jenkins-infra/contributor-spotlight#481

Reproduction steps

No response

[lemeurherve]

As I don't have any permission on that repository I can't do anything about it.

Damien suggested in #jenkins-infra to "only disabl[e] the branch requirement on the infra.ci [status check] and be done with it. (If deployment fails, then you can still see the infra.ci build to reproduce)"

[timja]

Isn't there another folder for these previews? like jenkins.io?

[lemeurherve]

WDYM by folder for these previews?

Related: we need to switch them to GitHub Pages cf #4958

[timja]

Netlify said that was a mistake and doubled the allowance and wiped bills not sure that we do need to

[timja]

Its this:

There's probably other issues around for this but imo it makes no sense for a build whose sole purpose is to provide previews from contributors who likely do not have write access.

For now I'll remove the branch protection setting for that.

[timja]

You should be unblocked @krisstern, although I don't think this configuration is right, probably better to wait for @dduportal to be back for that though.

[krisstern]

Thanks @timja! I have checked that we have indeed been unblocked now. Yes, I will wait until after Damien has returned in a few weeks to revisit this issue.

[dduportal]

As I don't have any permission on that repository I can't do anything about it.

Damien suggested in #jenkins-infra to "only disabl[e] the branch requirement on the infra.ci [status check] and be done with it. (If deployment fails, then you can still see the infra.ci build to reproduce)"

Quick summary (food for thought):

• Non "trusted" users should not be able to execute custom code on infra.ci. At all. Usual contributors of websites are not trusted. That is the reason why the Jenkins setting to not kick a build is set up like this. We've been attacked by external contributors through PR in the past, trying to extract credentials from infra.ci.

  • Hence the proposal to make the check as "non required" and consider the PR preview feature as "optional".

• Might be interesting to reconsider even using our Jenkins Infra for PR preview and fully switch to Netlify PRs? Would be easier. The reason why we have NOT done this in the past was because Netlify (with PR preview builds) and infra.ci (for production deployment) where quickly out of sync for NodeJS, Hugo, Ruby and others tools. Also, Netlify does not have Docker available (www.jenkins.io builds). We quickly ended up in perfectly fine PR preview but production deployment failure.

  • Using Netlify for production is not possible given we have not enough credits.

• Deploying PR preview builds to ci.jenkins.io would make it easier. But since a credential is required to publish to netlify (and ci.jenkins.io is always considered unsecure and untrusted since it is public), we moved the PR preview to infra.ci which creates this challenge.

Since the context has changed (more docs. contributors, more dynmaic, cleaner tooling, etc.) here is the idea discussed in the past months with @smerle33 and @lemeurherve:

• ci.jenkins.io is used again for PR previews. But these PR builds only generates a docs.zip archived artifacts and that is all.
  • Same pipeline and tools as the production deployment on infra.ci. If it fails on infra.ci, it won't be due to user error or tooling error, but only production issue.
  • Contributors will have only 1 build on ci.jio for PR checks.
• We create a new web service which is responsible to retrieve from ci.jenkins.io the docs.zip generated by a website build and deploys to netlify.
  • This pattern already exists today with incrementals.jenkins.io
  • The Netlify credentials are not available within ci.jenkins.io
• Bonus: we can optimize by inttroducing a pipeline library for generating website, constraining the way we do it (sanitize package.json to avoid custom commands, etc.)

[timja]

Makes somewhat sense but I would expect an untrusted node pool could do the same.

The Jenkinsfile already only loads the credentials at the time of deployment.
The pipeline isn't modifiable.
It uses a custom binary not relying on anything in the build for deployment, e.g. an npm package.

As long as users cant '/usr/local/bin': https://github.com/jenkins-infra/jenkins.io/blob/b7ede805914be25c73f524a3e98f34e0e0cc486a/Jenkinsfile_k8s#L63

then it should be safe.

Not saying there's no vector but I think most of them are similar and it comes down to whats in the resulting files.

I get there may be datadog keys or something else on the VM image itself if its running a VM but that would be the same for ci.jenkins.io agents wouldn't it?

---

Not objecting to the proposal but questioning if its 100% needed.

[dduportal]

Makes somewhat sense but I would expect an untrusted node pool could do the same.

The Jenkinsfile already only loads the credentials at the time of deployment. The pipeline isn't modifiable. It uses a custom binary not relying on anything in the build for deployment, e.g. an npm package.

As long as users cant '/usr/local/bin': https://github.com/jenkins-infra/jenkins.io/blob/b7ede805914be25c73f524a3e98f34e0e0cc486a/Jenkinsfile_k8s#L63

then it should be safe.

Not saying there's no vector but I think most of them are similar and it comes down to whats in the resulting files.

I get there may be datadog keys or something else on the VM image itself if its running a VM but that would be the same for ci.jenkins.io agents wouldn't it?

Not objecting to the proposal but questioning if its 100% needed.

Past attacks were using modified package.json with custom commands. They installed a trojan looking for environment variables in /proc and sending them to a remote server. You only need basic commands. It also hd a fallback in pure JS (bypassing tentatives to restrict path). I don’t see how we coule restrict from such attacks. Pipeline library could help.

I agree with you on the fact the amount of credentials available to the attacker are quite low (datadog key, netlify key, github app token with only read and ttl of 1h). However the next logical step of attackers would be transveral with the jenkins agent. Since we have sensitive stuff on infra.ci, i would rather not take any risk.

A webservice for this would be quite easy to build and cheap to operate. And a lot of benefits for users in the end