[JENKINS-70869] New option “clone” option to SCM-based retrievers to avoid `workspace/xxx@libs/` #55

jglick · 2023-03-02T19:52:17Z

With this new option you can avoid locking and updating the workspace/xxx@libs/ subdirectory at the job level; instead, the SCM is checked out directly into the build directory. This is particularly practical in combination with Git shallow clones. Reduces disk usage somewhat; may avoid contention in heavily concurrent jobs (since no lock is needed during SCM operations); and is generally simpler, bypassing issues such as that solved by jenkinsci/workflow-cps-global-lib-plugin#115.

Tested interactively with a large private library used by @cloudbees with >1600 commits and >200 files, using some combinations of both plain Git SCM source (ambient authn) and GitHub SCM source (PAT), with and without the new clone mode, with and without shallow clone option when using the new mode, and with and without caching. With clone mode on and using a shallow clone with GH PAT, could retrieve the lib for a build in a couple seconds in some cases (variable depending on GH response time), whereas a deep clone would typically take at least ten seconds.

basic impl
GUI config
tests
pack files into a JAR to reduce inode count? [JENKINS-70870] Save libraries as JAR files rather than unpacked #57
backport some improvements from [JENKINS-70870] Save libraries as JAR files rather than unpacked #57 so this is independently mergeable

…-lib-plugin into SCMSourceRetriever.clone

src/main/java/org/jenkinsci/plugins/workflow/libs/SCMSourceRetriever.java

jglick · 2023-03-10T15:40:14Z

Could be merged independently but I did a bunch of follow-on changes relating to this in #57 and it seems more straightforward to just merge them together (pending review).

…-lib-plugin into SCMSourceRetriever.clone

…/jenkinsci/pipeline-groovy-lib-plugin/pull/57/files/840cd6004501d8d9e52317288ebbbb86fddbd7a4..444bde7967e0fa5ad9242bb1c2e08a073b49f21e + jenkinsci@e5e95d0

…line-groovy-lib-plugin/pull/57/files/b7bd7741e05e751584afd78b1dc6f47dd06d4129..698ae6887eae199e4b5da9b478f5039265562350#diff-d9ea39311c121ef001c2131af309031434afafb5a19bd5961747f2c1f7082768R399

raul-arabaolaza · 2023-04-10T13:53:48Z

src/main/java/org/jenkinsci/plugins/workflow/libs/SCMBasedRetriever.java

+
+    @DataBoundSetter public void setLibraryPath(String libraryPath) {
+        libraryPath = Util.fixEmptyAndTrim(libraryPath);
+        if (libraryPath != null && !libraryPath.endsWith("/")) {


NiT: For my own personal knowledge, why using a hardcoded String and / instead of File.separator or Path?

Because this would be a relative path interpreted by FilePath and thus expected to use / regardless of the OS on which the controller runs.

raul-arabaolaza · 2023-04-10T14:01:25Z

src/main/java/org/jenkinsci/plugins/workflow/libs/SCMBasedRetriever.java

+    }
+
+    public String getLibraryPath() {
+        return libraryPath;


NiT: Can this eventually be part of an URL at some point of the usages? Like an API call somewhere in the execution stack?

Not sure what you are asking. This is a getter matching setLibraryPath as required by databinding (so you can round-trip a form in which a library is already defined).

jenkinsci/pipeline-groovy-lib-plugin#55 https://issues.jenkins.io/browse/JENKINS-70869

MarkEWaite

Two hours of testing showed no issues with the cloning that I enabled with modern SCM and with legacy SCM. I've made an optional comment in one of the help files in case additional help might benefit users that enable caching.

I'd love to have an incremental build of this so that I can include it in my plugins.txt file.

src/main/resources/org/jenkinsci/plugins/workflow/libs/SCMBasedRetriever/help-clone.html

MarkEWaite · 2023-04-10T18:04:37Z

src/main/java/org/jenkinsci/plugins/workflow/libs/SCMBasedRetriever.java

+     *
+     * <p>Used to prevent {@link #libraryPath} from being used for directory traversal.
+     */
+    static final Pattern PROHIBITED_DOUBLE_DOT = Pattern.compile("(^|.*[\\\\/])\\.\\.($|[\\\\/].*)");


I don't understand the left anchored | character in the regular expression. I'm accustomed to | as an OR expression combining two expressions where either can match.

In my interactive testing, all the patterns that I tried were correctly detected, so as far as I can tell, the regex is correct. However, I would like to understand the left anchored | character for my own benefit.

Not sure offhand. Someone else wrote this for a security fix; I am merely moving it from

pipeline-groovy-lib-plugin/src/main/java/org/jenkinsci/plugins/workflow/libs/SCMSourceRetriever.java

Line 95 in 06b0eff

static final Pattern PROHIBITED_DOUBLE_DOT = Pattern.compile("(^|.*[\\\\/])\\.\\.($|[\\\\/].*)");

Not part of a security fix, it was aded when libraryPath was added in jenkinsci/workflow-cps-global-lib-plugin#101.

(^|.*[\\\\/])

Either the start of the line, or any text followed by a directory separator

\\.\\.

Two dots

($|[\\\\/].*)

Either the end of the line, or a directory separator followed by any text

Vlatombe

LGTM

jglick · 2023-04-11T18:49:33Z

an incremental build of this

https://repo.jenkins-ci.org/incrementals/io/jenkins/plugins/pipeline-groovy-lib/649.v4a_8971b_1a_d8b_/ is getting published normally. I am not clear on what MarkEWaite/docker-lfs@a775455 was picking up, as jglick@c722f7acf8c2 does not exist.

MarkEWaite · 2023-04-11T18:58:39Z

an incremental build of this

https://repo.jenkins-ci.org/incrementals/io/jenkins/plugins/pipeline-groovy-lib/649.v4a_8971b_1a_d8b_/ is getting published normally. I am not clear on what MarkEWaite/docker-lfs@a775455 was picking up, as jglick@c722f7acf8c2 does not exist.

As far as I could tell from the jobs, 652 was in build 6 but not published as an incremental. I downloaded it from the build 6 artifacts and used it directly because it was not available as an incremental. I wanted to be sure that I was testing the most recent build of the pull request.

Then I fell back to use 649 in plugins.txt because it was available as an incremental. That's what I'm running now, on the assumption that differences between 649 and 652 are not critical to the function of the pull request.

jglick · 2023-04-11T19:20:53Z

not published as an incremental

Probably predated disabling of PR-merge mode.

…-lib-plugin into SCMSourceRetriever.clone

ysmaoui · 2024-02-26T09:59:05Z

In case someone is looking for how to use this ( like me )

in jenkins system config, global shared lib config:

config-as-code

New option SCMSourceRetriever.clone to avoid workspace/xxx@libs/

ce94834

jglick added the enhancement New feature or request label Mar 2, 2023

jglick mentioned this pull request Mar 2, 2023

Call GroovyClassLoader.close when build completes jenkinsci/workflow-cps-plugin#667

Merged

Merge branch 'master' of https://github.com/jenkinsci/pipeline-groovy…

fb585f4

…-lib-plugin into SCMSourceRetriever.clone

dwnusbaum reviewed Mar 2, 2023

View reviewed changes

src/main/java/org/jenkinsci/plugins/workflow/libs/SCMSourceRetriever.java Outdated Show resolved Hide resolved

This was referenced Mar 3, 2023

[JENKINS-70870] Save libraries as JAR files rather than unpacked #57

Draft

[JENKINS-69573] retrieve sources in tmpdir and rename #58

Open

jglick requested a review from Vlatombe March 23, 2023 14:08

jglick added 3 commits March 24, 2023 08:09

Merge branch 'master' of https://github.com/jenkinsci/pipeline-groovy…

ee7a558

…-lib-plugin into SCMSourceRetriever.clone

Factor out SCMBasedRetriever roughly backporting https://github.com…

76ad53e

…/jenkinsci/pipeline-groovy-lib-plugin/pull/57/files/840cd6004501d8d9e52317288ebbbb86fddbd7a4..444bde7967e0fa5ad9242bb1c2e08a073b49f21e + jenkinsci@e5e95d0

More advice on clone, backporting https://github.com/jenkinsci/pipe…

4a8971b

…line-groovy-lib-plugin/pull/57/files/b7bd7741e05e751584afd78b1dc6f47dd06d4129..698ae6887eae199e4b5da9b478f5039265562350#diff-d9ea39311c121ef001c2131af309031434afafb5a19bd5961747f2c1f7082768R399

jglick changed the title ~~New option SCMSourceRetriever.clone to avoid workspace/xxx@libs/~~ New option “clone” option to SCM-based retrievers to avoid workspace/xxx@libs/ Mar 24, 2023

jglick marked this pull request as ready for review March 24, 2023 13:20

jglick requested a review from a team as a code owner March 24, 2023 13:20

jglick changed the title ~~New option “clone” option to SCM-based retrievers to avoid workspace/xxx@libs/~~ [JENKINS-70869] New option “clone” option to SCM-based retrievers to avoid workspace/xxx@libs/ Mar 24, 2023

jglick requested a review from dwnusbaum March 28, 2023 12:12

dwnusbaum approved these changes Apr 10, 2023

View reviewed changes

jglick requested a review from MarkEWaite April 10, 2023 13:37

raul-arabaolaza approved these changes Apr 10, 2023

View reviewed changes

raul-arabaolaza reviewed Apr 10, 2023

View reviewed changes

MarkEWaite added a commit to MarkEWaite/docker-lfs that referenced this pull request Apr 10, 2023

Use pipeline groovy lib incremental 652.vc722f7a_cf8c2

a775455

jenkinsci/pipeline-groovy-lib-plugin#55 https://issues.jenkins.io/browse/JENKINS-70869

MarkEWaite approved these changes Apr 10, 2023

View reviewed changes

Vlatombe approved these changes Apr 11, 2023

View reviewed changes

jglick added 2 commits April 11, 2023 15:21

Merge branch 'master' of https://github.com/jenkinsci/pipeline-groovy…

7f1251d

…-lib-plugin into SCMSourceRetriever.clone

Switching some clone-mode messages to fine logger jenkinsci#55 (comment)

6d4bd49

jglick enabled auto-merge April 11, 2023 19:47

jglick merged commit aaceeb6 into jenkinsci:master Apr 11, 2023

jglick deleted the SCMSourceRetriever.clone branch April 11, 2023 21:07

[JENKINS-70869] New option “clone” option to SCM-based retrievers to avoid workspace/xxx@libs/ #55

[JENKINS-70869] New option “clone” option to SCM-based retrievers to avoid workspace/xxx@libs/ #55

Uh oh!

Conversation

jglick commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jglick commented Mar 10, 2023

Uh oh!

raul-arabaolaza Apr 10, 2023

Choose a reason for hiding this comment

Uh oh!

jglick Apr 10, 2023

Choose a reason for hiding this comment

Uh oh!

raul-arabaolaza Apr 10, 2023

Choose a reason for hiding this comment

Uh oh!

jglick Apr 10, 2023

Choose a reason for hiding this comment

Uh oh!

MarkEWaite left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MarkEWaite Apr 10, 2023

Choose a reason for hiding this comment

Uh oh!

jglick Apr 10, 2023

Choose a reason for hiding this comment

Uh oh!

dwnusbaum Apr 10, 2023

Choose a reason for hiding this comment

Uh oh!

Vlatombe left a comment

Choose a reason for hiding this comment

Uh oh!

jglick commented Apr 11, 2023

Uh oh!

MarkEWaite commented Apr 11, 2023

Uh oh!

jglick commented Apr 11, 2023

Uh oh!

ysmaoui commented Feb 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[JENKINS-70869] New option “clone” option to SCM-based retrievers to avoid `workspace/xxx@libs/` #55

[JENKINS-70869] New option “clone” option to SCM-based retrievers to avoid `workspace/xxx@libs/` #55

jglick commented Mar 2, 2023 •

edited

Loading