Skip to content

Commit af2dcd1

Browse files
committed
Merge remote-tracking branch 'jenkinsci/master' into snippetizerlink-cleanup
2 parents 09559d9 + c1fbd97 commit af2dcd1

File tree

91 files changed

+3500
-1003
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+3500
-1003
lines changed

.mvn/extensions.xml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<extensions xmlns="http://maven.apache.org/EXTENSIONS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/EXTENSIONS/1.0.0 http://maven.apache.org/xsd/core-extensions-1.0.0.xsd">
2+
<extension>
3+
<groupId>io.jenkins.tools.incrementals</groupId>
4+
<artifactId>git-changelist-maven-extension</artifactId>
5+
<version>1.0-beta-7</version>
6+
</extension>
7+
</extensions>

.mvn/maven.config

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
-Pconsume-incrementals
2+
-Pmight-produce-incrementals

Jenkinsfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
buildPlugin(jenkinsVersions: [null, '2.73.3'])
1+
buildPlugin(configurations: buildPlugin.recommendedConfigurations())

README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Pipeline Groovy Plugin
1+
# Pipeline: Groovy Plugin
22

33
[Wiki page](https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Groovy+Plugin)
44

@@ -26,8 +26,9 @@ parallel branches
2626
```
2727

2828
gets run as a Groovy program, with certain special function calls called *steps* performing Jenkins-specific operations.
29-
In this example the step `parallel` is defined in this plugin, while `node`, `retry`, `checkout`, and `sh` are defined in other plugins in the Pipeline suite.
30-
The `scm` global variable is defined in the Pipeline Multibranch plugin.
29+
In this example the step `parallel` is defined in this plugin, while `node`, `retry`, `checkout`, and `sh` are defined in other plugins in the Pipeline suite. The `scm` global variable is defined in the Pipeline Multibranch plugin.
30+
31+
The Groovy script is compiled to a class named `WorkflowScript`, so that is the name shown in stack traces instead of the script file-name (e.g. `Jenkinsfile`).
3132

3233
Unlike a regular Groovy program run from a command line, the complete state of a Pipeline build’s program is saved to disk every time an *asynchronous* operation is performed, which includes most Pipeline steps.
3334
Jenkins may be restarted while a build is running, and will resume running the program where it left off.
@@ -44,6 +45,10 @@ Scripts run with the sandbox disabled can make direct calls to Jenkins internal
4445

4546
The [Pipeline Snippet Generator epic](https://issues.jenkins-ci.org/browse/JENKINS-35393) covers issues with the tool used to provide samples of step syntax based on live configuration forms.
4647

48+
## History
49+
50+
This plugin was previously the "Workflow CPS plugin" or "Workflow Groovy Plugin". Accordingly it has the Maven `artifactId` `workflow-cps`, not `pipeline-groovy`.
51+
4752
## Technical design
4853

4954
The plugin uses the [Groovy CPS library](https://github.com/cloudbees/groovy-cps/) to implement a [continuation-passing style transformation](https://en.wikipedia.org/wiki/Continuation-passing_style) on the program as it is compiled.

doc/persistence.md

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# The Pipeline Persistence Model
2+
3+
# Data Model
4+
Running pipelines persist in 3 pieces:
5+
6+
1. The `FlowNode`s - stored by a `FlowNodeStorage` - this holds the FlowNodes created to map to `Step`s, and for block scoped Steps, start and end of blocks
7+
2. The `CpsFlowExecution` - this is currently stored in the WorkflowRun, and the primary pieces of interest are:
8+
* heads - the current "tips" of the Flow Graph, i.e. the FlowNodes that represent running steps and are appended to
9+
- A head maps to a `CpsThread` in the Pipeline program, within the `CpsThreadGroup`
10+
* starts - the `BlockStartNode`s marking the start(s) of the currently executing blocks
11+
* scripts - the loaded Pipeline script files (text)
12+
* persistedClean
13+
- If true, Pipeline saved its execution cleanly to disk and we *might* be able to resume it
14+
- If false, something went wrong saving the execution, so we cannot resume even if we'd otherwise be able to
15+
- If null, probably the build dates back to before this field was added - we check to see if this is running in a highly persistent DurabilityMode (Max_survivability generally)
16+
* done - if true, this execution completed, if false or un-set, the pipeline is a candidate to resume unless its only head is a FlowEndNode
17+
- The handling of false is for legacy reasons, since it was only recently made persistent.
18+
*
19+
* various other boolean flags & settings for the execution (durability setting, user that started the build, is it sandboxed, etc)
20+
3. The Program -- this is the current execution state of the Pipeline
21+
* This holds the Groovy state - the `CpsThreadGroup` - with runtime calls transformed by CPS so they can persist
22+
* The `CpsThread`s map to the running branches of the Pipeline
23+
* The program depends on the FlowNodes from the FlowNodeStorage, since it reads them by ID rather than storing them in the program
24+
* This also depends on the heads in the CpsFlowExecution, because its FlowHeads are loaded from the heads of the CpsFlowExecution
25+
* Also holds the CpsStepContext, i.e. the variables such as EnvVars, Executor and Workspace uses (the latter stored as Pickles)
26+
- The pickles will be specially restored when the Pipeline resumes since they don't serialize/deserialize normally
27+
28+
## Persistence Issues And Logic
29+
30+
Some basic rules:
31+
32+
1. If the FlowNodeStorage is corrupt, incomplete, or un-persisted, all manner of heck will break loose
33+
- In terms of Pipeline execution, the impact is like the Resonance Cascade from the Half-Life games
34+
- The pipeline can never be resumed (the key piece is missing)
35+
- Usually we fake up some placeholder FlowNodes to cover this situation and save them
36+
2. Whenever persisting data, the Pipeline *must* have the FlowNodes persisted on disk (via `storage.flush()` generally)
37+
in order to be able to load the heads and restore the program.
38+
3. Once we've set persistedClean as false and saved the FlowExecution, then it doesn't matter what we do -- the Pipeline will assume
39+
it already has incomplete persistence data (as with 1) when trying to resume. This is how we handle the low-durability modes, to
40+
avoid resuming a stale state of the Pipeline simply because we have old data persisted and are not updating it.

pom.xml

Lines changed: 26 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,12 @@
2828
<parent>
2929
<groupId>org.jenkins-ci.plugins</groupId>
3030
<artifactId>plugin</artifactId>
31-
<version>3.4</version>
31+
<version>3.43</version>
3232
<relativePath />
3333
</parent>
3434
<groupId>org.jenkins-ci.plugins.workflow</groupId>
3535
<artifactId>workflow-cps</artifactId>
36-
<version>2.46-SNAPSHOT</version> <!-- Temp version to help separate this work, remove before merge -->
36+
<version>${revision}${changelist}</version>
3737
<packaging>hpi</packaging>
3838
<name>Pipeline: Groovy</name>
3939
<url>https://wiki.jenkins.io/display/JENKINS/Pipeline+Groovy+Plugin</url>
@@ -47,8 +47,8 @@
4747
<connection>scm:git:git://github.com/jenkinsci/${project.artifactId}-plugin.git</connection>
4848
<developerConnection>scm:git:[email protected]:jenkinsci/${project.artifactId}-plugin.git</developerConnection>
4949
<url>https://github.com/jenkinsci/${project.artifactId}-plugin</url>
50-
<tag>HEAD</tag>
51-
</scm>
50+
<tag>${scmTag}</tag>
51+
</scm>
5252
<repositories>
5353
<repository>
5454
<id>repo.jenkins-ci.org</id>
@@ -62,25 +62,29 @@
6262
</pluginRepository>
6363
</pluginRepositories>
6464
<properties>
65-
<jenkins.version>2.62</jenkins.version>
65+
<revision>2.71</revision>
66+
<changelist>-SNAPSHOT</changelist>
67+
<jenkins.version>2.121.1</jenkins.version>
6668
<java.level>8</java.level>
6769
<no-test-jar>false</no-test-jar>
70+
<useBeta>true</useBeta>
6871
<git-plugin.version>3.1.0</git-plugin.version>
69-
<workflow-support-plugin.version>2.17</workflow-support-plugin.version>
70-
<scm-api-plugin.version>2.0.8</scm-api-plugin.version>
71-
<groovy-cps.version>1.24</groovy-cps.version>
72-
<jenkins-test-harness.version>2.33</jenkins-test-harness.version>
72+
<workflow-support-plugin.version>3.3</workflow-support-plugin.version>
73+
<scm-api-plugin.version>2.2.6</scm-api-plugin.version>
74+
<groovy-cps.version>1.28</groovy-cps.version>
75+
<structs-plugin.version>1.18</structs-plugin.version>
76+
<workflow-step-api-plugin.version>2.20</workflow-step-api-plugin.version>
7377
</properties>
7478
<dependencies>
7579
<dependency>
7680
<groupId>org.jenkins-ci.plugins.workflow</groupId>
7781
<artifactId>workflow-step-api</artifactId>
78-
<version>2.13</version>
82+
<version>${workflow-step-api-plugin.version}</version>
7983
</dependency>
8084
<dependency>
8185
<groupId>org.jenkins-ci.plugins.workflow</groupId>
8286
<artifactId>workflow-api</artifactId>
83-
<version>2.25</version>
87+
<version>2.30</version>
8488
</dependency>
8589
<dependency>
8690
<groupId>org.jenkins-ci.plugins.workflow</groupId>
@@ -95,7 +99,7 @@
9599
<dependency>
96100
<groupId>org.jenkins-ci.plugins</groupId>
97101
<artifactId>script-security</artifactId>
98-
<version>1.42</version>
102+
<version>1.58</version>
99103
</dependency>
100104
<dependency>
101105
<groupId>org.jenkins-ci.plugins</groupId>
@@ -105,12 +109,12 @@
105109
<dependency>
106110
<groupId>org.jenkins-ci.plugins</groupId>
107111
<artifactId>structs</artifactId>
108-
<version>1.14</version>
112+
<version>${structs-plugin.version}</version>
109113
</dependency>
110114
<dependency>
111115
<groupId>org.jenkins-ci.plugins</groupId>
112116
<artifactId>support-core</artifactId>
113-
<version>2.32</version>
117+
<version>2.43</version>
114118
<optional>true</optional>
115119
<exclusions>
116120
<exclusion>
@@ -119,6 +123,13 @@
119123
</exclusion>
120124
</exclusions>
121125
</dependency>
126+
<dependency>
127+
<groupId>org.jenkins-ci.plugins.workflow</groupId>
128+
<artifactId>workflow-step-api</artifactId>
129+
<version>${workflow-step-api-plugin.version}</version>
130+
<classifier>tests</classifier>
131+
<scope>test</scope>
132+
</dependency>
122133
<dependency>
123134
<groupId>org.jenkins-ci.plugins.workflow</groupId>
124135
<artifactId>workflow-support</artifactId>
@@ -141,7 +152,7 @@
141152
<dependency>
142153
<groupId>org.jenkins-ci.plugins.workflow</groupId>
143154
<artifactId>workflow-job</artifactId>
144-
<version>2.17</version>
155+
<version>2.26</version>
145156
<scope>test</scope>
146157
</dependency>
147158
<dependency>
@@ -168,12 +179,6 @@
168179
<version>2.5</version>
169180
<scope>test</scope>
170181
</dependency>
171-
<dependency>
172-
<groupId>org.jenkins-ci.plugins</groupId>
173-
<artifactId>matrix-auth</artifactId>
174-
<version>1.0.2</version>
175-
<scope>test</scope>
176-
</dependency>
177182
<dependency>
178183
<groupId>org.jenkins-ci.plugins</groupId>
179184
<artifactId>junit</artifactId>
@@ -248,12 +253,6 @@
248253
</exclusion>
249254
</exclusions>
250255
</dependency>
251-
<dependency>
252-
<groupId>org.jenkins-ci.plugins</groupId>
253-
<artifactId>credentials</artifactId>
254-
<version>2.1.8</version>
255-
<scope>test</scope>
256-
</dependency>
257256
<dependency>
258257
<groupId>org.jenkins-ci.plugins</groupId>
259258
<artifactId>config-file-provider</artifactId>
@@ -290,15 +289,6 @@
290289
</resource>
291290
</resources>
292291
<plugins>
293-
<plugin>
294-
<!-- why is this necessary!? otherwise compiling src/test/groovy/ fails -->
295-
<!-- fix attempted at https://github.com/jenkinsci/jenkins/commit/101507f49873de0239ccb7839649ea71187712b2 but apparently failed. -->
296-
<groupId>org.codehaus.gmaven</groupId>
297-
<artifactId>gmaven-plugin</artifactId>
298-
<configuration>
299-
<providerSelection>1.8</providerSelection>
300-
</configuration>
301-
</plugin>
302292
<plugin>
303293
<groupId>org.jenkins-ci.tools</groupId>
304294
<artifactId>maven-hpi-plugin</artifactId>

src/main/java/org/jenkinsci/plugins/workflow/cps/ContextVariableSet.java

Lines changed: 114 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,36 +24,136 @@
2424

2525
package org.jenkinsci.plugins.workflow.cps;
2626

27-
import org.jenkinsci.plugins.workflow.cps.persistence.PersistIn;
28-
29-
import javax.annotation.concurrent.Immutable;
27+
import com.google.common.util.concurrent.ListenableFuture;
28+
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
29+
import hudson.ExtensionList;
30+
import hudson.model.Result;
31+
import java.io.IOException;
3032
import java.io.Serializable;
3133
import java.util.ArrayList;
34+
import java.util.HashSet;
3235
import java.util.List;
33-
34-
import static org.jenkinsci.plugins.workflow.cps.persistence.PersistenceContext.NONE;
36+
import java.util.Set;
37+
import java.util.logging.Logger;
38+
import java.util.stream.Collectors;
39+
import javax.annotation.concurrent.Immutable;
40+
import org.jenkinsci.plugins.workflow.cps.persistence.PersistIn;
41+
import static org.jenkinsci.plugins.workflow.cps.persistence.PersistenceContext.PROGRAM;
42+
import org.jenkinsci.plugins.workflow.flow.FlowExecution;
43+
import org.jenkinsci.plugins.workflow.graph.FlowNode;
44+
import org.jenkinsci.plugins.workflow.steps.BodyInvoker;
45+
import org.jenkinsci.plugins.workflow.steps.DynamicContext;
46+
import org.jenkinsci.plugins.workflow.support.DefaultStepContext;
3547

3648
/**
37-
* @author Kohsuke Kawaguchi
49+
* Holds a set of contextual objects as per {@link BodyInvoker#withContext} in a given scope.
50+
* Also interprets {@link DynamicContext}.
3851
*/
3952
@Immutable
40-
@PersistIn(NONE)
53+
@PersistIn(PROGRAM)
4154
final class ContextVariableSet implements Serializable {
55+
56+
private static final Logger LOGGER = Logger.getLogger(ContextVariableSet.class.getName());
57+
4258
private final ContextVariableSet parent;
43-
private final List<Object> values = new ArrayList<Object>();
59+
private final List<Object> values = new ArrayList<>();
4460

4561
ContextVariableSet(ContextVariableSet parent) {
4662
this.parent = parent;
4763
}
4864

49-
<T> T get(Class<T> type) {
50-
for (ContextVariableSet s=this; s!=null; s=s.parent) {
51-
for (Object v : s.values) {
52-
if (type.isInstance(v))
53-
return type.cast(v);
65+
private static final ThreadLocal<Set<DynamicContextQuery>> dynamicContextClasses = ThreadLocal.withInitial(HashSet::new);
66+
67+
private static final class DynamicContextQuery {
68+
final DynamicContext dynamicContext;
69+
final Class<?> key;
70+
DynamicContextQuery(DynamicContext dynamicContext, Class<?> key) {
71+
this.dynamicContext = dynamicContext;
72+
this.key = key;
73+
}
74+
@Override public boolean equals(Object obj) {
75+
return obj instanceof DynamicContextQuery &&
76+
dynamicContext == ((DynamicContextQuery) obj).dynamicContext &&
77+
key == ((DynamicContextQuery) obj).key;
78+
}
79+
@Override public int hashCode() {
80+
return dynamicContext.hashCode() ^ key.hashCode();
81+
}
82+
}
83+
84+
interface ThrowingSupplier<T> {
85+
T get() throws IOException;
86+
}
87+
88+
<T> T get(Class<T> key, ThrowingSupplier<FlowExecution> execution, ThrowingSupplier<FlowNode> node) throws IOException, InterruptedException {
89+
for (Object v : values) {
90+
if (key.isInstance(v)) {
91+
LOGGER.fine(() -> "found a " + v.getClass().getName() + " in " + this);
92+
return key.cast(v);
5493
}
5594
}
56-
return null;
95+
class Delegate extends DefaultStepContext implements DynamicContext.DelegatedContext {
96+
@Override protected <T> T doGet(Class<T> key) throws IOException, InterruptedException {
97+
return ContextVariableSet.this.get(key, execution, node);
98+
}
99+
@Override protected FlowExecution getExecution() throws IOException {
100+
return execution.get();
101+
}
102+
@Override protected FlowNode getNode() throws IOException {
103+
return node.get();
104+
}
105+
@Override public void onSuccess(Object result) {
106+
throw new AssertionError();
107+
}
108+
@Override public boolean isReady() {
109+
throw new AssertionError();
110+
}
111+
@Override public ListenableFuture<Void> saveState() {
112+
throw new AssertionError();
113+
}
114+
@Override public void setResult(Result r) {
115+
throw new AssertionError();
116+
}
117+
@Override public BodyInvoker newBodyInvoker() throws IllegalStateException {
118+
throw new AssertionError();
119+
}
120+
@SuppressFBWarnings(value = "EQ_UNUSUAL", justification = "DefaultStepContext does not delegate to this")
121+
@SuppressWarnings("EqualsWhichDoesntCheckParameterClass")
122+
@Override public boolean equals(Object o) {
123+
throw new AssertionError();
124+
}
125+
@Override public int hashCode() {
126+
throw new AssertionError();
127+
}
128+
@Override public void onFailure(Throwable t) {
129+
throw new AssertionError();
130+
}
131+
}
132+
DynamicContext.DelegatedContext delegate = new Delegate();
133+
for (DynamicContext dynamicContext : ExtensionList.lookup(DynamicContext.class)) {
134+
DynamicContextQuery query = new DynamicContextQuery(dynamicContext, key);
135+
Set<DynamicContextQuery> dynamicStack = dynamicContextClasses.get();
136+
if (dynamicStack.add(query)) { // thus, being newly added to the stack
137+
try {
138+
T v = dynamicContext.get(key, delegate);
139+
if (v != null) {
140+
LOGGER.fine(() -> "looked up a " + v.getClass().getName() + " from " + dynamicContext + " in " + this);
141+
return v;
142+
}
143+
} finally {
144+
dynamicStack.remove(query);
145+
}
146+
}
147+
}
148+
if (parent != null) {
149+
return parent.get(key, execution, node);
150+
} else {
151+
return null;
152+
}
153+
}
154+
155+
@Override public String toString() {
156+
return "ContextVariableSet" + values.stream().map(Object::getClass).map(Class::getName).collect(Collectors.toList()) + (parent != null ? "<" + parent : "");
57157
}
58158

59159
/**

0 commit comments

Comments
 (0)