Skip to content

Commit 035e037

Browse files
MiguelAAeMiguelAAe
authored
feat(e2e-tests): pipeline tests (#7010)
* fix model assertions * refactor watcher store * load custom pipeline steps * load custom pipeline steps * pipeline chaining * watcher store can query by name and del previous k8s resources when running a new test suite * fix: wait until all resources are deleted * pipeline examples * small refactor of func names and comments * copyright * no need for this func * addressed comments: * format issues * watcher store now saves keys with namespace/kind/name * context with timeout when deleting resources * update k8s deps * updated map name of labels in server_steps.go
1 parent 6b822f7 commit 035e037

20 files changed

+731
-111
lines changed

‎tests/integration/godog/features/pipeline/chain.feature‎

Whitespace-only changes.

‎tests/integration/godog/features/pipeline/conditional.feature‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
@PipelineDeployment @Functional @Pipelines @Conditional
2+
Feature: Conditional pipeline with branching models
3+
This pipeline uses a conditional model to route data to either add10 or mul10.
4+
5+
Scenario: Deploy tfsimple-conditional pipeline and wait for readiness
6+
Given I deploy model spec with timeout "30s":
7+
"""
8+
apiVersion: mlops.seldon.io/v1alpha1
9+
kind: Model
10+
metadata:
11+
name: conditional-nbsl
12+
spec:
13+
storageUri: "gs://seldon-models/scv2/samples/triton_23-03/conditional"
14+
requirements:
15+
- triton
16+
- python
17+
"""
18+
And I deploy model spec with timeout "30s":
19+
"""
20+
apiVersion: mlops.seldon.io/v1alpha1
21+
kind: Model
22+
metadata:
23+
name: add10-nbsl
24+
spec:
25+
storageUri: "gs://seldon-models/scv2/samples/triton_23-03/add10"
26+
requirements:
27+
- triton
28+
- python
29+
"""
30+
And I deploy model spec with timeout "30s":
31+
"""
32+
apiVersion: mlops.seldon.io/v1alpha1
33+
kind: Model
34+
metadata:
35+
name: mul10-nbsl
36+
spec:
37+
storageUri: "gs://seldon-models/scv2/samples/triton_23-03/mul10"
38+
requirements:
39+
- triton
40+
- python
41+
"""
42+
Then the model "conditional-nbsl" should eventually become Ready with timeout "20s"
43+
And the model "add10-nbsl" should eventually become Ready with timeout "20s"
44+
And the model "mul10-nbsl" should eventually become Ready with timeout "20s"
45+
46+
And I deploy pipeline spec with timeout "30s":
47+
"""
48+
apiVersion: mlops.seldon.io/v1alpha1
49+
kind: Pipeline
50+
metadata:
51+
name: tfsimple-conditional-nbsl
52+
spec:
53+
steps:
54+
- name: conditional-nbsl
55+
- name: mul10-nbsl
56+
inputs:
57+
- conditional-nbsl.outputs.OUTPUT0
58+
tensorMap:
59+
conditional-nbsl.outputs.OUTPUT0: INPUT
60+
- name: add10-nbsl
61+
inputs:
62+
- conditional-nbsl.outputs.OUTPUT1
63+
tensorMap:
64+
conditional-nbsl.outputs.OUTPUT1: INPUT
65+
output:
66+
steps:
67+
- mul10-nbsl
68+
- add10-nbsl
69+
stepsJoin: any
70+
"""
71+
Then the pipeline "tfsimple-conditional-nbsl" should eventually become Ready with timeout "40s"

‎tests/integration/godog/features/pipeline/deployment.feature‎

Whitespace-only changes.

‎tests/integration/godog/features/pipeline/input_chaining.feature‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
@PipelineDeployment @Functional @Pipelines @ModelChainingFromInputs
2+
Feature: Pipeline model chaining using inputs and outputs
3+
This pipeline chains tfsimple1 into tfsimple2 using both inputs and outputs.
4+
5+
Scenario: Deploy tfsimples-input pipeline and wait for readiness
6+
Given I deploy model spec with timeout "30s":
7+
"""
8+
apiVersion: mlops.seldon.io/v1alpha1
9+
kind: Model
10+
metadata:
11+
name: chain-from-input-tfsimple1-yhjo
12+
spec:
13+
storageUri: "gs://seldon-models/triton/simple"
14+
requirements:
15+
- tensorflow
16+
memory: 100Ki
17+
18+
"""
19+
And I deploy model spec with timeout "20s":
20+
"""
21+
apiVersion: mlops.seldon.io/v1alpha1
22+
kind: Model
23+
metadata:
24+
name: chain-from-input-tfsimple2-yhjo
25+
spec:
26+
storageUri: "gs://seldon-models/triton/simple"
27+
requirements:
28+
- tensorflow
29+
memory: 100Ki
30+
"""
31+
Then the model "chain-from-input-tfsimple1-yhjo" should eventually become Ready with timeout "20s"
32+
Then the model "chain-from-input-tfsimple2-yhjo" should eventually become Ready with timeout "20s"
33+
34+
And I deploy pipeline spec with timeout "30s":
35+
"""
36+
apiVersion: mlops.seldon.io/v1alpha1
37+
kind: Pipeline
38+
metadata:
39+
name: chain-from-input-tfsimples-input-yhjo
40+
spec:
41+
steps:
42+
- name: chain-from-input-tfsimple1-yhjo
43+
- name: chain-from-input-tfsimple2-yhjo
44+
inputs:
45+
- chain-from-input-tfsimple1-yhjo.inputs.INPUT0
46+
- chain-from-input-tfsimple1-yhjo.outputs.OUTPUT1
47+
tensorMap:
48+
chain-from-input-tfsimple1-yhjo.outputs.OUTPUT1: INPUT1
49+
output:
50+
steps:
51+
- chain-from-input-tfsimple2-yhjo
52+
"""
53+
Then the pipeline "chain-from-input-tfsimples-input-yhjo" should eventually become Ready with timeout "40s"

‎tests/integration/godog/features/pipeline/input_tensors.feature‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
@PipelineDeployment @Functional @Pipelines @PipelineInputTensors
2+
Feature: Pipeline using direct input tensors
3+
This pipeline directly routes pipeline input tensors INPUT0 and INPUT1 into separate models.
4+
5+
Scenario: Deploy pipeline-inputs pipeline and wait for readiness
6+
Given I deploy model spec with timeout "30s":
7+
"""
8+
apiVersion: mlops.seldon.io/v1alpha1
9+
kind: Model
10+
metadata:
11+
name: mul10-tw2x
12+
spec:
13+
storageUri: "gs://seldon-models/scv2/samples/triton_23-03/mul10"
14+
requirements:
15+
- triton
16+
- python
17+
"""
18+
And I deploy model spec with timeout "30s":
19+
"""
20+
apiVersion: mlops.seldon.io/v1alpha1
21+
kind: Model
22+
metadata:
23+
name: add10-tw2x
24+
spec:
25+
storageUri: "gs://seldon-models/scv2/samples/triton_23-03/add10"
26+
requirements:
27+
- triton
28+
- python
29+
"""
30+
Then the model "mul10-tw2x" should eventually become Ready with timeout "20s"
31+
And the model "add10-tw2x" should eventually become Ready with timeout "20s"
32+
33+
And I deploy pipeline spec with timeout "30s":
34+
"""
35+
apiVersion: mlops.seldon.io/v1alpha1
36+
kind: Pipeline
37+
metadata:
38+
name: pipeline-inputs-tw2x
39+
spec:
40+
steps:
41+
- name: mul10-tw2x
42+
inputs:
43+
- pipeline-inputs-tw2x.inputs.INPUT0
44+
tensorMap:
45+
pipeline-inputs-tw2x.inputs.INPUT0: INPUT
46+
- name: add10-tw2x
47+
inputs:
48+
- pipeline-inputs-tw2x.inputs.INPUT1
49+
tensorMap:
50+
pipeline-inputs-tw2x.inputs.INPUT1: INPUT
51+
output:
52+
steps:
53+
- mul10-tw2x
54+
- add10-tw2x
55+
"""
56+
Then the pipeline "pipeline-inputs-tw2x" should eventually become Ready with timeout "20s"

‎tests/integration/godog/features/pipeline/join.feature‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
@PipelineDeployment @Functional @Pipelines @ModelJoin
2+
Feature: Pipeline model join
3+
This pipeline joins outputs from tfsimple1 and tfsimple2 and feeds them into tfsimple3.
4+
5+
Scenario: Deploy tfsimples-join pipeline and wait for readiness
6+
Given I deploy model spec with timeout "30s":
7+
"""
8+
apiVersion: mlops.seldon.io/v1alpha1
9+
kind: Model
10+
metadata:
11+
name: join-tfsimple1-w4e3
12+
spec:
13+
storageUri: "gs://seldon-models/triton/simple"
14+
requirements:
15+
- tensorflow
16+
memory: 100Ki
17+
"""
18+
And I deploy model spec with timeout "30s":
19+
"""
20+
apiVersion: mlops.seldon.io/v1alpha1
21+
kind: Model
22+
metadata:
23+
name: join-tfsimple2-w4e3
24+
spec:
25+
storageUri: "gs://seldon-models/triton/simple"
26+
requirements:
27+
- tensorflow
28+
memory: 100Ki
29+
"""
30+
And I deploy model spec with timeout "30s":
31+
"""
32+
apiVersion: mlops.seldon.io/v1alpha1
33+
kind: Model
34+
metadata:
35+
name: join-tfsimple3-w4e3
36+
spec:
37+
storageUri: "gs://seldon-models/triton/simple"
38+
requirements:
39+
- tensorflow
40+
memory: 100Ki
41+
"""
42+
Then the model "join-tfsimple1-w4e3" should eventually become Ready with timeout "20s"
43+
And the model "join-tfsimple2-w4e3" should eventually become Ready with timeout "20s"
44+
And the model "join-tfsimple3-w4e3" should eventually become Ready with timeout "20s"
45+
46+
And I deploy pipeline spec with timeout "30s":
47+
"""
48+
apiVersion: mlops.seldon.io/v1alpha1
49+
kind: Pipeline
50+
metadata:
51+
name: join-pipeline-w4e3
52+
spec:
53+
steps:
54+
- name: join-tfsimple1-w4e3
55+
- name: join-tfsimple2-w4e3
56+
- name: join-tfsimple3-w4e3
57+
inputs:
58+
- join-tfsimple1-w4e3.outputs.OUTPUT0
59+
- join-tfsimple2-w4e3.outputs.OUTPUT1
60+
tensorMap:
61+
join-tfsimple1-w4e3.outputs.OUTPUT0: INPUT0
62+
join-tfsimple2-w4e3.outputs.OUTPUT1: INPUT1
63+
output:
64+
steps:
65+
- join-tfsimple3-w4e3
66+
"""
67+
Then the pipeline "join-pipeline-w4e3" should eventually become Ready with timeout "40s"

‎tests/integration/godog/features/pipeline/model_chaining.feature‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
@ModelChaining @Functional @Pipelines
2+
Feature: Pipeline model chaining
3+
This pipeline chains tfsimple1 into tfsimple2 using tensorMap.
4+
5+
Scenario: Deploy tfsimples pipeline and wait for readiness
6+
Given I deploy model spec with timeout "30s":
7+
"""
8+
apiVersion: mlops.seldon.io/v1alpha1
9+
kind: Model
10+
metadata:
11+
name: model-chain-tfsimple1-iuw3
12+
spec:
13+
storageUri: "gs://seldon-models/triton/simple"
14+
requirements:
15+
- tensorflow
16+
memory: 100Ki
17+
18+
"""
19+
And I deploy model spec with timeout "20s":
20+
"""
21+
apiVersion: mlops.seldon.io/v1alpha1
22+
kind: Model
23+
metadata:
24+
name: model-chain-tfsimple2-iuw3
25+
spec:
26+
storageUri: "gs://seldon-models/triton/simple"
27+
requirements:
28+
- tensorflow
29+
memory: 100Ki
30+
"""
31+
Then the model "model-chain-tfsimple1-iuw3" should eventually become Ready with timeout "20s"
32+
Then the model "model-chain-tfsimple2-iuw3" should eventually become Ready with timeout "20s"
33+
When I deploy pipeline spec with timeout "20s":
34+
"""
35+
apiVersion: mlops.seldon.io/v1alpha1
36+
kind: Pipeline
37+
metadata:
38+
name: model-chain-tfsimples-iuw3
39+
spec:
40+
steps:
41+
- name: model-chain-tfsimple1-iuw3
42+
- name: model-chain-tfsimple2-iuw3
43+
inputs:
44+
- model-chain-tfsimple1-iuw3
45+
tensorMap:
46+
model-chain-tfsimple1-iuw3.outputs.OUTPUT0: INPUT0
47+
model-chain-tfsimple1-iuw3.outputs.OUTPUT1: INPUT1
48+
output:
49+
steps:
50+
- model-chain-tfsimple2-iuw3
51+
"""
52+
Then the pipeline "model-chain-tfsimples-iuw3" should eventually become Ready with timeout "40s"

‎tests/integration/godog/features/pipeline/trigger_joins.feature‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
@PipelineDeployment @Functional @Pipelines @TriggerJoins
2+
Feature: Pipeline using trigger joins
3+
This pipeline uses trigger joins to decide whether mul10 or add10 should run.
4+
5+
Scenario: Deploy trigger-joins pipeline and wait for readiness
6+
Given I deploy model spec with timeout "30s":
7+
"""
8+
apiVersion: mlops.seldon.io/v1alpha1
9+
kind: Model
10+
metadata:
11+
name: mul10-99lo
12+
spec:
13+
storageUri: "gs://seldon-models/scv2/samples/triton_23-03/mul10"
14+
requirements:
15+
- triton
16+
- python
17+
"""
18+
And I deploy model spec with timeout "30s":
19+
"""
20+
apiVersion: mlops.seldon.io/v1alpha1
21+
kind: Model
22+
metadata:
23+
name: add10-99lo
24+
spec:
25+
storageUri: "gs://seldon-models/scv2/samples/triton_23-03/add10"
26+
requirements:
27+
- triton
28+
- python
29+
"""
30+
Then the model "mul10-99lo" should eventually become Ready with timeout "20s"
31+
And the model "add10-99lo" should eventually become Ready with timeout "20s"
32+
33+
And I deploy pipeline spec with timeout "30s":
34+
"""
35+
apiVersion: mlops.seldon.io/v1alpha1
36+
kind: Pipeline
37+
metadata:
38+
name: trigger-joins-99lo
39+
spec:
40+
steps:
41+
- name: mul10-99lo
42+
inputs:
43+
- trigger-joins-99lo.inputs.INPUT
44+
triggers:
45+
- trigger-joins-99lo.inputs.ok1
46+
- trigger-joins-99lo.inputs.ok2
47+
triggersJoinType: any
48+
- name: add10-99lo
49+
inputs:
50+
- trigger-joins-99lo.inputs.INPUT
51+
triggers:
52+
- trigger-joins-99lo.inputs.ok3
53+
output:
54+
steps:
55+
- mul10-99lo
56+
- add10-99lo
57+
stepsJoin: any
58+
"""
59+
Then the pipeline "trigger-joins-99lo" should eventually become Ready with timeout "20s"

‎tests/integration/godog/go.mod‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,11 @@ require (
99
github.com/sirupsen/logrus v1.9.3
1010
github.com/spf13/pflag v1.0.7
1111
google.golang.org/grpc v1.73.0
12-
gopkg.in/yaml.v3 v3.0.1
13-
k8s.io/api v0.34.2
14-
k8s.io/apimachinery v0.34.2
15-
k8s.io/client-go v0.34.2
12+
k8s.io/apiextensions-apiserver v0.34.3
13+
k8s.io/apimachinery v0.34.3
14+
k8s.io/client-go v0.34.3
1615
sigs.k8s.io/controller-runtime v0.22.4
16+
sigs.k8s.io/yaml v1.6.0
1717
)
1818

1919
replace (
@@ -71,13 +71,13 @@ require (
7171
google.golang.org/protobuf v1.36.6 // indirect
7272
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
7373
gopkg.in/inf.v0 v0.9.1 // indirect
74-
k8s.io/apiextensions-apiserver v0.34.1 // indirect
74+
gopkg.in/yaml.v3 v3.0.1 // indirect
75+
k8s.io/api v0.34.3 // indirect
7576
k8s.io/klog/v2 v2.130.1 // indirect
7677
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
7778
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
7879
knative.dev/pkg v0.0.0-20250128013458-efddeac3ec35 // indirect
7980
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
8081
sigs.k8s.io/randfill v1.0.0 // indirect
8182
sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
82-
sigs.k8s.io/yaml v1.6.0 // indirect
8383
)

0 commit comments

Comments
 (0)