feat: Add pipeline run parallelism config #12442
Draft
+2,471
−110
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
google-oss-prow
bot
requested review from
HumairAK,
droctothorpe,
mprahl and
zazulam
|
Hi @sduvvuri1603. Thanks for your PR. I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
sduvvuri1603
force-pushed
the
feature/pipeline-run-parallelism
branch
2 times, most recently
from
99f2fc8 to
d34a1b2
Compare
Contributor
|
/retest |
sduvvuri1603
force-pushed
the
feature/pipeline-run-parallelism
branch
7 times, most recently
from
82756e1 to
60a35d8
Compare
|
@sduvvuri1603: Cannot trigger testing until a trusted user reviews the PR and leaves an DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Contributor
|
/ok-to-test |
Contributor
|
/retest |
google-oss-prow
bot
requested review from
DharmitD,
alyssacgoins and
gmfrasca
Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com>
Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com>
Re-add the argo workflow package alias so the new semaphore logic compiles; merge dropped the import which broke go build. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Regenerated the workflow compiler fixtures after the merge so they now include the fully qualified ml-pipeline address and updated dependency ordering. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the branch-specific skip so workflows build and load the envoy image just like master. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve the latest pipeline version when a recurring run is created with only a pipeline ID, and copy that version’s ID/name into the job so the API surfaces a concrete pipelineVersionId. Update the unit test to verify the stored job now references the resolved version. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Remove temporary CI debug paths for missing images, TLS readiness logging, forced MinIO usage, and TLS overlay ordering so behavior matches upstream. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Lower sleep_seconds in the max_active_runs test pipeline to avoid repeated timeouts while still exercising parallelism. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Update the compiled workflow golden to match the reduced sleep_seconds default in the max_active_runs pipeline. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
sduvvuri1603
force-pushed
the
feature/pipeline-run-parallelism
branch
from
66b1ef5 to
ebaa256
Compare
Shorten max-active-runs sleep defaults and regenerate compiler goldens to keep workflow expectations aligned with the updated pipeline spec. Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
/retest |
sduvvuri1603
force-pushed
the
feature/pipeline-run-parallelism
branch
4 times, most recently
from
5a3efbf to
f434f6c
Compare
Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…run-parallelism Signed-off-by: sduvvuri1603 <sduvvuri@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
sduvvuri1603
force-pushed
the
feature/pipeline-run-parallelism
branch
from
f434f6c to
f253b4e
Compare
Contributor
Author
|
/retest |
Signed-off-by: Sruthi Duvvuri <sduvvuri@redhat.com>
Contributor
Author
|
/retest |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Validation
e2e_utils.go+ invocation inpipeline_e2e_test.go), rebuilt the test cluster with the fresh backend images, exercised the focused pipeline_run_parallelism scenario, and then ran the end-to-end suite to confirm the new check passes with the concurrency cap enforced.Note:
Hardened .github/actions/test-and-report/action.yml: the workflow now waits for the ml-pipeline deployment, port-forwards svc/ml-pipeline to localhost:8888, and curls /apis/v2beta1/healthz (TLS-aware). This eliminates the dial tcp [::1]:8888: connect: connection refused failures that intermittently broke the parallelism/recurring-run E2E suite.
Follow up to PR - remove unused semaphore_key and mutex_name fields