Do not fail PipelineRun if pvc creation error is because of exceeded quotas #8903

vdemeester · 2025-07-22T10:47:42Z

Changes

In case of the PVC creation (from volumeclaimtemplate) is due to a
quota error (quota exceeded), do not fail with a permanent error, and
instead mark the PipelineRun as pending. Once there is some quota
available back, it will be able to start.

Signed-off-by: Vincent Demeester [email protected]

Closes #7672

/kind feature

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
Has Tests included if any functionality added or changed
pre-commit Passed
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

PipelineRun do not fail anymore if the pvc creation is due to an exceeded quota ; it will be requeued instead (until quota is available or it times out)

vdemeester · 2025-07-22T10:48:14Z

/wip
I need to update/create some e2e tests and makes sure it times out.

tekton-robot · 2025-07-22T11:02:54Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	91.6%	91.2%	-0.4
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	77.4%	-6.3

tekton-robot · 2025-07-22T11:04:34Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	91.6%	91.2%	-0.4
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	77.4%	-6.3

vdemeester · 2025-07-23T09:13:30Z

/hold

tekton-robot · 2025-07-23T09:35:22Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	91.6%	91.2%	-0.4
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	77.4%	-6.3

vdemeester · 2025-07-23T10:54:24Z

/retest

vdemeester · 2025-07-23T11:57:40Z

Timeout is taken into account 👼🏼

twoGiants · 2025-07-24T15:44:33Z

/assign

twoGiants · 2025-07-24T15:44:56Z

/cc @twoGiants

twoGiants

Good job, it makes sense to set the status to pending and let it retry 😸 👍.

Now it will re-queue right away. Do you want to make it configurable at some point like suggested in the issue?

My comments are below. I would add unit tests, simplify the conditional logic and remove the re-declaration of the errors in affinity_assistant.go.

pkg/reconciler/volumeclaim/pvchandler.go

twoGiants · 2025-07-24T16:20:13Z

pkg/reconciler/volumeclaim/pvchandler.go

Unit tests for the new conditional logic in CreatePVCFromVolumeClaimTemplate should be added.

pkg/reconciler/pipelinerun/affinity_assistant.go

twoGiants · 2025-07-24T16:25:20Z

pkg/reconciler/pipelinerun/affinity_assistant_test.go

@@ -577,54 +577,56 @@ func TestCreateOrUpdateAffinityAssistantsAndPVCs_Failure(t *testing.T) {
 		name:        "pvc creation failed - per workspace",
 		failureType: "pvc",
 		aaBehavior:  aa.AffinityAssistantPerWorkspace,
-		expectedErr: fmt.Errorf("%w: failed to create PVC pvc-b9eea16dce: error creating persistentvolumeclaims", ErrPvcCreationFailed),
+		expectedErr: fmt.Errorf("%w for pvc-b9eea16dce: error creating persistentvolumeclaims", ErrPvcCreationFailed),


You could use volumeclaim.ErrPvcCreationFailed instead, then you don't need to re-declare them in affinity_assistant.go.

pkg/reconciler/pipelinerun/pipelinerun.go

pkg/reconciler/volumeclaim/pvchandler.go

pkg/reconciler/pipelinerun/pipelinerun.go

…quotas In case of the PVC creation (from volumeclaimtemplate) is due to a quota error (quota exceeded), do not fail with a permanent error, and instead mark the PipelineRun as pending. Once there is some quota available back, it will be able to start. Signed-off-by: Vincent Demeester <[email protected]>

tekton-robot · 2025-08-19T19:55:45Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	92.1%	91.6%	-0.5
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	75.9%	-7.7

tekton-robot · 2025-08-19T19:58:35Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	92.1%	91.6%	-0.5
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	75.9%	-7.7

pkg/reconciler/volumeclaim/pvchandler.go

waveywaves · 2025-08-20T06:34:20Z

/retest

tekton-robot · 2025-08-20T08:14:36Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	92.1%	91.6%	-0.5
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	75.9%	-7.7

tekton-robot · 2025-08-20T08:25:44Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	92.1%	91.6%	-0.5
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	88.9%	5.2

tekton-robot · 2025-08-20T08:53:48Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	92.1%	91.6%	-0.5

Co-authored-by: Stanislav Jakuschevskij <[email protected]>

tekton-robot · 2025-08-20T10:15:13Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	92.1%	91.6%	-0.5
pkg/reconciler/volumeclaim/pvchandler.go	83.7%	88.9%	5.2

aThorp96

Looks good. Thanks!

tekton-robot · 2025-08-20T14:56:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aThorp96, waveywaves

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [waveywaves]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

khrm

/lgtm

tekton-robot · 2025-08-20T14:59:16Z

@khrm: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

aThorp96 · 2025-08-20T19:28:28Z

I believe the isConflict() addition here is necessary because of this kubernetes bug: kubernetes/kubernetes#67761

mbpavan · 2025-08-29T08:30:18Z

/cherry-pick release-v1.3.x

waveywaves · 2025-08-29T20:43:55Z

/cherry-pick release-v1.3.x

github-project-automation bot added this to Tekton Community Roadmap Jul 22, 2025

tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jul 22, 2025

github-project-automation bot moved this to Todo in Tekton Community Roadmap Jul 22, 2025

tekton-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 22, 2025

tekton-robot requested review from dibyom and pritidesai July 22, 2025 10:47

tekton-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jul 22, 2025

vdemeester added this to the v1.3.0 (LTS) milestone Jul 22, 2025

vdemeester force-pushed the 7672-retry-pvc-quota branch from 4a5edbf to 52464a8 Compare July 22, 2025 10:53

tekton-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2025

vdemeester force-pushed the 7672-retry-pvc-quota branch from 52464a8 to 6b0c7ae Compare July 23, 2025 09:22

tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 23, 2025

tekton-robot assigned twoGiants Jul 24, 2025

tekton-robot requested a review from twoGiants July 24, 2025 15:44

twoGiants suggested changes Jul 24, 2025

View reviewed changes

vdemeester modified the milestones: v1.3.0 (LTS), v1.4.0 Jul 30, 2025

aThorp96 reviewed Aug 19, 2025

View reviewed changes

pkg/reconciler/volumeclaim/pvchandler.go Outdated Show resolved Hide resolved

waveywaves requested changes Aug 19, 2025

View reviewed changes

khrm reviewed Aug 20, 2025

View reviewed changes

pkg/reconciler/volumeclaim/pvchandler.go Outdated Show resolved Hide resolved

waveywaves mentioned this pull request Aug 20, 2025

feat: add backoff retries and timeout specific for PVC creation and #8981

Open

vdemeester force-pushed the 7672-retry-pvc-quota branch 2 times, most recently from 69cc896 to 0758e65 Compare August 20, 2025 08:13

vdemeester force-pushed the 7672-retry-pvc-quota branch from 0758e65 to 3c2ea80 Compare August 20, 2025 08:42

Refactor CreatePVCFromVolumClaimTemplate condition logic

e0410bf

Co-authored-by: Stanislav Jakuschevskij <[email protected]>

vdemeester force-pushed the 7672-retry-pvc-quota branch from 3c2ea80 to e0410bf Compare August 20, 2025 10:03

waveywaves self-assigned this Aug 20, 2025

waveywaves approved these changes Aug 20, 2025

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2025

aThorp96 approved these changes Aug 20, 2025

View reviewed changes

khrm reviewed Aug 20, 2025

View reviewed changes

vdemeester added the lgtm Indicates that a PR is ready to be merged. label Aug 20, 2025

tekton-robot merged commit c836c24 into tektoncd:main Aug 20, 2025
27 checks passed

github-project-automation bot moved this from Todo to Done in Tekton Community Roadmap Aug 20, 2025

vdemeester deleted the 7672-retry-pvc-quota branch August 20, 2025 16:38

This was referenced Sep 1, 2025

Do not fail PipelineRun if pvc creation error is because of exceeded … #8999

Closed

[release-v1.3.x] Do not fail PipelineRun if pvc creation error is because of exceeded quotas #9000

Merged

Do not fail PipelineRun if pvc creation error is because of exceeded quotas #8903

Do not fail PipelineRun if pvc creation error is because of exceeded quotas #8903

Conversation

vdemeester commented Jul 22, 2025

Changes

Submitter Checklist

Release Notes

Uh oh!

vdemeester commented Jul 22, 2025

Uh oh!

tekton-robot commented Jul 22, 2025

Uh oh!

tekton-robot commented Jul 22, 2025

Uh oh!

vdemeester commented Jul 23, 2025

Uh oh!

tekton-robot commented Jul 23, 2025

Uh oh!

vdemeester commented Jul 23, 2025

Uh oh!

vdemeester commented Jul 23, 2025

Uh oh!

twoGiants commented Jul 24, 2025

Uh oh!

twoGiants commented Jul 24, 2025

Uh oh!

twoGiants left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

twoGiants Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

twoGiants Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tekton-robot commented Aug 19, 2025

Uh oh!

tekton-robot commented Aug 19, 2025

Uh oh!

Uh oh!

waveywaves commented Aug 20, 2025

Uh oh!

tekton-robot commented Aug 20, 2025

Uh oh!

tekton-robot commented Aug 20, 2025

Uh oh!

tekton-robot commented Aug 20, 2025

Uh oh!

tekton-robot commented Aug 20, 2025

Uh oh!

aThorp96 left a comment

Choose a reason for hiding this comment

Uh oh!

tekton-robot commented Aug 20, 2025

Uh oh!

khrm left a comment

Choose a reason for hiding this comment

Uh oh!

tekton-robot commented Aug 20, 2025

Uh oh!

Uh oh!

aThorp96 commented Aug 20, 2025

Uh oh!

mbpavan commented Aug 29, 2025

Uh oh!

waveywaves commented Aug 29, 2025

Uh oh!

Uh oh!