Skip to content

fix: issue 1228 race condition when rollout restart target#1229

Open
duong-se wants to merge 2 commits intohashicorp:mainfrom
duong-se:fix/issue-1228-race-condition-when-rollout-restart
Open

fix: issue 1228 race condition when rollout restart target#1229
duong-se wants to merge 2 commits intohashicorp:mainfrom
duong-se:fix/issue-1228-race-condition-when-rollout-restart

Conversation

@duong-se
Copy link
Copy Markdown

@duong-se duong-se commented Mar 2, 2026

Please reference github issue #1228

Describe the bug
We are experiencing a potential race condition when using multiple Vault dynamic secrets (e.g., Database secrets and GCP secrets) in the same Kubernetes workload via Vault Secrets Operator (VSO).

If two dynamic secrets are rotated within an extremely short time window (nanoseconds to sub-second), both secret updates may trigger a rollout restart at effectively the same timestamp (e.g., 2026-02-27T14:26:59Z).

Since VSO updates the annotation:
vso.secrets.hashicorp.com/restartedAt
with second-level precision, it is possible that both secret rotations set the exact same annotation value.

As a result:

  • The first secret rotation (e.g., database) updates the annotation and triggers a rollout restart.
  • The second secret rotation (e.g., GCP) happens within the same second and updates the annotation with the same value.
  • Kubernetes does not detect any change in the pod template because the annotation value is identical.
  • The second rollout restart is effectively ignored.
    This leads to inconsistent pod states where:
  • Some pods load old GCP credentials.
  • Some pods load new GCP credentials.
  • Database credentials may already be updated.
    This creates a temporary but critical inconsistency across replicas.

PCI review checklist

  • I have documented a clear reason for, and description of, the change I am making.

  • If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

  • If applicable, I've documented the impact of any changes to security controls.

    Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.

@duong-se duong-se requested a review from a team as a code owner March 2, 2026 15:54
@hashicorp-cla-app
Copy link
Copy Markdown

hashicorp-cla-app bot commented Mar 2, 2026

CLA assistant check
All committers have signed the CLA.

@hashicorp-cla-app
Copy link
Copy Markdown

CLA assistant check

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes


Duong Pham seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

Have you signed the CLA already but the status is still pending? Recheck it.

@duong-se duong-se force-pushed the fix/issue-1228-race-condition-when-rollout-restart branch from 14a420a to 5aaf7fd Compare March 2, 2026 16:00
@duong-se duong-se force-pushed the fix/issue-1228-race-condition-when-rollout-restart branch from 5aaf7fd to c6086e3 Compare March 2, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant