-
Notifications
You must be signed in to change notification settings - Fork 171
Closed
Description
What happened:
A build pod with the same build ID may be launched twice during the following condition.
Prerequisite:
Assume multiple buildcluster-queue-workers (buildcluster-queue-worker-A, buildcluster-queue-worker-B) are running, and a message with buildId: 12345 is queued in RabbitMQ.
- buildcluster-queue-worker-A receives the build message (buildId: 12345) from RabbitMQ and starts a build pod (podname: 12345-abcd)
- Before buildcluster-queue-worker-A returns the Ack to RabbitMQ, communication between buildcluster-queue-worker-A and RabbitMQ fails for some reason and Ack cannot be returned
- RabbitMQ requeues the message for the build (buildId: 12345)
- buildcluster-queue-worker-B receives the requeued message and starts another build pod (podname: 12345-efgh). As a result, two build pods with the same build ID are active.
What you expected to happen:
If build pods are started twice, the last build process should be stopped.
How to reproduce it:.
This occurs in limited situation, but this can happen when multiple buildcluster-queue-workers are running and only some buildcluster-queue-workers have an unstable connection to RabbitMQ.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels