Skip to content

Flaky postgres integration test: read replica phase races the primary's first backup #282

Description

@sdairs

Problem

The Read Replica phase of cloud_postgres_crud_lifecycle creates the replica seconds after the primary reaches running, but replica provisioning gates on the primary having a completed first backup. Two observed failure modes:

Not AWS capacity — create is accepted/rejected instantly, no capacity errors in logs.

Proposal

The test only needs to prove the API surface works, not that provisioning completes. Drop the wait for postgres read replica running step and keep:

  1. Create → assert response shape (isPrimary=false, name, sane state)
  2. Replica visible in postgres_service_get_list alongside primary (any state)
  3. postgres_service_get on replica returns matching provider/region (any state)
  4. Delete replica → confirm gone → delete primary

Deleting a still-provisioning replica is proven safe: the failed PR #215 run's cleanup deleted the stuck replica in ~11s and the primary deleted cleanly after.

Callouts

  1. The non-empty hostname/connection_string assertions currently ride on the running-state poll result. They test the provisioned instance, not the API — drop them, or keep as tolerant checks on the create/get response if populated early.
  2. Dropping the wait doesn't fix the first failure mode: create itself can still be rejected with the "no backups yet" 400. Wrap create in a retry that treats that specific 400 as retryable, otherwise the flake just moves to the create step.

Worth separately flagging to the cloud team that the API inconsistently accepts a replica create against a backup-less primary and leaves it non-running for 30+ min — the 400 rejection seems like the more correct behaviour.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions