[DPE-10203] test(async-replication): cover the ticket's full dead-DC teardown sequence - #1914
Draft
marceloneppel wants to merge 13 commits into
Draft
Conversation
marceloneppel
added
the
not bug or enhancement
2 tasks
github-actions
Bot
added
the
Libraries: Out of sync
marceloneppel
added a commit
that referenced
this pull request
Aug 26, 2026
Carry the scoped _relation test refactor so #1914's diff stays test-file-only. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
marceloneppel
force-pushed
the
dpe-10203-async-replication-recovery
branch
from
September 1, 2026 18:00
9700e8e to
336a4d6
Compare
marceloneppel
force-pushed
the
dpe-10203-dead-dc-recovery-test
branch
4 times, most recently
from
September 1, 2026 20:16
f03e521 to
ece19b7
Compare
…uence Three-model regression test for DPE-10203 replicating the reported scenario end to end: the whole datacenter dies (the cluster's units and its Raft-witness watcher alike), the standby is force-promoted, the dead relation is attacked with remove-relation --force (the ticket's Issue 1, for which Juju delivers no events) and the consumed offer is cleared with remove-saas --force — neither delivering relation-broken — then create-replication re-establishes replication to a fresh cluster. Asserts the labelless-secret contract of the fix: the owner owns the shared secret with no label (nothing for a stale alias to collide with) and no side registers a consumer-side label alias under either legacy name. The remove-saas workaround tolerates 'not found' so the test stays valid if Juju ever honors remove-relation --force. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…h model The dead-DC teardown regression only asserted secret-label hygiene; the client data itself was never exercised. Relate postgresql-test-app to every cluster, start continuous writes on the primary before the DC dies, and assert the pre-death data survives the force-promotion and re-appears on the fresh re-replication target — mirroring the original async replication tests, which relate the test app to each cluster. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…down The three-model fixtures switch the shared JujuFixture to each extra model; jubilant's destroy_model then nulls the fixture's model when the destroyed model matches, so the temp-model teardown assert (juju.model is not None) failed after the last extra model was destroyed — reported as an ERROR that masked the passing assertions. Switching back to the previous model after each destroy keeps the fixture consistent. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
create-replication and promote-to-primary read _get_primary_cluster, which counts an orphaned promoted-cluster-counter left in peer data by a dead-DC teardown whose relation-broken never fired. With the only reconciler in update-status, the action fails with "There is already a replication set up." (or promote-to-primary wrongly sees a primary) until an update-status cycle happens to run — which the dead-DC recovery window cannot rely on. Both actions now clear the stale counter before their guard; the mirror check inside clear_stale_promotion keeps live replications untouched. The update- status reconciler stays (idempotent, cheap, and covers paths that don't go through the actions). The dead-DC integration test drops its 10-minute retry loop, whose only purpose was absorbing that update- status latency. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…n types Condense the multi-paragraph dead-DC comments to the repo's short single-purpose comments, trim the regression test module docstring to the same shape, and narrow two of the teardown exception handlers to their concrete failure types (DeployedWithoutTrustError, RetryError, ModelError). The get_standby_leader handler keeps Exception: it probes across three layers whose failure modes cannot be enumerated here, and a crash would defeat the counter-clear reconciliation. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
marceloneppel
force-pushed
the
dpe-10203-dead-dc-recovery-test
branch
from
September 1, 2026 20:32
f994689 to
6b2957e
Compare
…he forbidden list The labelless-ownership fix landed without any secret labels; only the legacy consumer alias can go stale. Trim the constant to that one label and rename it accordingly. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…rade The refresh deploys Charmhub 16/stable (whose charm owns the shared secret under the legacy label) and then upgrades to the labelless design. The adoption path must re-use the same secret id — a switch would wedge any consumer still running label-attaching code during the cross-version window. Assert the published secret id is unchanged across the refresh; nothing in the upgrade test asserted secret identity before. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…id helper jubilant's Juju.cli() returns the command output as a string, not a subprocess result; the helper read .stdout on it and crashed with AttributeError on both architectures during the upgrade run. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
… test The dead-DC recovery test was missing its spread task, so the spread-based integration matrix never ran it. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…g switch API jubilant has no Juju.switch; the teardown must restore the fixture's model by plain attribute assignment (the same mechanism destroy_model uses to null it). Also trim the module docstring and two inline comments to the repo's comment style. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
The helper collected the values but never returned them, so test_data_replication crashed with len(None) as soon as the spread matrix started running it. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
2 tasks
… file Move the helpers introduced with the dead-DC recovery test into high_availability_helpers_new so the other async-replication tests can reuse them, matching the existing convention: - get_async_secret_labels (was _async_secret_labels) - consumer_alias_exists (was _consumer_alias_exists) - wait_resilient (was _wait_resilient) - start_continuous_writes (was _start_continuous_writes) - get_published_secret_id (was _published_secret_id, from the upgrade test) The multi-model pytest fixtures stay in the dead-DC test module: they are fixtures bound to that scenario's model suffixes, not shared helpers. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
… move Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
marceloneppel
force-pushed
the
dpe-10203-dead-dc-recovery-test
branch
from
September 5, 2026 15:38
cb975d1 to
d5096c3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
DPE-10203: the fix set in #1913 has no CI coverage — the dead-DC recovery scenario needs three models, which the spread-based integration matrix doesn't run. The reported sequence also has steps a naive test would skip: the whole datacenter dies (the watcher's machine together with the cluster's units — "all Rome units"), and the dead relation is first attacked with
remove-relation --force— the ticket's Issue 1, for which Juju delivers no events — before theremove-saas --forceworkaround clears the offer.Solution
A three-model jubilant integration test replicating the ticket's teardown sequence end to end: dead datacenter (2xPG + Raft-witness watcher per DC), force-promotion,
remove-relation --forcethenremove-saas --force(neither deliveringrelation-broken), andcreate-replicationto a fresh cluster — which must succeed rather than deadlock. Asserts the labelless-secret contract of #1913: the owner owns the shared secret with no label, and no side registers a consumer-side label alias under either legacy name. Theremove-saasworkaround tolerates "not found", so the test stays valid if Juju ever honorsremove-relation --force.Alongside the new test, the existing upgrade integration test additionally asserts the shared secret id is stable across the refresh (an id switch would wedge any consumer still running label-attaching code mid-upgrade).
Stacked on #1913; merge after it.
Checklist