Waiting for vault to finish raft leader election after scale-in / scale-out (Only 1.15 and 1.6)

[motjuste]

Bug Description

*Note: Filed by @canonical/solutions-qa as part of Charm QA

Vault stuck in waiting state with message Waiting for vault to finish raft leader election after scale-in/scale-out test.

This seems to only occur with channel 1.15 and 1.16, not 1.18.

To Reproduce

1. deploy the following minimal-bundle:
2. initialise, unseal, authorise vault; deployment succeeds and settles down
3. scale vault-k8s to 0 units
4. scale vault-k8s to 1 units
5. initialise, unseal vault; vault stuck waiting for at least 15 minutes with Waiting for vault to finish raft leader election.

Minimal-bundle:

applications:
  neighbor:
    base: ubuntu@22.04
    channel: 6/stable
    charm: mongodb-k8s
    options: {}
    revision: 113
    scale: 1
    trust: true
  self-signed-certificates:
    base: ubuntu@24.04
    channel: 1/stable
    charm: self-signed-certificates
    options: {}
    revision: 317
    scale: 1
    trust: true
  target:
    base: ubuntu@22.04
    channel: 1.16/stable
    charm: vault-k8s
    options:
      common_name: charmqa
    revision: 323
    scale: 1
    trust: true
bundle: kubernetes
relations:
- - neighbor:certificates
  - target:vault-pki
- - self-signed-certificates:certificates
  - target:tls-certificates-pki

Environment

Juju 3/stable, controlling a K8s cloud. Please see this run in Test Observer (from Canonical VPN).

Relevant log output

ERROR Failure in test_scale_in_and_scale_out_charm: juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active], units: [active] (applications: ['target'], units: ['target/0'])
INFO Juju Status:
--------------------------------------------------------------------------------
Model                           Controller                Cloud/Region    Version  SLA          Timestamp
model-21111182587-260118114424  21111182587-260118114424  k8s-production  3.6.12   unsupported  12:16:45Z

App                       Version  Status   Scale  Charm                     Channel      Rev  Address         Exposed  Message
neighbor                  6.0.24   active       1  mongodb-k8s               6/stable     113  10.152.183.154  no       
self-signed-certificates           active       1  self-signed-certificates  1/stable     317  10.152.183.224  no       
target                             waiting      1  vault-k8s                 1.16/stable  323  10.152.183.65   no       Waiting for vault to finish raft leader election

Unit                         Workload  Agent  Address     Ports  Message
neighbor/0*                  active    idle   10.1.1.58          Primary.
self-signed-certificates/0*  active    idle   10.1.2.185         
target/0*                    waiting   idle   10.1.4.128         Waiting for vault to finish raft leader election

Integration provider                   Requirer                     Interface         Type     Message
neighbor:database-peers                neighbor:database-peers      mongodb-peers     peer     
neighbor:ldap-peers                    neighbor:ldap-peers          ldap-peers        peer     
neighbor:status-peers                  neighbor:status-peers        status-peers      peer     
neighbor:upgrade-version-a             neighbor:upgrade-version-a   upgrade           peer     
self-signed-certificates:certificates  target:tls-certificates-pki  tls-certificates  regular  
target:vault-peers                     target:vault-peers           vault-peer        peer     
target:vault-pki                       neighbor:certificates        tls-certificates  regular  
--------------------------------------------------------------------------------

Additional context

No response

[saltiyazan]

Hi @motjuste
Thank you for openning the issue. We will take a look, but do you have any reason not to use 1.18?

[motjuste]

This is part of charm integration testing done by @canonical/solutions-qa where we test all currently published versions of all charms from Charmhub.