Charm throws an error when multiple vault units are integrated with an auto-unsealer

[DanielArndt]

Bug Description

When integrating with an autounseal provider, non-leader units throw a SecretNotFoundError.

The charm recovers, but this still shouldn't happen.

To Reproduce

1. juju deploy vault-a -n 3
2. juju deploy vault-b -n 3
3. Initialtize, unseal, and authorize vault a and vault b
4. juju integrate vault-a:vault-autounseal-provides vault-b:vault-autounseal-requires

Environment

juju version: 3.4.6-genericlinux-amd64
commit: dfc1a4b

Relevant log output

unit-vault-k8s-1: 16:24:11 ERROR unit.vault-k8s/1.juju-log vault-autounseal-requires:3: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/model.py", line 3319, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-vault-k8s-1/secret-info-get', '--label', 'vault-autounseal-token', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/model.py", line 3702, in _run_for_secret
    return self._run(*args, return_output=return_output, use_json=use_json)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/model.py", line 3321, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR secret "vault-autounseal-token" not found


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/./src/charm.py", line 1618, in <module>
    main(VaultCharm)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/__init__.py", line 343, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/_main.py", line 543, in main
    manager.run()
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/_main.py", line 529, in run
    self._emit()
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/_main.py", line 518, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name, self._juju_context)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/_main.py", line 134, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/lib/charms/vault_k8s/v0/vault_autounseal.py", line 453, in _on_relation_changed
    self.on.vault_autounseal_details_ready.emit(
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/./src/charm.py", line 403, in _configure
    self._generate_vault_config_file()
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/./src/charm.py", line 1290, in _generate_vault_config_file
    autounseal_details=self._get_autounseal_configuration(),
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/./src/charm.py", line 1164, in _get_autounseal_configuration
    self._get_autounseal_vault_token(autounseal_details),
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/./src/charm.py", line 1191, in _get_autounseal_vault_token
    self._set_juju_secret(AUTOUNSEAL_TOKEN_SECRET_LABEL, {"token": vault.token})
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/./src/charm.py", line 1267, in _set_juju_secret
    secret.set_content(content)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/model.py", line 1511, in set_content
    self._id = self.get_info().id
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/model.py", line 1488, in get_info
    return self._backend.secret_info_get(id=self.id, label=self.label)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/model.py", line 3716, in secret_info_get
    result = self._run_for_secret('secret-info-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-vault-k8s-1/charm/venv/ops/model.py", line 3705, in _run_for_secret
    raise SecretNotFoundError() from e
ops.model.SecretNotFoundError
unit-vault-unsealer-0: 16:24:11 INFO juju.worker.uniter.operation ran "vault-autounseal-provides-relation-joined" hook (via hook dispatching script: dispatch)
unit-vault-k8s-2: 16:24:11 ERROR unit.vault-k8s/2.juju-log vault-autounseal-requires:3: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/model.py", line 3319, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-vault-k8s-2/secret-info-get', '--label', 'vault-autounseal-token', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/model.py", line 3702, in _run_for_secret
    return self._run(*args, return_output=return_output, use_json=use_json)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/model.py", line 3321, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR secret "vault-autounseal-token" not found


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/./src/charm.py", line 1618, in <module>
    main(VaultCharm)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/__init__.py", line 343, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/_main.py", line 543, in main
    manager.run()
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/_main.py", line 529, in run
    self._emit()
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/_main.py", line 518, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name, self._juju_context)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/_main.py", line 134, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/lib/charms/vault_k8s/v0/vault_autounseal.py", line 453, in _on_relation_changed
    self.on.vault_autounseal_details_ready.emit(
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/framework.py", line 347, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/framework.py", line 857, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/framework.py", line 947, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/./src/charm.py", line 403, in _configure
    self._generate_vault_config_file()
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/./src/charm.py", line 1290, in _generate_vault_config_file
    autounseal_details=self._get_autounseal_configuration(),
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/./src/charm.py", line 1164, in _get_autounseal_configuration
    self._get_autounseal_vault_token(autounseal_details),
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/./src/charm.py", line 1191, in _get_autounseal_vault_token
    self._set_juju_secret(AUTOUNSEAL_TOKEN_SECRET_LABEL, {"token": vault.token})
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/./src/charm.py", line 1267, in _set_juju_secret
    secret.set_content(content)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/model.py", line 1511, in set_content
    self._id = self.get_info().id
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/model.py", line 1488, in get_info
    return self._backend.secret_info_get(id=self.id, label=self.label)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/model.py", line 3716, in secret_info_get
    result = self._run_for_secret('secret-info-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-vault-k8s-2/charm/venv/ops/model.py", line 3705, in _run_for_secret
    raise SecretNotFoundError() from e
ops.model.SecretNotFoundError

Additional context

No response

[gruyaume]

@DanielArndt do you know if we are still experimenting this issue?

[DanielArndt]

@gruyaume Yes, I just experienced it recently (this week or last week, can't quite remember).