Skip to content

Charm gets stuck trying to escape "error" state #191

@jsimpso

Description

@jsimpso

Bug Description

When the lego-operator charm moves directly from error state to maintenance state, it is unable to successfully complete a maintenance operation.

The charm attempts to set status back to error here:

lego-operator/src/charm.py

Lines 431 to 434 in d6fd0fa

previous_status = self.unit.status
self.unit.status = MaintenanceStatus(message)
yield
self.unit.status = previous_status

This trips an error because Ops doesn't consider error to be a settable state:

https://github.com/canonical/operator/blob/6337b1f045be75dc96860113acddf93bf86bb76f/ops/model.py#L77-L78

To Reproduce

  1. Deploy lego with config or conditions that put it in an error state
  2. Trigger the maintenance_status context manager

Environment

4/edge rev 274 // juju 3.6.14 // OpenStack machine model

Relevant log output

2026-02-04 01:18:02 WARNING unit.lego-ingress-ps7-snapstore/1.update-status logger.go:60 2026/02/04 01:18:02 [INFO] acme: Registering account for is-admin@canonical.com
2026-02-04 01:18:02 WARNING unit.lego-ingress-ps7-snapstore/1.update-status logger.go:60 2026/02/04 01:18:02 [INFO] [grafana.ubunet.canonical.com] acme: Obtaining bundled SAN certificate given a CSR
2026-02-04 01:18:03 WARNING unit.lego-ingress-ps7-snapstore/1.update-status logger.go:60 2026/02/04 01:18:03 [INFO] [grafana.ubunet.canonical.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/...
2026-02-04 01:18:03 WARNING unit.lego-ingress-ps7-snapstore/1.update-status logger.go:60 2026/02/04 01:18:03 [INFO] [grafana.ubunet.canonical.com] acme: authorization already valid; skipping challenge
2026-02-04 01:18:03 WARNING unit.lego-ingress-ps7-snapstore/1.update-status logger.go:60 2026/02/04 01:18:03 [INFO] [grafana.ubunet.canonical.com] acme: Validations succeeded; requesting certificates
2026-02-04 01:18:03 WARNING unit.lego-ingress-ps7-snapstore/1.update-status logger.go:60 2026/02/04 01:18:03 [INFO] Wait for certificate [timeout: 30s, interval: 500ms]
2026-02-04 01:18:08 WARNING unit.lego-ingress-ps7-snapstore/1.update-status logger.go:60 2026/02/04 01:18:08 [INFO] [grafana.ubunet.canonical.com] Server responded with a certificate.
2026-02-04 01:18:08 DEBUG unit.lego-ingress-ps7-snapstore/1.juju-log server.go:405 No SANs found in certificate
2026-02-04 01:18:08 DEBUG unit.lego-ingress-ps7-snapstore/1.juju-log server.go:405 No SANs found in certificate
2026-02-04 01:18:08 INFO unit.lego-ingress-ps7-snapstore/1.juju-log server.go:405 Certificate relation data updated
2026-02-04 01:18:08 INFO unit.lego-ingress-ps7-snapstore/1.juju-log server.go:405 Certificate relation data updated
2026-02-04 01:18:08 INFO unit.lego-ingress-ps7-snapstore/1.juju-log server.go:405 generated certificate for domain grafana.ubunet.canonical.com
2026-02-04 01:18:08 ERROR unit.lego-ingress-ps7-snapstore/1.juju-log server.go:405 Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/src/charm.py", line 395, in <module>
    main(LegoCharm)  # type: ignore
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/__init__.py", line 356, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 504, in main
    manager.run()
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 488, in run
    self._emit()
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 423, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 467, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/framework.py", line 1036, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/src/charm.py", line 96, in _configure
    self._configure_certificates()
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/src/charm.py", line 115, in _configure_certificates
    with self.maintenance_status(
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/src/charm.py", line 278, in maintenance_status
    self.unit.status = previous_status
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/model.py", line 649, in status
    self._backend.status_set(
  File "/var/lib/juju/agents/unit-lego-ingress-ps7-snapstore-1/charm/venv/lib/python3.10/site-packages/ops/model.py", line 3902, in status_set
    raise InvalidStatusError(f'status must be in {_SETTABLE_STATUS_NAMES}, not {status!r}')
ops.model.InvalidStatusError: status must be in ('active', 'blocked', 'maintenance', 'waiting'), not 'error'
2026-02-04 01:18:08 ERROR juju.worker.uniter.operation runhook.go:180 hook "update-status" (via hook dispatching script: dispatch) failed: exit status 1
2026-02-04 01:18:08 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "update-status" hook
2026-02-04 01:18:49 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "update-status" hook

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions