ServiceMeshConsumer in service_mesh library fails when the charm consuming the lib is scaled to more than 1 unit

[adhityaravi]

Bug Description

In ServiceMeshConsumer class provided by the service_mesh library, we access the application data bag in most of the event handlers. So when the charm consuming this library is scaled to more than one unit, all the units try to write to the app data bag which results in the charm units that are not leaders to go into an error state. Since app data bag can only be modified by the leader unit.

To Reproduce

juju deploy isito-k8s --trust --channel=2/edge
juju deploy istio-beacon-k8s --trust --channel=2/edge
juju deploy alertmanager-k8s --trust --channel=2/edge
juju scale-application alertmanager-k8s 2
juju status --watch 1s

Environment

juju v3.6.8
service mesh and core charms from channel 2/edge
microk8s substrate

Relevant log output

unit-alertmanager-k8s-1: 13:55:49 ERROR unit.alertmanager-k8s/1.juju-log service-mesh:3: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 3422, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-alertmanager-k8s-1/relation-get', '-r', '3', '-', 'alertmanager-k8s', '--app', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/src/charm.py", line 656, in <module>
    main(AlertmanagerCharm)
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/main.py", line 39, in main
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 474, in main
    manager.run()
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 458, in run
    self._emit()
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 402, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 437, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 923, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1029, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/lib/charms/istio_beacon_k8s/v0/service_mesh.py", line 324, in _relations_changed
    self.update_service_mesh()
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/lib/charms/istio_beacon_k8s/v0/service_mesh.py", line 350, in update_service_mesh
    self._relation.data[self._charm.app]["policies"] = json.dumps(mesh_policies)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 1936, in __setitem__
    self.update({key: value})
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 1976, in update
    if (key not in self and val != '') or (key in self and val != self[key])
        ^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 879, in __contains__
    return key in self._data
                  ^^^^^^^^^^
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 872, in _data
    data = self._lazy_data = self._load()
                             ^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 1838, in _load
    return self._backend.relation_get(self.relation.id, self._entity.name, self._is_app)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 3501, in relation_get
    raw_data_content = self._run(*args, return_output=True, use_json=True)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-alertmanager-k8s-1/charm/venv/lib/python3.12/site-packages/ops/model.py", line 3424, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR permission denied

Additional context

No response

[ca-scribner]

This looks like a bug on our part. To be clear, all units are allowed to access application data. but in our case update_service_mesh is updating application data. My quick take is that method should have a leader guard.

[adhityaravi]

Yea, my bad. Ive edited the issue description.