Refresh the component graph (and pipeline formulas) when it is updated

[llucax]

What's needed?

We need to be able to pick up and adapt to changes in the component graph dynamically. This means we need to rebuild formulas in the data pipeline and power manager (at least) to accommodate to the new graph.

Proposed solution

Alternatives

Polling

The solution we can implement now would be to poll the microgrid API (fetch components and connections) every some amount of time, compare the new component graph to the current one, and trigger an update if it changed.

Event based

We could have a more efficient way to detect changes if the microgrid API is extended to add an RPC that streams updates to the component graph.

Updates

We would need to update all the affected formulas in the data pipeline, like formulas in *Pools, etc. and the power manager to also adapt to the new graph.

[llucax]

Some historical comments present in an old internal issue about this (so they doesn't get lost, but take with a grain of salt because they could be very outdated):

From @ela-kotulska-frequenz, dated Aug/Sep 2022:

---

Component graph can change anytime. python-sdk should handle it and refresh the api.
How we should check it if component graph changed:

• Trigger request to the API every 30 minutes to get component graph and compare with the previous one
• Has a stream with the microgrid API. It will send Component graph everytime when there is new change.
• Component graph can change only in the UI. So maybe Cloud connector could change config file whenever component graph will change.
• Each component has a timestamp informing when it was recently changed. Maybe we could use that to find out if the component graph changed.

---

Requirements

Watch for changes in component graph. When the component graph will change we should notify all actors about that.

Problem 1

1. It should be made in right order. First component graph should be updated, then actors should get notification about that change. Reason:
   • It is async - we can't be sure that MicrogridApi will get notification before other actors,
   • Updating component graph is async task (it has awaits)
   • It might take some time, because of correction feature ¹.
   • Creating component graph could fail - in that case we use old graph and users should not be notified.

Solution
We have singleton microgrid_api that stores microgrid_api_client, and component_graph.

Microgrid_api should be an actor that waits fro changes in component graph. If there is a change it should update component graph (if possible) and send to the actors that component graph has been changed. MicrogridApi should notify the actors about the change using channels.

• The channel for microgrid_api could be global (like microgrid_api instance). We would have extra method get_microgrid_api_notification_channel that returns receiver for that channel.
• The channel for microgrid_api will be created in the microgrid_api instance. microgrid_api will has property update_notification_channel that returns receiver for that channel.

Problems with both solutions: microgrid_api would be the only actor that can be called directly (it has method microgrid_api, component_graph).

Problem 2

3. How to watch for changes in component graph (described above in the first comment)

---

Problem 2

We decided to:

• Trigger request to the API every 30 minutes to get component graph and compare with the previous one
• Has a stream with the microgrid API. It will send Component graph everytime when there is new change.

Because other solution require synchronization. For example: Cloud sync can change sdk config file before the microgrid API with change component graph.

---

Footnotes

1. UI or cloud connector can give incomplete graph. In that case we should correct it by site.

   src/frequenz/sdk/microgrid/graph.py:_correct_graph_errors

   Should handle below cases:

   • It is not required for PV_ARRAY, GRID to be in component graph from UI and in SDK component graph (users don't needs it)
   • grid meter has no predecessors then it is grid meter.
   • meter can be dangling - it has no successor and predecessor ( for example CHP meter )
   • If the leaf is battery (bms) then his path should be bms <- inverter (hybrid) <- bms_mter <- root. BMS might be connected to more then one inverter.
   • If the inverter is leaf then his path should be inverter (solar or hybrid) <- pv_meter <- root.
   • If the leaf is ev_charger then the path should be ev_charger <- ev_meter <- root
   • root might be junction or grid

   The type of the inverter/meter should be retrieved from protobuf Component.data.variant field. It is not implemented, yet.
   We should wait for that first.

   UPDATE:
   The feature should be implemented in a way that it does not break if the microgrid API doesn't expose the variant. It should gracefully fails in that case. Possibly in the same as in every case when the graph is invalid. This would allow us to update app etc. even the microgrid API service hasn't been updated yet ↩