Implement the ShiftingMatryoshka algorithm for the PowerManager

[shsms]

This PR removes the set_operating_point feature and replaces the power manager's original Matryoshka algorithm with the new ShiftingMatryoshka algorithm.

With the new algorithm, power proposals from actors are added to get the target power for the components, and higher-priority actors can limit the bounds available to lower-priority actors.

[llucax]

I just gave this a quick look, only to the structural changes.

I wonder why removing the old algorithm if we can support more than one. I can understand if it is to remove the maintenance cost, but I would delay the removal until we collected more experience.

[shsms]

I wonder why removing the old algorithm if we can support more than one. I can understand if it is to remove the maintenance cost, but I would delay the removal until we collected more experience.

This was because pylint was complaining about duplicate lines, plus it is a separate commit, which should be easy to revert if necessary. If you prefer, I can drop that commit, but we'll need to find a way to silence pylint either by refactoring the tests or pylint config.

But if we remove the current PR's limiting-only option like I mentioned in some comments above, this would stop being an issue as well, so we can revisit this after we decide that.

[llucax]

This was because pylint was complaining about duplicate lines, plus it is a separate commit, which should be easy to revert if necessary. If you prefer, I can drop that commit, but we'll need to find a way to silence pylint either by refactoring the tests or pylint config.

We should definitely not remove code because pylint complains about duplication :D

Can't we just add a # pylint: disable=<duplication> at the beginning of the file(s) reporting duplication?

[Copilot]

Pull Request Overview

This PR replaces the original Matryoshka algorithm with the new ShiftingMatryoshka algorithm in the power manager while removing the “set_operating_point” parameter from pool APIs and associated documentation.

• Introduces a new algorithm implementation in _shifting_matryoshka.py that aggregates proposals using shifted bounds.
• Updates documentation, tests, and pool creation methods to reflect the removal of the set_operating_point parameter.
• Refactors the power managing actor to use a unified subscriptions structure and maps the algorithm selection accordingly.

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/frequenz/sdk/microgrid/_power_managing/_shifting_matryoshka.py	New algorithm implementation and handling of bounds via power shifting.
src/frequenz/sdk/microgrid/init.py	Updated documentation to describe the new handling of power proposals.
src/frequenz/sdk/timeseries/*	Removed obsolete set_operating_point parameter and cleaned up related docstrings.
tests/*	Updated tests to match changes in target power calculation and bounds shifting.
RELEASE_NOTES.md	Updated to document removal of the set_operating_point parameter and details of the new algorithm.
src/frequenz/sdk/microgrid/_power_managing/_power_managing_actor.py	Refactored to use the new algorithm selection and unified subscriptions mapping.
src/frequenz/sdk/microgrid/_data_pipeline.py	Removed extra set_operating_point parameter from pool creation methods.

Comments suppressed due to low confidence (2)

src/frequenz/sdk/microgrid/_power_managing/_shifting_matryoshka.py:147

• [nitpick] Verify that subtracting the proposal power from the lower bound produces the intended result in all edge cases. Consider adding an inline comment to explain the rationale behind shifting the bounds in this manner.

lower_bound = lower_bound - proposal_power

src/frequenz/sdk/microgrid/_power_managing/_power_managing_actor.py:79

• [nitpick] Consider renaming '_subscriptions' to a more descriptive name (e.g. '_report_subscriptions') to clarify that these senders are used for reporting status.

self._subscriptions: dict[frozenset[int], dict[int, Sender[_Report]]] = {}

[llucax]

Didn't do a super in-depth review, in particular I didn't try to validate or fully understand the algorithm code, but I did follow the examples/new tests to understand the algorithm and it looks good to me and it was easier to follow (not sure if because I'm now more familiar with the algorithm or because the new wording focused on shifting :D).

Just a few minor comments, otherwise LGTM.

One final question though. The old matryoshka algorithm still works as before the operating point setting/group/shifting was added, right? The removal of the set_operating_point option only affects that?

[llucax]

I would move this docs to the class, as the module is private, it will never be rendered in the documentation website otherwise (unless you are explicitly including it in docs/.

[shsms]

oh wow, I forgot to update these docs. These are for the old matryoshka algorithm. I will do that and move it to the class.

[llucax]

Hehe, right, I was reviewing commit by commit and commented on this when I saw you added the file but then didn't realized you didn't update it afterwards.

[shsms]

updated

[llucax]

I'm shaking while about bounds due to PTSD after fighting with bounds updates for the v0.17 microgrid API. I hope it is not too hard to merge 😱

[shsms]

it should be fine because you're handling it in the compatibility layer from the client right?

[llucax]

Half and half, what was hard was how to cope with bounds being None/NaN/0, as sometimes 0 means None/NaN because we were blindly reading the protobuf value, and the default in protobuf (if the value is missing) is 0 for int. So these semantic differences I think are not handled centrally in the compatibility layer, but I'm not sure right now.

[shsms]

We should expect that the API's bounds are always correct. If we get None/NaN/0 from the API, then those should be treated as 0.

[llucax]

Are all these checks for proposal_power, proposal_lower and proposal_upper intentionally covering for an eventual 0.0 besides None or was the intention to only match None? Since in the code below, you are actually checking for None explicitly, I guess it is intentional, but maybe it would be best to add some comment to say it explicitly (either way).

[shsms]

intentionally covering for an eventual 0.0 besides None or was the intention to only match None

I don't understand what that means. Of course, I can add comments for these, though.

[shsms]

added more comments, also fixed a bug.

[llucax]

I was talking specifically about using a nullable int as a boolean implicitly, the next_proposal.bounds.lower or lower_bound, f proposal_power and ..., etc. In these cases both 0 and None will be considered False, so the question was if this was considered. Sometimes I feel it might be good to have a linting rule that requires writing if (proposal_power == 0 or proposal_power is None) and (... (but only sometimes, because other times I guess it can be overkill 😆 )

[shsms]

Oh damn, you're right, we can't continue here if proposal_power == 0. Stupid Python and its weird implicit booleans. I will make a PR next week.

Sometimes I feel it might be good to have a linting rule that requires writing if (proposal_power == 0 or proposal_power is None) and (... (but only sometimes, because other times I guess it can be overkill 😆 )

That sounds like a great idea. It is not overkill. Expressing our intentions clearly is always the way to go.

[llucax]

Upvote so it is ready when we switch to ruff :)

• Suggested lint rule: disallow implicit-cast-to-bool astral-sh/ruff#10949

[shsms]

Looks like I had already changed this before merging:

frequenz-sdk-python/src/frequenz/sdk/microgrid/_power_managing/_shifting_matryoshka.py

Line 136 in 6d106d1

if proposal_power is not None:

So nothing to do.

[llucax]

Maybe like in the test, adding a column with the "adjusted power" could add some extra clarity.

[shsms]

done

[llucax]

All (or at least some of) the old documentation here should still be useful for people interested in the old Matryoshka algorithm, so not sure if replacing it is the best choice. I wonder if the docs shouldn't be moved to the algorithms themselves, and either referenced here (or see if there is a way to include it with mkdocs, but this won't work/look nice when reading the docs from the editor). Maybe in here we can just say more generically that there are 2 available algorithms, say which is the default, and then just link to the algorithm docs for details? If the plan is to remove the old algo soon, I guess it is fine to move forward as it is, but if we really want to support more than one algorithm, the docs should be structured in a way that we can explain all of them and switch the default easily without rewriting the whole __init__.py docs.

[shsms]

The code for the old algorithm is still there, but users cannot choose it. Chances are we'll not need the old algorithm anymore. We could introduce new algorithms in the future just for specific component types, like EVs, etc. But we don't have to change the docs structure until we offer multiple algorithms for them to choose from.

[ela-kotulska-frequenz]

I am sorry but I still don't understand new approach fully. :D

For example (test based on test_shifting_matryoshka.py):

    system_bounds = _base_types.SystemBounds(
        timestamp=datetime.now(tz=timezone.utc),
        inclusion_bounds=timeseries.Bounds(
            lower=Power.from_watts(-1000.0), upper=Power.from_watts(1000.0)
        ),
        exclusion_bounds=None,
    )

    tester = StatefulTester(batteries, system_bounds)
    tester.tgt_power(priority=7, power=150.0, bounds=(100.0, 200.0), expected=150.0)
    tester.tgt_power(priority=6, power=50.0, bounds=(0, 150.0), expected=200.0)

Two actors (not aware of each other and not aware of any bounds) sets power and bounds.
I expected: target_power = 100 and bounds = (100..150) ,
Actual result: target_power = 200.
So two things happen:

1. target_power is bigger then any power proposal
2. Actors with priority 6 should narrow the upper bound to 150 but it doesn't.

Is it expected and why?

Another question: How actor should should be aware of other actor and sdk bounds? I am asking because some actors don't look at any bounds, now. They assume PowerManager will update bounds.
But with this algorithm actor should be aware of the power proposed by another actors , because his proposed power will be combined with it.

[shsms]

I am sorry but I still don't understand new approach fully. :D

Okay, that's good because now I know I was right to be skeptical about there being no questions when I explained it last Tuesday. But it was a nice few days when I thought that I must have done a really good job explaining and that we finally have created a fully consistent, easy-to-understand algorithm. :D We can go through it again on Monday.

Two actors (not aware of each other and not aware of any bounds) sets power and bounds. I expected: target_power = 100 and bounds = (100..150) , Actual result: target_power = 200.

The original matryoshka algorithm would have produced 100 here. The new algorithm adds the request powers from all the actors.

So two things happen:

1. target_power is bigger then any power proposal

Yes, it is the sum of the two actors. That is how the shifting matryoshka algorithm differs from the older algorithm, which was just limiting.

2. Actors with priority 6 should narrow the upper bound to 150 but it doesn't.

Actor pri 6 narrows the bounds only for itself and for actors with a lower pri than 6. Actor pri 7 sets 150. And that becomes the 0 point for Actor pri 6. Because Actor 7 limits bounds to 100..200, according to the new zero point, the available range for Actor 6 becomes -50..50. When Actor 6 sets power to 50, it has it is within its available range.

You could try setting other powers from 6. higher than 50, lower than 50, and negative values.

Is it expected and why?

Another question: How actor should should be aware of other actor and sdk bounds? I am asking because some actors don't look at any bounds, now. They assume PowerManager will update bounds. But with this algorithm actor should be aware of the power proposed by another actors , because his proposed power will be combined with it.

No, they are still independent. They don't have access to what's going on with the actual batteries, or what other actors are setting.