Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions RELEASE_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@
## New Features

- The `ConfigManagingActor` can now take multiple configuration files as input, allowing to override default configurations with custom configurations.
* A new `frequenz.sdk.config.load_config()` function is available to load configurations using `marshmallow_dataclass`es with correct type hints.
- Implement and standardize logging configuration with the following changes:
* Add LoggerConfig and LoggingConfig to standardize logging configuration.
* Create LoggingConfigUpdater to handle runtime config updates.
* Add `LoggerConfig` and `LoggingConfig` to standardize logging configuration.
* Create `LoggingConfigUpdater` to handle runtime config updates.
Comment on lines -25 to +27
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

* Support individual log level settings for each module.

## Bug Fixes
Expand Down
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ plugins:
- https://docs.python.org/3/objects.inv
- https://frequenz-floss.github.io/frequenz-channels-python/v1.1/objects.inv
- https://frequenz-floss.github.io/frequenz-client-microgrid-python/v0.5/objects.inv
- https://lovasoa.github.io/marshmallow_dataclass/html/objects.inv
- https://marshmallow.readthedocs.io/en/stable/objects.inv
- https://networkx.org/documentation/stable/objects.inv
- https://numpy.org/doc/stable/objects.inv
- https://typing-extensions.readthedocs.io/en/stable/objects.inv
Expand Down
2 changes: 2 additions & 0 deletions src/frequenz/sdk/config/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@

from ._config_managing import ConfigManagingActor
from ._logging_config_updater import LoggerConfig, LoggingConfig, LoggingConfigUpdater
from ._util import load_config

__all__ = [
"ConfigManagingActor",
"LoggingConfig",
"LoggerConfig",
"LoggingConfigUpdater",
"load_config",
]
45 changes: 45 additions & 0 deletions src/frequenz/sdk/config/_util.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# License: MIT
# Copyright © 2024 Frequenz Energy-as-a-Service GmbH

"""Utilities to deal with configuration."""

from collections.abc import Mapping
from typing import Any, TypeVar, cast

from marshmallow_dataclass import class_schema

T = TypeVar("T")
"""Type variable for configuration classes."""


def load_config(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe load_marshmallow_config ?

Copy link
Contributor Author

@llucax llucax Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if it is worth adding all of that to the name of the function, after all the idea was to make the loading short and neat, and it is documented what's expected. It will be a runtime error passing anything that is not a marshmallow_dataclass (actually no, I think a plain dataclass will work too! 1).

Also, this won't be able to load using a plain marshmallow.Schema, it actually needs a marshmallow_dataclass.dataclass, so if we want to make it unambigous it would have to be at least load_marshmallow_dataclass_config() 😬. At least for me if I read load_marshmallow_config() I would expect it to take a plain marshmallow.Schema.

Footnotes

  1. marshmallow_dataclass.class_schema also accept a plain dataclass and gets a schema out of it on the fly.

Copy link
Contributor Author

@llucax llucax Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I will update the docs and tests to reflect this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated!

cls: type[T],
config: Mapping[str, Any],
/,
**marshmallow_load_kwargs: Any,
) -> T:
"""Load a configuration from a dictionary into an instance of a configuration class.
The configuration class is expected to be a [`dataclasses.dataclass`][], which is
used to create a [`marshmallow.Schema`][] schema to validate the configuration
dictionary.
To customize the schema derived from the configuration dataclass, you can use
[`marshmallow_dataclass.dataclass`][] to specify extra metadata.
Additional arguments can be passed to [`marshmallow.Schema.load`][] using keyword
arguments.
Args:
cls: The configuration class.
config: The configuration dictionary.
**marshmallow_load_kwargs: Additional arguments to be passed to
[`marshmallow.Schema.load`][].
Returns:
The loaded configuration as an instance of the configuration class.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the function is internal but IMO the documentation should include the Raises section as it is the only way the caller of the function can handle validation errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it could be a good addition, but the docs say very clearly that this is just a wrapper for marshmallow.Schema.load(), so I guess users can infer it from there. I can add it in a separate PR if you think it is worth it though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add it in a separate PR if you think it is worth it though.

No, it's probably not worth it. Though I'd explicitly add it if the function is exposed to the user at some point

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is already exposed via frequenz.sdk.config though :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, would you mind to add it in a different PR then? 👼

I have some other concern that I'd like to mention before adding any Raise docs.

def test_none_settings() -> None:
    """Test loading None settings."""
    config: dict[str, Any] = {"key": "value"}

    text: str = "text"

    with pytest.raises(marshmallow.ValidationError) as exc_info:
        loaded_config = load_config(SimpleConfig, None)  # mypy complains incompatible type "None";

    with pytest.raises(marshmallow.ValidationError) as exc_info:
        loaded_config = load_config(
            SimpleConfig, config.get("unknown", None)
        )  # mypy is happpy with `None` when `get()` is used

Since mypy doesn’t raise any issues when get() is used, we can assume marshmallow will handle it. Marshmallow will raise ValidationError for the None case if default marshmallow parameters are used but that might be different when non-default parameters are set, for instance, unknown=INCLUDE will make marshmallow to raise a TypeError if None is passed as a config.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for the record, we had a quick call and decided it would be impossible to document every pitfall we can have with marshmallow in this function, so we'll leave it like this. If we touch this code again, the docs can be made a bit more clear about this being just a wrapper over marshmallow.Schema.load().

"""
instance = class_schema(cls)().load(config, **marshmallow_load_kwargs)
# We need to cast because `.load()` comes from marshmallow and doesn't know which
# type is returned.
return cast(T, instance)
69 changes: 69 additions & 0 deletions tests/config/test_util.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# License: MIT
# Copyright © 2024 Frequenz Energy-as-a-Service GmbH

"""Tests for the config utilities."""

import dataclasses
from typing import Any

import marshmallow
import marshmallow_dataclass
import pytest
from pytest_mock import MockerFixture

from frequenz.sdk.config._util import load_config


@dataclasses.dataclass
class SimpleConfig:
"""A simple configuration class for testing."""

name: str
value: int


@marshmallow_dataclass.dataclass
class MmSimpleConfig:
"""A simple configuration class for testing."""

name: str = dataclasses.field(metadata={"validate": lambda s: s.startswith("test")})
value: int


def test_load_config_dataclass() -> None:
"""Test that load_config loads a configuration into a configuration class."""
config: dict[str, Any] = {"name": "test", "value": 42}

loaded_config = load_config(SimpleConfig, config)
assert loaded_config == SimpleConfig(name="test", value=42)

config["name"] = "not test"
loaded_config = load_config(SimpleConfig, config)
assert loaded_config == SimpleConfig(name="not test", value=42)


def test_load_config_marshmallow_dataclass() -> None:
"""Test that load_config loads a configuration into a configuration class."""
config: dict[str, Any] = {"name": "test", "value": 42}
loaded_config = load_config(MmSimpleConfig, config)
assert loaded_config == MmSimpleConfig(name="test", value=42)

config["name"] = "not test"
with pytest.raises(marshmallow.ValidationError):
_ = load_config(MmSimpleConfig, config)


def test_load_config_type_hints(mocker: MockerFixture) -> None:
"""Test that load_config loads a configuration into a configuration class."""
mock_class_schema = mocker.Mock()
mock_class_schema.return_value.load.return_value = {"name": "test", "value": 42}
mocker.patch(
"frequenz.sdk.config._util.class_schema", return_value=mock_class_schema
)
config: dict[str, Any] = {}

# We add the type hint to test that the return type (hint) is correct
_: MmSimpleConfig = load_config(MmSimpleConfig, config, marshmallow_arg=1)
mock_class_schema.return_value.load.assert_called_once_with(
config, marshmallow_arg=1
)
Loading