Skip to content

Upgrade agent service (Pydantic v2) #6508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 10, 2024
Merged

Upgrade agent service (Pydantic v2) #6508

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0567fea
upgrade reqs
giancarloromeo Oct 9, 2024
214be22
fix code
giancarloromeo Oct 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 22 additions & 13 deletions services/agent/requirements/_base.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ aiohttp==3.8.5
# aiodocker
aiosignal==1.2.0
# via aiohttp
annotated-types==0.7.0
# via pydantic
anyio==3.6.2
# via
# httpx
Expand Down Expand Up @@ -41,16 +43,10 @@ click==8.1.3
# uvicorn
dnspython==2.2.1
# via email-validator
email-validator==1.3.0
email-validator==2.2.0
# via pydantic
fastapi==0.96.0
fastapi==0.115.0
Copy link
Member

@pcrespov pcrespov Oct 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK so we also upgrade fastapi to the latest?

Copy link
Contributor Author

@giancarloromeo giancarloromeo Oct 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup!

# via
# -c requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../requirements/constraints.txt
# -r requirements/../../../packages/service-library/requirements/_fastapi.in
# -r requirements/_base.in
# prometheus-fastapi-instrumentator
Expand Down Expand Up @@ -106,27 +102,38 @@ prometheus-client==0.19.0
# prometheus-fastapi-instrumentator
prometheus-fastapi-instrumentator==6.1.0
# via -r requirements/../../../packages/service-library/requirements/_fastapi.in
pydantic==1.10.2
pydantic==2.9.2
# via
# -c requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/settings-library/requirements/_base.in
# -c requirements/../../../packages/service-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../requirements/constraints.txt
# -r requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/../../../packages/settings-library/requirements/_base.in
# -r requirements/_base.in
# fastapi
# pydantic-extra-types
# pydantic-settings
pydantic-core==2.23.4
# via pydantic
pydantic-extra-types==2.9.0
# via -r requirements/../../../packages/models-library/requirements/_base.in
pydantic-settings==2.5.2
# via
# -r requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/../../../packages/settings-library/requirements/_base.in
pygments==2.15.1
# via rich
pyrsistent==0.19.2
# via jsonschema
python-dateutil==2.8.2
# via arrow
python-dotenv==1.0.0
# via -r requirements/_base.in
# via
# -r requirements/_base.in
# pydantic-settings
rich==13.4.2
# via
# -r requirements/../../../packages/settings-library/requirements/_base.in
Expand All @@ -143,7 +150,7 @@ sniffio==1.3.0
# via
# anyio
# httpx
starlette==0.27.0
starlette==0.38.6
# via
# -c requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
Expand All @@ -154,10 +161,12 @@ starlette==0.27.0
# fastapi
typer==0.12.3
# via -r requirements/../../../packages/settings-library/requirements/_base.in
typing-extensions==4.4.0
typing-extensions==4.12.2
# via
# aiodocker
# fastapi
# pydantic
# pydantic-core
# typer
uvicorn==0.19.0
# via
Expand Down
19 changes: 11 additions & 8 deletions services/agent/requirements/_test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ aiosignal==1.2.0
# via
# -c requirements/_base.txt
# aiohttp
annotated-types==0.7.0
# via
# -c requirements/_base.txt
# pydantic
anyio==3.6.2
# via
# -c requirements/_base.txt
Expand Down Expand Up @@ -81,8 +85,6 @@ ecdsa==0.19.0
# moto
# python-jose
# sshpubkeys
exceptiongroup==1.2.2
# via pytest
faker==27.0.0
# via -r requirements/_test.in
flask==3.0.3
Expand Down Expand Up @@ -178,11 +180,15 @@ pyasn1==0.6.0
# rsa
pycparser==2.22
# via cffi
pydantic==1.10.2
pydantic==2.9.2
# via
# -c requirements/../../../requirements/constraints.txt
# -c requirements/_base.txt
# aws-sam-translator
pydantic-core==2.23.4
# via
# -c requirements/_base.txt
# pydantic
pyparsing==3.1.2
# via moto
pyrsistent==0.19.2
Expand Down Expand Up @@ -257,16 +263,13 @@ sshpubkeys==3.3.1
# via moto
sympy==1.13.2
# via cfn-lint
tomli==2.0.1
# via
# coverage
# pytest
typing-extensions==4.4.0
typing-extensions==4.12.2
# via
# -c requirements/_base.txt
# aws-sam-translator
# cfn-lint
# pydantic
# pydantic-core
urllib3==2.2.2
# via
# -c requirements/../../../requirements/constraints.txt
Expand Down
12 changes: 1 addition & 11 deletions services/agent/requirements/_tools.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,22 +77,12 @@ setuptools==69.2.0
# -c requirements/_base.txt
# -c requirements/_test.txt
# pip-tools
tomli==2.0.1
# via
# -c requirements/_test.txt
# black
# build
# mypy
# pip-tools
# pylint
tomlkit==0.13.2
# via pylint
typing-extensions==4.4.0
typing-extensions==4.12.2
# via
# -c requirements/_base.txt
# -c requirements/_test.txt
# astroid
# black
# mypy
virtualenv==20.26.3
# via pre-commit
Expand Down
2 changes: 1 addition & 1 deletion services/agent/src/simcore_service_agent/core/application.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def create_app() -> FastAPI:
# SETTINGS
settings = ApplicationSettings.create_from_envs()
_setup_logger(settings)
logger.debug(settings.json(indent=2))
logger.debug(settings.model_dump_json(indent=2))

assert settings.SC_BOOT_MODE # nosec
app = FastAPI(
Expand Down
15 changes: 10 additions & 5 deletions services/agent/src/simcore_service_agent/core/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from typing import Final

from models_library.basic_types import BootModeEnum, LogLevel
from pydantic import AnyHttpUrl, Field, NonNegativeInt, validator
from pydantic import AliasChoices, AnyHttpUrl, Field, NonNegativeInt, field_validator
from settings_library.base import BaseCustomSettings
from settings_library.r_clone import S3Provider
from settings_library.utils_logging import MixinLoggingSettings
Expand All @@ -11,16 +11,21 @@

class ApplicationSettings(BaseCustomSettings, MixinLoggingSettings):
LOGLEVEL: LogLevel = Field(
LogLevel.WARNING.value, env=["AGENT_LOGLEVEL", "LOG_LEVEL", "LOGLEVEL"]
LogLevel.WARNING.value,
validation_alias=AliasChoices(
"AGENT_LOGLEVEL",
"LOG_LEVEL",
"LOGLEVEL",
),
)
SC_BOOT_MODE: BootModeEnum | None

AGENT_VOLUMES_LOG_FORMAT_LOCAL_DEV_ENABLED: bool = Field(
default=False,
env=[
validation_alias=AliasChoices(
"AGENT_VOLUMES_LOG_FORMAT_LOCAL_DEV_ENABLED",
"LOG_FORMAT_LOCAL_DEV_ENABLED",
],
),
description="Enables local development log format. WARNING: make sure it is disabled if you want to have structured logs!",
)
AGENT_VOLUMES_CLEANUP_TARGET_SWARM_STACK_NAME: str = Field(
Expand All @@ -47,7 +52,7 @@ class ApplicationSettings(BaseCustomSettings, MixinLoggingSettings):
)
AGENT_PROMETHEUS_INSTRUMENTATION_ENABLED: bool = True

@validator("LOGLEVEL")
@field_validator("LOGLEVEL")
@classmethod
def valid_log_level(cls, value) -> LogLevel:
return LogLevel(cls.validate_log_level(value))
8 changes: 4 additions & 4 deletions services/agent/tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,11 @@
import pytest
import simcore_service_agent
from aiodocker.volumes import DockerVolume
from common_library.pydantic_networks_extension import HttpUrlLegacy
from models_library.basic_types import BootModeEnum
from models_library.services import RunID
from moto.server import ThreadedMotoServer
from pydantic import HttpUrl, parse_obj_as
from pydantic import HttpUrl, TypeAdapter
from settings_library.r_clone import S3Provider
from simcore_service_agent.core.settings import ApplicationSettings

Expand Down Expand Up @@ -187,9 +188,8 @@ def caplog_info_debug(


@pytest.fixture(scope="module")
def mocked_s3_server_url(mocked_aws_server: ThreadedMotoServer) -> HttpUrl:
def mocked_s3_server_url(mocked_aws_server: ThreadedMotoServer) -> HttpUrlLegacy:
Copy link
Contributor Author

@giancarloromeo giancarloromeo Oct 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This HttpUrlLegacy has been introduced to be compliant with the previous HttpUrl (Pydantic v1). It solves the issue with the trailing slash.

See: https://docs.pydantic.dev/latest/migration/#url-and-dsn-types-in-pydanticnetworks-no-longer-inherit-from-str

# pylint: disable=protected-access
return parse_obj_as(
HttpUrl,
return TypeAdapter(HttpUrlLegacy).validate_python(
f"http://{mocked_aws_server._ip_address}:{mocked_aws_server._port}", # noqa: SLF001
)
2 changes: 1 addition & 1 deletion services/agent/tests/unit/test_modules_volumes_cleanup_s3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ async def test_store_to_s3(
)

await _download_files_from_bucket(
endpoint=mocked_s3_server_url,
endpoint=f"{mocked_s3_server_url}",
access_key="xxx",
secret_key="xxx",
bucket_name=bucket,
Expand Down
Loading