Skip to content

Upgrade invitations service (Pydantic v2) #6513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Oct 11, 2024
Merged

Upgrade invitations service (Pydantic v2) #6513

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
5d76e4d
upgrade reqs
giancarloromeo Oct 10, 2024
cafb49f
fix code
giancarloromeo Oct 10, 2024
c141118
fix mypy
giancarloromeo Oct 10, 2024
8e49f76
fix mypy
giancarloromeo Oct 10, 2024
90f4556
fix model copy
giancarloromeo Oct 10, 2024
204ecf3
fix deprecated dict
giancarloromeo Oct 10, 2024
f70fa05
fix deprecated method
giancarloromeo Oct 10, 2024
d781534
fix url type
giancarloromeo Oct 10, 2024
676e65e
make openapi.json
giancarloromeo Oct 10, 2024
998585d
fix url
giancarloromeo Oct 10, 2024
bbd7485
fix guest email strategy
giancarloromeo Oct 10, 2024
3962f85
fix import
giancarloromeo Oct 10, 2024
15349dd
fix Url serialization
giancarloromeo Oct 10, 2024
2d79cb0
fix url
giancarloromeo Oct 10, 2024
7ad1a1d
add aliases uniqueness test
giancarloromeo Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 23 additions & 19 deletions services/invitations/requirements/_base.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ aiormq==6.8.0
# via aio-pika
aiosignal==1.3.1
# via aiohttp
annotated-types==0.7.0
# via pydantic
anyio==4.3.0
# via
# fast-depends
Expand All @@ -31,10 +33,6 @@ arrow==1.3.0
# -r requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/../../../packages/service-library/requirements/_base.in
async-timeout==4.0.3
# via
# aiohttp
# redis
attrs==23.2.0
# via
# aiohttp
Expand Down Expand Up @@ -69,18 +67,10 @@ dnspython==2.6.1
# via email-validator
email-validator==2.1.1
# via pydantic
exceptiongroup==1.2.0
# via anyio
fast-depends==2.4.2
# via faststream
fastapi==0.99.1
fastapi==0.115.0
# via
# -c requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../requirements/constraints.txt
# -r requirements/../../../packages/service-library/requirements/_fastapi.in
# -r requirements/_base.in
# prometheus-fastapi-instrumentator
Expand Down Expand Up @@ -149,12 +139,11 @@ prometheus-fastapi-instrumentator==6.1.0
# via -r requirements/../../../packages/service-library/requirements/_fastapi.in
pycparser==2.21
# via cffi
pydantic==1.10.14
pydantic==2.9.2
# via
# -c requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/settings-library/requirements/_base.in
# -c requirements/../../../packages/service-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../requirements/constraints.txt
Expand All @@ -165,14 +154,30 @@ pydantic==1.10.14
# -r requirements/../../../packages/settings-library/requirements/_base.in
# fast-depends
# fastapi
# pydantic-extra-types
# pydantic-settings
pydantic-core==2.23.4
# via pydantic
pydantic-extra-types==2.9.0
# via
# -r requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/_base.in
pydantic-settings==2.5.2
# via
# -r requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/../../../packages/service-library/requirements/../../../packages/settings-library/requirements/_base.in
# -r requirements/../../../packages/settings-library/requirements/_base.in
pygments==2.17.2
# via rich
pyinstrument==4.6.2
# via -r requirements/../../../packages/service-library/requirements/_base.in
python-dateutil==2.9.0.post0
# via arrow
python-dotenv==1.0.1
# via uvicorn
# via
# pydantic-settings
# uvicorn
pyyaml==6.0.1
# via
# -c requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
Expand Down Expand Up @@ -214,7 +219,7 @@ sniffio==1.3.1
# via
# anyio
# httpx
starlette==0.27.0
starlette==0.38.6
# via
# -c requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
# -c requirements/../../../packages/service-library/requirements/../../../packages/models-library/requirements/../../../requirements/constraints.txt
Expand All @@ -241,12 +246,11 @@ typing-extensions==4.10.0
# via
# aiodebug
# aiodocker
# anyio
# fastapi
# faststream
# pydantic
# pydantic-core
# typer
# uvicorn
uvicorn==0.29.0
# via
# -r requirements/../../../packages/service-library/requirements/_fastapi.in
Expand Down
14 changes: 0 additions & 14 deletions services/invitations/requirements/_test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@ coverage==7.6.1
# via
# -r requirements/_test.in
# pytest-cov
exceptiongroup==1.2.0
# via
# -c requirements/_base.txt
# anyio
# hypothesis
# pytest
faker==27.0.0
# via -r requirements/_test.in
h11==0.14.0
Expand Down Expand Up @@ -90,11 +84,3 @@ sortedcontainers==2.4.0
# via hypothesis
termcolor==2.4.0
# via pytest-sugar
tomli==2.0.1
# via
# coverage
# pytest
typing-extensions==4.10.0
# via
# -c requirements/_base.txt
# anyio
11 changes: 0 additions & 11 deletions services/invitations/requirements/_tools.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,22 +70,11 @@ ruff==0.6.1
# via -r requirements/../../../requirements/devenv.txt
setuptools==73.0.1
# via pip-tools
tomli==2.0.1
# via
# -c requirements/_test.txt
# black
# build
# mypy
# pip-tools
# pylint
tomlkit==0.13.2
# via pylint
typing-extensions==4.10.0
# via
# -c requirements/_base.txt
# -c requirements/_test.txt
# astroid
# black
# mypy
virtualenv==20.26.3
# via pre-commit
Expand Down
16 changes: 8 additions & 8 deletions services/invitations/src/simcore_service_invitations/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from cryptography.fernet import Fernet
from models_library.emails import LowerCaseEmailStr
from models_library.invitations import InvitationContent, InvitationInputs
from pydantic import EmailStr, HttpUrl, ValidationError, parse_obj_as
from pydantic import EmailStr, HttpUrl, TypeAdapter, ValidationError
from rich.console import Console
from servicelib.utils_secrets import generate_password
from settings_library.utils_cli import (
Expand Down Expand Up @@ -96,19 +96,19 @@ def invite(
ctx: typer.Context,
email: str = typer.Argument(
...,
callback=lambda v: parse_obj_as(LowerCaseEmailStr, v),
callback=lambda v: TypeAdapter(LowerCaseEmailStr).validate_python(v),
help="Custom invitation for a given guest",
),
issuer: str = typer.Option(
..., help=InvitationInputs.__fields__["issuer"].field_info.description
..., help=InvitationInputs.model_fields["issuer"].description
),
trial_account_days: int = typer.Option(
None,
help=InvitationInputs.__fields__["trial_account_days"].field_info.description,
help=InvitationInputs.model_fields["trial_account_days"].description,
),
product: str = typer.Option(
None,
help=InvitationInputs.__fields__["product"].field_info.description,
help=InvitationInputs.model_fields["product"].description,
),
):
"""Creates an invitation link for user with 'email' and issued by 'issuer'"""
Expand All @@ -117,7 +117,7 @@ def invite(

invitation_data = InvitationInputs(
issuer=issuer,
guest=parse_obj_as(EmailStr, email),
guest=TypeAdapter(EmailStr).validate_python(email),
trial_account_days=trial_account_days,
extra_credits_in_usd=None,
product=product,
Expand All @@ -142,14 +142,14 @@ def extract(ctx: typer.Context, invitation_url: str):
try:
invitation: InvitationContent = extract_invitation_content(
invitation_code=extract_invitation_code_from_query(
parse_obj_as(HttpUrl, invitation_url)
TypeAdapter(HttpUrl).validate_python(invitation_url)
),
secret_key=settings.INVITATIONS_SECRET_KEY.get_secret_value().encode(),
default_product=settings.INVITATIONS_DEFAULT_PRODUCT,
)
assert invitation.product is not None # nosec

print(invitation.json(indent=1)) # noqa: T201
print(invitation.model_dump_json(indent=1)) # noqa: T201

except (InvalidInvitationCodeError, ValidationError):
_err_console.print("[bold red]Invalid code[/bold red]")
Expand Down
18 changes: 13 additions & 5 deletions services/invitations/src/simcore_service_invitations/core/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
from functools import cached_property

from models_library.products import ProductName
from pydantic import Field, HttpUrl, PositiveInt, SecretStr, validator
from pydantic import (
AliasChoices,
Field,
HttpUrl,
PositiveInt,
SecretStr,
field_validator,
)
from settings_library.base import BaseCustomSettings
from settings_library.basic_types import BuildTargetEnum, LogLevel, VersionTag
from settings_library.utils_logging import MixinLoggingSettings
Expand Down Expand Up @@ -38,22 +45,23 @@ class _BaseApplicationSettings(BaseCustomSettings, MixinLoggingSettings):
# RUNTIME -----------------------------------------------------------

INVITATIONS_LOGLEVEL: LogLevel = Field(
default=LogLevel.INFO, env=["INVITATIONS_LOGLEVEL", "LOG_LEVEL", "LOGLEVEL"]
default=LogLevel.INFO,
validation_alias=AliasChoices("INVITATIONS_LOGLEVEL", "LOG_LEVEL", "LOGLEVEL"),
)
INVITATIONS_LOG_FORMAT_LOCAL_DEV_ENABLED: bool = Field(
default=False,
env=[
validation_alias=AliasChoices(
"INVITATIONS_LOG_FORMAT_LOCAL_DEV_ENABLED",
"LOG_FORMAT_LOCAL_DEV_ENABLED",
],
),
description="Enables local development log format. WARNING: make sure it is disabled if you want to have structured logs!",
)

@cached_property
def LOG_LEVEL(self):
return self.INVITATIONS_LOGLEVEL

@validator("INVITATIONS_LOGLEVEL")
@field_validator("INVITATIONS_LOGLEVEL")
@classmethod
def valid_log_level(cls, value: str) -> str:
return cls.validate_log_level(value)
Expand Down
56 changes: 21 additions & 35 deletions services/invitations/src/simcore_service_invitations/services/invitations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,22 @@
import base64
import binascii
import logging
from typing import Any, ClassVar, cast
from urllib import parse

from common_library.pydantic_networks_extension import HttpUrlLegacy
from cryptography.fernet import Fernet, InvalidToken
from models_library.invitations import InvitationContent, InvitationInputs
from models_library.products import ProductName
from pydantic import HttpUrl, ValidationError, parse_obj_as
from pydantic import ConfigDict, HttpUrl, TypeAdapter, ValidationError
from starlette.datastructures import URL

_logger = logging.getLogger(__name__)


def _to_initial(v: str):
return v[0]


class InvalidInvitationCodeError(Exception):
...

Expand All @@ -23,9 +27,9 @@ class _ContentWithShortNames(InvitationContent):
@classmethod
def serialize(cls, model_obj: InvitationContent) -> str:
"""Exports to json using *short* aliases and values in order to produce shorter codes"""
model_w_short_aliases_json: str = cls.construct(
**model_obj.dict(exclude_unset=True)
).json(exclude_unset=True, by_alias=True)
model_w_short_aliases_json: str = cls.model_construct(
**model_obj.model_dump(exclude_unset=True)
).model_dump_json(exclude_unset=True, by_alias=True)
# NOTE: json arguments try to minimize the amount of data
# serialized. The CONS is that it relies on models in the code
# that might change over time. This might lead to some datasets in codes
Expand All @@ -35,36 +39,18 @@ def serialize(cls, model_obj: InvitationContent) -> str:
@classmethod
def deserialize(cls, raw_json: str) -> InvitationContent:
"""Parses a json string and returns InvitationContent model"""
model_w_short_aliases = cls.parse_raw(raw_json)
return InvitationContent.construct(
**model_w_short_aliases.dict(exclude_unset=True)
model_w_short_aliases = cls.model_validate_json(raw_json)
return InvitationContent.model_construct(
**model_w_short_aliases.model_dump(exclude_unset=True)
)

class Config:
allow_population_by_field_name = True # NOTE: can parse using field names
allow_mutation = False
anystr_strip_whitespace = True
model_config = ConfigDict(
# NOTE: Can export with alias: short aliases to minimize the size of serialization artifact
fields: ClassVar[dict[str, Any]] = {
"issuer": {
"alias": "i",
},
"guest": {
"alias": "g",
},
"trial_account_days": {
"alias": "t",
},
"extra_credits_in_usd": {
"alias": "e",
},
"product": {
"alias": "p",
},
"created": {
"alias": "c",
},
}
alias_generator=_to_initial,
Copy link
Contributor Author

@giancarloromeo giancarloromeo Oct 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the alias_generator removes the need of the code above.

Copy link
Member

@pcrespov pcrespov Oct 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation you did of _to_initial has the risk of producing inadvertently duplicate aliases as we extend the number of fields. An explicit mapping, as we have now, makes a mistake more obvious but I would also add an assert in the code.

I propose to either do an explicit map as an implementation or your first-letter implementation of _to_initial generator but in both cases make sure that there are no repetitions as

assert not [item for item, count in Counter([field.alias for field in _ContentWithShortNames.__fields__.values()]).items() if count > 1]  #nosec

Copy link
Contributor Author

@giancarloromeo giancarloromeo Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the assert statement as test to catch alias repetition early and have a clean separation between validation and model logic.

populate_by_name=True, # NOTE: can parse using field names
frozen=True,
str_strip_whitespace=True,
)


#
Expand All @@ -79,9 +65,9 @@ def _build_link(
r = URL("/registration").include_query_params(invitation=code_url_safe)

# Adds query to fragment
base_url = f"{base_url.rstrip('/')}/"
base_url = f"{base_url}/"
url = URL(base_url).replace(fragment=f"{r}")
return cast(HttpUrl, parse_obj_as(HttpUrl, f"{url}"))
return TypeAdapter(HttpUrlLegacy).validate_python(f"{url}")


def _fernet_encrypt_as_urlsafe_code(
Expand Down Expand Up @@ -124,7 +110,7 @@ def create_invitation_link_and_content(
code = _create_invitation_code(content, secret_key)
# Adds message as the invitation in query
link = _build_link(
base_url=base_url,
base_url=f"{base_url}",
code_url_safe=code.decode(),
)
return link, content
Expand Down
2 changes: 1 addition & 1 deletion services/invitations/tests/unit/api/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def client(app_environment: EnvVarsDict) -> Iterator[TestClient]:
print(f"app_environment={json.dumps(app_environment)}")

app = create_app()
print("settings:\n", app.state.settings.json(indent=1))
print("settings:\n", app.state.settings.model_dump_json(indent=1))
with TestClient(app, base_url="http://testserver.test") as client:
yield client

Expand Down
Loading
Loading