What is the purpose of DandiBaseModel.unvalidated?

[candleindark]

What is the purpose of DandiBaseModel.unvalidated?

dandi-schema/dandischema/models.py

Lines 398 to 414 in 68f4d1e

	def unvalidated(__pydantic_cls__: Type[M], **data: Any) -> M:
	"""Allow model to be returned without validation"""
	for name, field in __pydantic_cls__.__fields__.items():
	try:
	data[name]
	except KeyError:
	# if field.required:
	# value = None
	if field.default_factory is not None:
	value = field.default_factory()
	elif field.default is None:
	# deepcopy is quite slow on None
	value = None
	else:
	value = deepcopy(field.default)
	data[name] = value
	return __pydantic_cls__.construct(**data)

It seems to me that it is for creating a DandiBaseModel model object (or an object of a subclass of DandiBaseModel) without going through the validation process provided by Pydantic but keeping the default value either provided by the default_factory or default.

If my guess is correct, then there is a problem with this method.

1. This function is not needed at all.
   1. BaseModel.construct in the Pydantic V1 (as of the latest version but not sure about earlier version) fulfills the purpose above.
   2. BaseModel.model_construct in Pydantic V2 fulfills the purpose above as documented in https://docs.pydantic.dev/latest/api/base_model/#pydantic.main.BaseModel.model_construct.
2. This function has a problem of handling a required field that has no default specified. For example, the field of x in the follow model is required and doesn't have a default. However, the logic of the unvalidated method would set the x attribute of the produced model instance to None in Pydantic V1 and PydanticUndefined in Pydantic V2.

# For pydantic v1
from pydantic import BaseModel


class Model(BaseModel):
    x: int


xf = Model.__fields__["x"]

print(f"xf: {xf!r}")
print(f"xf.default: {xf.default}")
print(f"xf.default_factory: {xf.default_factory}")

# for pydantic V2
from pydantic import BaseModel


class Model(BaseModel):
    x: int


xf = Model.model_fields["x"]

print(f"xf: {xf!r}")
print(f"xf.default: {xf.default}")
print(f"xf.default_factory: {xf.default_factory}")

[yarikoptic]

attn @satra and @jwodder who would know better. FWIW unvalidated was added long (3 years) ago in a41ecab , so might indeed be some kind of a relic and could be replaced with built-in pydantic functionality now: @candleindark you can try and see if any tests would break to make judgement on that.

[satra]

the place where it seems to be used is the next classmethod and as @candleindark noted, the only difference from construct which it calls is how undefined values are handled. it's quite possible this is unnecessary anymore and was only a temporary state to aid debugging. i would suggest removing it and the other function and seeing if it breaks anything elsewhere, and after removing calling model_construct to see where it breaks (if any). we do have some types of validation routines in the code.

[candleindark]

In the context of Pydantic V1, replacing the use of unvalidated with construct failed only two tests related to the existence of unvalidated in the model object, test_dandiset and test_asset.

dandi-schema/dandischema/tests/test_models.py

Lines 43 to 48 in 68f4d1e

	def test_dandiset() -> None:
	assert Dandiset.unvalidated()
	def test_asset() -> None:
	assert Asset.unvalidated()

.

Defining unvalidated as the following passed all the tests in the project.

    @classmethod
    def unvalidated(__pydantic_cls__: Type[M], **data: Any) -> M:
        """Allow model to be returned without validation"""
        return __pydantic_cls__.construct(**data)

Additionally, construct in Pydantic V1 and model_construct in Pydantic V2 exhibit the behavior that fulfill the purpose mentioned in the first post of this discussion.

# Behavior illustration in Pydantic V1

from pydantic import BaseModel, Field


class Model(BaseModel):
    no_default: int
    with_default: int = 1
    with_default_factory: int = Field(default_factory=lambda: 42)


for field_name, field_info in Model.__fields__.items():
    print(f"{field_name}: {field_info!r}")
    print(f"required: {field_info.required}")
    print(f"default: {field_info.default}")
    print(f"default_factory: {field_info.default_factory}")
    print("\n========================================")


m = Model.construct()

# Note the `no_default` attribute is not present in the model object
assert m.dict() == {'with_default': 1, 'with_default_factory': 42}

# Behavior illustration in Pydantic V2

from pydantic import BaseModel, Field


class Model(BaseModel):
    no_default: int
    with_default: int = 1
    with_default_factory: int = Field(default_factory=lambda: 42)


for field_name, field_info in Model.model_fields.items():
    print(f"{field_name}: {field_info!r}")
    print(f"required: {field_info.is_required()}")
    print(f"default: {field_info.default}")
    print(f"default_factory: {field_info.default_factory}")
    print("\n========================================")


m = Model.model_construct()

# Note the `no_default` attribute is not present in the model object
assert m.model_dump() == {'with_default': 1, 'with_default_factory': 42}

If the purpose mentioned in the first post is indeed the intend purpose of unvalidated, then unvalidated indeed should be removed.

[jwodder]

As far as I understood it, unvalidated() is for cheating Pydantic and constructing an object that doesn't have all the declared fields, for use when you're not provided with all the necessary fields but you want an object anyway. I believe (see here) this is what enables Dandi API to accept & serve invalid metadata, something I previously railed against here.

I fully support eliminating this method and making it impossible to construct invalid metadata objects. If we want to be able to accept user input that lacks fields, then we should have a model that actually declares those fields optional.

[satra]

thanks for the reminder @jwodder.

just pinging @djarecka here so we keep this in mind as we implement a linkml version (where we will also use mixins rather than inheritance). i am also thinking of requirements as a separate layer defined by use cases/services.

[candleindark]

I fully support eliminating this method and making it impossible to construct invalid metadata objects. If we want to be able to accept user input that lacks fields, then we should have a model that actually declares those fields optional.

I can't agree more.

[candleindark]

FTR: DandiBaseModel.unvalidated will be removed as part of #203.

[yarikoptic]

left my 1c: #203 (comment)