Support for MongoDB ObjectID in Pydantic Models

[SkandaPrasad-S]

Initial Checks

• I have searched Google & GitHub for similar requests and couldn't find anything
• I have read and followed the docs and still think this feature is missing

Description

I am using Pydantic in conjunction with FastAPI and MongoDB, and I would like to request support for MongoDB ObjectID as one of the field types in Pydantic models.

Background:
Currently, Pydantic supports various field types, but there isn't a specific type for MongoDB ObjectID. MongoDB ObjectID is commonly used in FastAPI applications with MongoDB databases, and having native support for it in Pydantic would greatly enhance the validation capabilities.

Suggested Solution:
I propose adding a dedicated type for MongoDB ObjectID in Pydantic, similar to how EmailStr and other specialized types work. This could be achieved by introducing a type like MongoObjectId or extending the existing PyObjectId to handle MongoDB ObjectID validation seamlessly.

from pydantic import BaseModel
from pydantic_mongo import MongoObjectId

class MyClass(BaseModel):
    my_id: MongoObjectId

Additional Context:
MongoDB ObjectID validation is a common requirement in FastAPI applications, especially when dealing with MongoDB databases. Adding native support in Pydantic would simplify the code and improve the developer experience.

I would love it if we could do something like the below with FASTAPI that works seamlessly in generating the OpenAPI schema as well.
This is because all the workarounds for the above issue face OpenAPI schema issues.

@router.get("/get_id/{my_id}")
def get_goal_by_id(my_class: MyClass ):
    return my_class.my_id

Affected Components

• Compatibility between releases
• Data validation/parsing
• Data serialization - .model_dump() and .model_dump_json()
• JSON Schema
• Dataclasses
• Model Config
• Field Types - adding or changing a particular data type
• Function validation decorator
• Generic Models
• Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
• Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

[Ale-Cas]

@SkandaPrasad-S if you're using pydantic, FastAPI and pymongo, you can validate the _id field using bson.ObjectId and arbitrary_types_allowed = True, I attached an example with pydantic v1 and the same can be achieved in v2 using ConfigDict.

Anyway, I agree with you that ideally setting arbitrary_types_allowed should not be needed and there should be a standard validator.

[sydney-runkle]

@SkandaPrasad-S,

I think this is a valuable feature request, but belongs on the pydantic-extra-types repo. I'll move this request there 👍.

I might also suggest using Beanie, a helpful package for integrating usage of pydantic with mongodb: https://beanie-odm.dev/api-documentation/fields/#pydanticobjectid

[sydney-runkle]

I'd also be open to closing this issue - seems like a specific enough request that corresponds with Beanie's functionality, and could also just reside in user code.

[SkandaPrasad-S]

I would love to have this in pydnatic instead of beanie. I understand that it probably might make sense @sydney-runkle, but I would rather have it in pydantic as one of the inbuilt types, especially with the increase in FARM stack usage.

from typing import Any

from bson import ObjectId
from pydantic_core import core_schema


class PyObjectId(str):
    """To create a pydantic Object that validates bson ObjectID"""

    @classmethod
    def __get_pydantic_core_schema__(cls, _source_type: Any, _handler: Any) -> core_schema.CoreSchema:
        return core_schema.json_or_python_schema(
            json_schema=core_schema.str_schema(),
            python_schema=core_schema.union_schema(
                [
                    core_schema.is_instance_schema(ObjectId),
                    core_schema.chain_schema(
                        [
                            core_schema.str_schema(),
                            core_schema.no_info_plain_validator_function(cls.validate),
                        ]
                    ),
                ]
            ),
            serialization=core_schema.plain_serializer_function_ser_schema(lambda x: str(x)),
        )

    @classmethod
    def validate(cls, value) -> ObjectId:
        if not ObjectId.is_valid(value):
            raise ValueError("Invalid ObjectId")

        return ObjectId(value)

I would rather not have this in every single MongoDB app that I make with FastAPI.
What do you think?

[SkandaPrasad-S]

Also
@Ale-Cas I ttried the exact same code you have

class SingleGoal(GoalRecordStored):
    """Goal Record Model with ID."""
    from bson import ObjectId

    id: ObjectId = Field(
        alias="_id",
        description="Unique identifier in mongo db"
    )
    
    class Config:
        arbitary_types_allowed = True

But I ended up with the below error

>>> models.goals import SingleGoal
  File "<stdin>", line 1
    models.goals import SingleGoal
                 ^^^^^^
SyntaxError: invalid syntax
>>> from models.goals import SingleGoal 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "E:\Perfection\GoalsRankedBackEnd\models\goals.py", line 48, in <module>
    class SingleGoal(GoalRecordStored):
  File "E:\Perfection\GoalsRankedBackEnd\myvenv\Lib\site-packages\pydantic\_internal\_model_construction.py", line 92, in __new__
    private_attributes = inspect_namespace(
                         ^^^^^^^^^^^^^^^^^^
  File "E:\Perfection\GoalsRankedBackEnd\myvenv\Lib\site-packages\pydantic\_internal\_model_construction.py", line 372, in inspect_namespace
    raise PydanticUserError(
pydantic.errors.PydanticUserError: A non-annotated attribute was detected: `ObjectId = <class 'bson.objectid.ObjectId'>`. All model fields require a type annotation; if `ObjectId` is not meant to be a field, you may be able to resolve this error by annotating it as a `ClassVar` or updating `model_config['ignored_types']`.

For further information visit https://errors.pydantic.dev/2.5/u/model-field-missing-annotation

[Ale-Cas]

@SkandaPrasad-S I agree with you that having a custom type for Mongo ObjectId would be a good addition.
Are you going to open the PR yourself? if not, I would, since I'm interested in having this type as well.

Anyway, there are some issues with the snippet you provided:

1. there's a typo, it's not arbitary_types_allowed but arbitrary_types_allowed
2. from the error logs it looks like you're using pydantic v2.5, but as I mentioned in my snippet that was meant to work with pydantic v1, since from v2 the Config class has been replaced with ConfigDict

If you're working with pydantic v2, try this instead:

from bson import ObjectId
from pydantic import BaseModel, Field, ConfigDict

class SingleGoal(BaseModel):
    """Goal Record Model with ID."""
    
    model_config = ConfigDict(arbitrary_types_allowed=True)

    id: ObjectId = Field(
        alias="_id",
        description="Unique identifier in mongo db"
    )

I found this "workaround" to work very well in projects where you use pydantic and pymongo, but not beanie.

[SkandaPrasad-S]

Ahh interesting, I see why it did not work @Ale-Cas. Thank you so much, makes more sense now.

I am going to raise the PR myself if that would not be a problem for this type if that is okay! @sydney-runkle ?

[SkandaPrasad-S]

Created a PR At #151

[alex-pythonista]

I would love to have this in pydnatic instead of beanie. I understand that it probably might make sense @sydney-runkle, but I would rather have it in pydantic as one of the inbuilt types, especially with the increase in FARM stack usage.

from typing import Any

from bson import ObjectId
from pydantic_core import core_schema


class PyObjectId(str):
    """To create a pydantic Object that validates bson ObjectID"""

    @classmethod
    def __get_pydantic_core_schema__(cls, _source_type: Any, _handler: Any) -> core_schema.CoreSchema:
        return core_schema.json_or_python_schema(
            json_schema=core_schema.str_schema(),
            python_schema=core_schema.union_schema(
                [
                    core_schema.is_instance_schema(ObjectId),
                    core_schema.chain_schema(
                        [
                            core_schema.str_schema(),
                            core_schema.no_info_plain_validator_function(cls.validate),
                        ]
                    ),
                ]
            ),
            serialization=core_schema.plain_serializer_function_ser_schema(lambda x: str(x)),
        )

    @classmethod
    def validate(cls, value) -> ObjectId:
        if not ObjectId.is_valid(value):
            raise ValueError("Invalid ObjectId")

        return ObjectId(value)

I would rather not have this in every single MongoDB app that I make with FastAPI. What do you think?

This is a beautiful piece of work. Thank you @SkandaPrasad-S

[zifo-10]

You can use the pyobjectID package to deal with ObjectID.
package can be used in Pydantic models , Here are examples of how to implement it:

from pydantic import BaseModel
from pyobjectID import PyObjectId, MongoObjectId

class User(BaseModel):
    id: PyObjectId  # Automatically validates and converts to ObjectId
    name: str
    email: str

class GetFromMongo(BaseModel):
    id: MongoObjectId  # Automatically validates and converts to string

• In the User model, the id field will be validated as a valid ObjectId, and any string assigned to it will be converted to an ObjectId.
• In the GetFromMongo model, the id field will be validated as a valid ObjectId and will be converted to a string.

You can perform basic operations such as generating, validating, and converting ObjectIds:

from pyobjectID import (generate, PyObjectId, 
                        MongoObjectId, is_valid)

# Generate a new ObjectId
new_id = generate()
print(f"Generated ObjectId: {new_id}")

# Validate an ObjectId
valid = is_valid(new_id)
print(f"Is the ObjectId valid? {valid}")

# Convert a string to a PyObjectId
py_object_id = PyObjectId.to_object(new_id)
print(f"PyObjectId: {py_object_id}")

# Convert a PyObjectId to a string
mongo_object_id = MongoObjectId.to_string(py_object_id)
print(f"MongoObjectId: {mongo_object_id}")

You can install the package: pip install pyobjectid