Msgpack now working

Equality etc fixed

Author: nickyoung-github

README.md

@@ -2,11 +2,12 @@
 
 # Table of Contents
 1. [Overview](#Overview)
-2. [No Copy](#No-Copy)
-3. [Supported Types](#Supported-Types)
-4. [Inheritance](#Inheritance)
-5. [Msgpack](#Msgpack)
-6. [Generated Code](#Generated-Code)
+2. [Why Not Protobufs ?](#Why-Not-Protobufs)
+3. [No Copy](#No-Copy)
+4. [Supported Types](#Supported-Types)
+5. [Inheritance](#Inheritance)
+6. [Msgpack](#Msgpack)
+7. [Generated Code](#Generated-Code)
 
 
 ## Overview
@@ -26,7 +27,8 @@ and [pybind11](https://pybind11.readthedocs.io/en/stable/index.html) code for bi
 corresponding .cpp file.
 
 The intended use of this package is for defining behaviour-less data classes, to be shared between python and C++. E.g.,
-a common object model for financial modelling.
+a common object model for financial modelling. Furthr, we want idiomatic classes for each language, not mutants like
+Protobuf-generated python classes.
 
 Note that the typcal python developer experience is now somewhat changed, in that it's necessary to build/install
 the project. I personally use JetBrains CLion, in place of PyCharm for such projects.
@@ -64,17 +66,37 @@ You can create an instance of the pybind class from your original using `get_pyb
     generated = get_pybind_value(orig)
 
 
+# Why Not Protobufs?
+
+A very good question. Protobufs are frankly a PITA to use: they have poor to no variant support, the generated
+code is ugly and idiosyncratic, they're large and painful to copy around etc.
+
+AVRO is more friendly but generates python classes dynamically, which confuses IDEs like Pycharm. I do think a good
+solution is something like [pydantic_avro](https://github.com/godatadriven/pydantic-avro/tree/main/src/pydantic_avro)
+where one can define the classes using pydantic, generate the AVRO schema and then the generateed C++ etc. I might
+well try and converge this project with that approach.
+
+I was inspired to some degree by this [blog](https://mikeloomisgg.github.io/2019-07-02-making-a-serialization-library/).
+
+
 ## No Copy
 
-What if you would like a single representation of the data, shared between C++ and python. Then fear not,
-`BaseModelNoCopy` and `dataclass` are your friends!
+One annoyance of multi-language representations of data objects is that you often end up copying data around where
+you'd prefer to share a single copy. This is the raison d'etre for Protobufs and its ilk. In this project I've created
+implementations of `BaseModel` and `dataclass` which allow python to use the underlying C++ data representation, rather
+than holding its own copy.
 
-Deriving from this `BaseModelNoCopy` will give you equivalent functionality of as pydantic's `BaseModel`. The
+Deriving from this `BaseModel` will give you equivalent functionality of as pydantic's `BaseModel`. The
 annotations are re-written using `computed_field`, with property getters and setters operating on the generated pybind
-class, which is instantiated behind the scenes in `init`
+class, which is instantiated behind the scenes in `init`. Note that this will make some operations (especially those
+that access __dict__) less efficient. I've also plumbed the computed fields into the JSON schema, so these objects can
+be used with [FastAPI](https://fastapi.tiangolo.com).
 
 `dataclass` works similarly, adding properties to the dataclass, so that the exisitng get and set functionality works
-seamless in accessing the generated pybind class (also set via a shimmed `init`)
+seamless in accessing the generated pybind class (also set via a shimmed `init`).
+
+Using regular `dataclass` or `BaseModel` as members of classes defined with the pydantic_bind versions is very
+inefficient and not recommended.
 
 
 ## Supported Types
@@ -92,11 +114,12 @@ The following python -> C++ mappings are supported (there are likely others I sh
 - pydantic.BaseModel --> struct
 - pydantic_bind.BaseModelNoCopy --> struct
 - dataclass --> struct
+- Enum -> enum
 
 ## Inheritance
 
 I have tested single inheritance (see [Generated Code](#Generated-code)). Multiple inheritance may work ... or it
-may not. I'd generally advise against using it for data classes
+may not. I'd generally advise against using it for data classes.
 
 
 ## Msgpack
@@ -109,21 +132,28 @@ project with my rather rudimentary cmake skillz!) Changes include:
 - Fixing includes
 - Support for std::optional
 - Support for std::variant
+- Support for enums
+
+A likely future enhancement will be to use [cereal](https://github.com/USCiLab/cereal) and add a mgspack adaptor.
+However, I haven't quite worked out how to do that yet.
 
 
 ## Generated Code
 
-Code is generated into a directory structure underneath `<top level>/generated`
+Code is generated into a directory structure underneath `<top level>/generated`.
 
-Headers are installed to `<top level>/include`
+Headers are installed to `<top level>/include`.
 
-Compiled pybind modules are installed into `<original module path>/__pybind__`
+Compiled pybind modules are installed into `<original module path>/__pybind__`.
 
-For C++ usage, you need only the headers, the compiled code is for pybind/python usage only
+For C++ usage, you need only the headers, the compiled code is for pybind/python usage only.
 
 For the example below, `common_object_model/common_object_model/v1/common/__pybind__/foo.cpython-311-darwin.so` will
 be installed (obviously with corresponding qualifiers for Linux/Windows). `get_pybind_value()` searches this
-directory
+directory.
+
+Imports/includes should work seamlessly (the python import scheme will be copied). I have tested this but not
+completely rigorously.
 
 *common_object_model/common_object_model/v1/common/foo.py:*
 
@@ -290,18 +320,24 @@ will generate the following files:
         py::class_<DCFoo>(m, "DCFoo")
             .def(py::init<>())
             .def(py::init<std::optional<std::string>, int>(), py::arg("my_string"), py::arg("my_int"))
+            .def("to_msg_pack", &DCFoo::to_msg_pack)
+            .def_static("from_msg_pack", &DCFoo::from_msg_pack<Baz>)
             .def_readwrite("my_string", &DCFoo::my_string)
             .def_readwrite("my_int", &DCFoo::my_int);
 
         py::class_<Foo>(m, "Foo")
             .def(py::init<bool, Weekday>(), py::arg("my_bool")=true, py::arg("my_day")=SUNDAY)
+            .def("to_msg_pack", &Foo::to_msg_pack)
+            .def_static("from_msg_pack", &Foo::from_msg_pack<Baz>)
             .def_readwrite("my_bool", &Foo::my_bool)
             .def_readwrite("my_day", &Foo::my_day);
 
         py::class_<Bar>(m, "Bar")
             .def(py::init<>())
             .def(py::init<std::string, bool, Weekday, int, std::optional<std::string>>(), py::arg("my_string"), py::arg("my_bool")=true,
                 py::arg("my_day")=SUNDAY, py::arg("my_int")=123, py::arg("my_optional_string")=std::nullopt)
+            .def("to_msg_pack", &Bazr:to_msg_pack)
+            .def_static("from_msg_pack", &Bar::from_msg_pack<Baz>)
             .def_readwrite("my_string", &Bar::my_string)
             .def_readwrite("my_int", &Bar::my_int)
             .def_readwrite("my_optional_string", &Bar::my_optional_string);
@@ -310,6 +346,8 @@ will generate the following files:
             .def(py::init<>())
             .def(py::init<DCFoo, Foo, std::chrono::system_clock::time_point, std::variant<std::string, double>>(), py::arg("my_dc_foo"),
                 py::arg("my_foo"), py::arg("my_date"), py::arg("my_variant")=123.0)
+            .def("to_msg_pack", &Baz::to_msg_pack)
+            .def_static("from_msg_pack", &Baz::from_msg_pack<Baz>)
             .def_readwrite("my_dc_foo", &Baz::my_dc_foo)
             .def_readwrite("my_foo", &Baz::my_foo)
             .def_readwrite("my_date", &Baz::my_date)

pydantic_bind/__init__.py

@@ -1,7 +1,7 @@
-from .base import BaseModelNoCopy, dataclass, get_pybind_type, get_pybind_value
+from .base import BaseModel, dataclass, get_pybind_type, get_pybind_value
 
 __all__ = (
-    BaseModelNoCopy,
+    BaseModel,
     dataclass,
     get_pybind_type,
     get_pybind_value

pydantic_bind/base.py

@@ -2,7 +2,7 @@
 from enum import Enum, EnumType
 from functools import cache, wraps
 from importlib import import_module
-from pydantic import BaseModel as BaseModel, ConfigDict, computed_field
+from pydantic import BaseModel as PydanticBaseModel, ConfigDict, computed_field
 from pydantic.fields import ComputedFieldInfo, FieldInfo
 from pydantic.json_schema import GenerateJsonSchema
 from pydantic._internal._config import ConfigWrapper
@@ -13,27 +13,24 @@
 from pydantic_core import PydanticUndefined
 import sys
 from types import UnionType
-from typing import Any, Dict, List, Type, Optional, Union, cast, get_args, get_origin
+from typing import Any, Dict, List, Optional, Sequence, Type, Union, cast, get_args, get_origin
 
 
 class UnconvertableValue(Exception):
     pass
 
 
-class __IBaseModelNoCopy:
-    pass
-
-
-def field_info_iter(model_class: ModelMetaclass):
+def field_info_iter(model_class):
     if is_dataclass(model_class):
         for field_name, field in model_class.__dataclass_fields__.items():
             yield field_name, field.type, field.default
-    elif issubclass(model_class, BaseModelNoCopy):
-        for field_name, field in model_class.__pydantic_decorators__.computed_fields.items():
-            yield field_name, field.info.return_type, field.info.default
-    else:
-        for field_name, field in model_class.model_fields.items():
-            yield field_name, field.annotation, field.default
+    elif issubclass(model_class, PydanticBaseModel):
+        if hasattr(model_class, "__has_pybind_impl__"):
+            for field_name, field in model_class.__pydantic_decorators__.computed_fields.items():
+                yield field_name, field.info.return_type, field.info.default
+            else:
+                for field_name, field in model_class.model_fields.items():
+                    yield field_name, field.annotation, field.default
 
 
 @cache
@@ -69,41 +66,45 @@ def get_pybind_value(obj):
 def _get_pybind_value(obj, default_to_self: bool = True):
     if isinstance(obj, Enum):
         return get_pybind_type(type(obj)).__entries[obj.name][0]
-    elif is_dataclass(obj):
-        return get_pybind_type(type(obj))(**{name: _get_pybind_value(getattr(obj, name))
-                                             for name in obj.__dataclass_fields__.keys()})
-    elif isinstance(obj, __IBaseModelNoCopy):
-        return get_pybind_type(type(obj))(**{name: _get_pybind_value(getattr(obj, name))
-                                             for name in obj.model_computed_fields.keys()})
-    elif isinstance(obj, BaseModel):
-        return get_pybind_type(type(obj))(**{name: _get_pybind_value(getattr(obj, name))
-                                             for name in obj.model_fields.keys()})
+    elif is_dataclass(obj) or isinstance(obj, PydanticBaseModel):
+        typ = type(obj)
+        pybind_type = get_pybind_type(typ)
+        name_iter = (name for name, _, _ in field_info_iter(typ))
+
+        if hasattr(typ, "__has_pybind_impl__"):
+            return pybind_type(**{name: getattr(obj.pybind_impl, name) for name in name_iter})
+        else:
+            return pybind_type(**{name: _get_pybind_value(getattr(obj, name)) for name in name_iter})
     elif default_to_self:
         return obj
     else:
-        raise UnconvertableValue("Only dataclasses and pydantic classes supported")
+        raise UnconvertableValue("Only builtins, dataclasses and pydantic classes supported")
 
 
 def from_pybind_value(value, typ: Type):
     origin = get_origin(typ)
     args = get_args(typ)
-    is_dc = is_dataclass(typ)
 
     if origin is Optional:
         typ = args[0]
-    elif origin in (Union, UnionType):
+        args = get_args(typ)
+
+    if origin in (Union, UnionType):
         typ = next(a for a in args if a.__name__ == type(value).__name__)
 
+    is_dc_or_pydantic = is_dataclass(typ) or issubclass(typ, PydanticBaseModel)
+
     if issubclass(typ, Enum):
         return typ[value.name]
-    elif issubclass(typ, __IBaseModelNoCopy) or (is_dc and hasattr(typ, "__no_copy__")):
-        return typ(__pybind_impl__=value)
-    elif is_dc or issubclass(typ, BaseModel):
-        # This is quite inefficient
-        kwargs = {}
-        for field_name, field_type, _ in field_info_iter(typ):
-            kwargs[field_name] = from_pybind_value(getattr(value, field_name), field_type)
-        return typ(**kwargs)
+    elif is_dc_or_pydantic:
+        if hasattr(typ, "__has_pybind_impl__"):
+            return typ(__pybind_impl__=value)
+        else:
+            # This is quite inefficient
+            kwargs = {}
+            for field_name, field_type, _ in field_info_iter(typ):
+                kwargs[field_name] = from_pybind_value(getattr(value, field_name), field_type)
+            return typ(**kwargs)
     else:
         return value
 
@@ -194,6 +195,7 @@ def __new__(
                 namespace[name] = prop
 
         cls = cast(ModelMetaclass, super().__new__(mcs, cls_name, bases, namespace, **kwargs))
+        cls.__has_pybind_impl__ = True
         cls.__pydantic_decorators__.__annotations__["computed_fields"] = dict[str, Decorator[PropertyFieldInfo]]
         cls.__signature__ = ClassAttribute(
             '__signature__', generate_model_signature(cls.__init__, field_infos, config_wrapper)
@@ -226,7 +228,13 @@ def json_schema_extra(schema: Dict[str, Any], model_class: ModelMetaclassNoCopy)
         properties[alias] = field_schema
 
 
-class BaseModelNoCopy(BaseModel, __IBaseModelNoCopy, metaclass=ModelMetaclassNoCopy):
+def _from_msg_pack(cls, data: Sequence[int]):
+    typ = get_pybind_type(cls)
+    pybind_impl, _error_code = typ.from_msg_pack(data)
+    return cls(__pybind_impl__=pybind_impl)
+
+
+class BaseModel(PydanticBaseModel, metaclass=ModelMetaclassNoCopy):
     model_config = ConfigDict(json_schema_extra=json_schema_extra)
 
     @property
@@ -237,9 +245,14 @@ def model_computed_fields(self) -> dict[str, PropertyFieldInfo]:
     def pybind_impl(self):
         return self.__pybind_impl
 
-    def __init__(self, **kwargs):
-        super().__init__()
+    def to_msg_pack(self):
+        return self.__pybind_impl.to_msg_pack()
+
+    @classmethod
+    def from_msg_pack(cls, data: Sequence[int]):
+        return _from_msg_pack(cls, data)
 
+    def __init__(self, **kwargs):
         __pybind_impl__ = kwargs.pop("__pybind_impl__", None)
         if __pybind_impl__:
             self.__pybind_impl = __pybind_impl__
@@ -266,19 +279,45 @@ def __init__(self, **kwargs):
                 raise RuntimeError(f"Missing required fields: {missing_required}")
 
             pybind_type = get_pybind_type(type(self))
-            self.__pybind_impl = pybind_type(**kwargs)
+            object.__setattr__(self, "_BaseModel__pybind_impl", pybind_type(**kwargs))
+
+        super().__init__()
+
+    @property
+    def __dict__(self):
+        return {name: from_pybind_value(getattr(self, name), typ) for name, typ, _ in field_info_iter(type(self))}
+
+    @__dict__.setter
+    def __dict__(self, value: dict):
+        try:
+            object.__getattribute__(self, "_BaseModel__pybind_impl")
+            for name, value in value.items():
+                object.__setattr__(self, name, value)
+        except AttributeError:
+            self.__init__(**value)
 
 
 def __dataclass_init(init):
     @wraps(init)
     def wrapper(self, *args, __pybind_impl__=None, **kwargs):
-        self.pybind_impl = __pybind_impl__ or get_pybind_type(type(self))()
-        init(self, *args, **kwargs)
-        arse = True
+        if __pybind_impl__:
+            self.__pybind_impl = __pybind_impl__
+        else:
+            self.__pybind_impl = get_pybind_type(type(self))()
+            init(self, *args, **kwargs)
 
     return wrapper
 
 
+def to_msg_pack(self):
+    return self.pybind_impl.to_msg_pack()
+
+
+@classmethod
+def from_msg_pack(cls, data: Sequence[int]):
+    return _from_msg_pack(cls, data)
+
+
 def dataclass(cls=None, /, *, init=True, repr=True, eq=True, order=False,
               unsafe_hash=False, frozen=False, match_args=True,
               kw_only=False, slots=False, weakref_slot=False):
@@ -290,6 +329,10 @@ def dataclass(cls=None, /, *, init=True, repr=True, eq=True, order=False,
         setattr(cls, name, property(fget=_getter(name, field.type), fset=_setter(name, field.type)))
 
     ret.__init__ = __dataclass_init(ret.__init__)
-    ret.__no_copy__ = True
+    ret.__has_pybind_impl__ = True
+
+    ret.to_msg_pack = to_msg_pack
+    ret.from_msg_pack = from_msg_pack
+    ret.pybind_impl = property(fget=lambda self: self.__pybind_impl)
 
     return ret