fix(llmobs): improve serialization defaults for pydantic models [backport 3.2] (#12767)

Backport cf86284 from #12635 to 3.2.

[MLOB-2314]
Cherry-pick from @helmus' fix in #12617, only additions are tests and
release note. Thanks @helmus for the contribution!

This PR improves our serialization defaults for non-JSON serializable
objects, primarily pydantic models and enum/sentinel classes.
- pydantic models are not JSON serializable directly, but have a
`model_dump()` method which converts the object into a json serializable
representation. We'll account for this in our serialization logic,
although there is a small risk that a non-pydantic object also
implements `model_dump()` without being json-serializable. However this
should still be caught in our non-serializable default code.
- enum/sentinel classes (like OpenAI's `NOT_GIVEN`
[class](https://github.com/openai/openai-python/blob/main/src/openai/_types.py#L133))
are not JSON serializable but usually have a str/repr implementation.
We'll try converting the object to a string first before defaulting to
our placeholder (`[Unserializable object: <repr(obj)>]`).

Note: this PR does not introduce a library dependency, only adds a
testing dependency on `Pydantic`.

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)


[MLOB-2314]:
https://datadoghq.atlassian.net/browse/MLOB-2314?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Author: Yun-Kim

.riot/requirements/12c5529.txt

@@ -2,85 +2,85 @@
 # This file is autogenerated by pip-compile with Python 3.10
 # by the following command:
 #
-#    pip-compile --no-annotate .riot/requirements/12c5529.in
+#    pip-compile --allow-unsafe --no-annotate .riot/requirements/12c5529.in
 #
-aiohappyeyeballs==2.4.3
-aiohttp==3.10.10
-aiosignal==1.3.1
+aiohappyeyeballs==2.4.8
+aiohttp==3.11.13
+aiosignal==1.3.2
 annotated-types==0.7.0
-anyio==4.6.2.post1
+anyio==4.8.0
 appdirs==1.4.4
 async-timeout==4.0.3
-attrs==24.2.0
-certifi==2024.8.30
-charset-normalizer==3.4.0
-coverage[toml]==7.6.3
+attrs==25.1.0
+certifi==2025.1.31
+charset-normalizer==3.4.1
+coverage[toml]==7.6.12
 dataclasses-json==0.6.7
-datasets==3.0.1
+datasets==3.3.2
 dill==0.3.8
 distro==1.9.0
 exceptiongroup==1.2.2
-filelock==3.16.1
-frozenlist==1.4.1
-fsspec[http]==2024.6.1
+filelock==3.17.0
+frozenlist==1.5.0
+fsspec[http]==2024.12.0
 h11==0.14.0
-httpcore==1.0.6
-httpx==0.27.2
-huggingface-hub==0.26.0
+httpcore==1.0.7
+httpx==0.28.1
+huggingface-hub==0.29.2
 hypothesis==6.45.0
 idna==3.10
 iniconfig==2.0.0
-jiter==0.6.1
+jiter==0.8.2
 jsonpatch==1.33
 jsonpointer==3.0.0
-langchain==0.2.16
-langchain-community==0.2.17
-langchain-core==0.2.41
+langchain==0.2.17
+langchain-community==0.2.19
+langchain-core==0.2.43
 langchain-openai==0.1.25
 langchain-text-splitters==0.2.4
-langsmith==0.1.136
-marshmallow==3.23.0
-mock==5.1.0
+langsmith==0.1.147
+marshmallow==3.26.1
+mock==5.2.0
 multidict==6.1.0
 multiprocess==0.70.16
 mypy-extensions==1.0.0
 nest-asyncio==1.6.0
 numpy==1.26.4
-openai==1.52.0
+openai==1.65.3
 opentracing==2.4.0
-orjson==3.10.7
-packaging==24.1
+orjson==3.10.15
+packaging==24.2
 pandas==2.2.3
 pluggy==1.5.0
-propcache==0.2.0
-pyarrow==17.0.0
-pydantic==2.9.2
-pydantic-core==2.23.4
+propcache==0.3.0
+pyarrow==19.0.1
+pydantic==2.10.6
+pydantic-core==2.27.2
 pysbd==0.3.4
-pytest==8.3.3
+pytest==8.3.5
 pytest-asyncio==0.21.1
-pytest-cov==5.0.0
+pytest-cov==6.0.0
 pytest-mock==3.14.0
 python-dateutil==2.9.0.post0
-pytz==2024.2
+pytz==2025.1
 pyyaml==6.0.2
 ragas==0.1.21
-regex==2024.9.11
+regex==2024.11.6
 requests==2.32.3
 requests-toolbelt==1.0.0
-six==1.16.0
+six==1.17.0
 sniffio==1.3.1
 sortedcontainers==2.4.0
-sqlalchemy==2.0.36
+sqlalchemy==2.0.38
 tenacity==8.5.0
-tiktoken==0.8.0
-tomli==2.0.2
-tqdm==4.66.5
+tiktoken==0.9.0
+tomli==2.2.1
+tqdm==4.67.1
 typing-extensions==4.12.2
 typing-inspect==0.9.0
-tzdata==2024.2
-urllib3==2.2.3
-vcrpy==6.0.2
-wrapt==1.16.0
+tzdata==2025.1
+urllib3==2.3.0
+vcrpy==7.0.0
+wrapt==1.17.2
 xxhash==3.5.0
-yarl==1.15.5
+yarl==1.18.3

.riot/requirements/146f2d8.txt

@@ -2,85 +2,85 @@
 # This file is autogenerated by pip-compile with Python 3.8
 # by the following command:
 #
-#    pip-compile --no-annotate .riot/requirements/146f2d8.in
+#    pip-compile --allow-unsafe --no-annotate --resolver=backtracking .riot/requirements/146f2d8.in
 #
-aiohappyeyeballs==2.4.3
-aiohttp==3.10.10
+aiohappyeyeballs==2.4.4
+aiohttp==3.10.11
 aiosignal==1.3.1
 annotated-types==0.7.0
 anyio==4.5.2
 appdirs==1.4.4
 async-timeout==4.0.3
-attrs==24.2.0
-certifi==2024.8.30
-charset-normalizer==3.4.0
+attrs==25.1.0
+certifi==2025.1.31
+charset-normalizer==3.4.1
 coverage[toml]==7.6.1
 dataclasses-json==0.6.7
-datasets==3.0.1
+datasets==3.1.0
 dill==0.3.8
 distro==1.9.0
 exceptiongroup==1.2.2
 filelock==3.16.1
-frozenlist==1.4.1
-fsspec[http]==2024.6.1
+frozenlist==1.5.0
+fsspec[http]==2024.9.0
 h11==0.14.0
-httpcore==1.0.6
-httpx==0.27.2
-huggingface-hub==0.26.0
+httpcore==1.0.7
+httpx==0.28.1
+huggingface-hub==0.29.2
 hypothesis==6.45.0
 idna==3.10
 iniconfig==2.0.0
-jiter==0.6.1
+jiter==0.8.2
 jsonpatch==1.33
 jsonpointer==3.0.0
-langchain==0.2.16
-langchain-community==0.2.17
-langchain-core==0.2.41
+langchain==0.2.17
+langchain-community==0.2.19
+langchain-core==0.2.43
 langchain-openai==0.1.25
 langchain-text-splitters==0.2.4
-langsmith==0.1.136
+langsmith==0.1.147
 marshmallow==3.22.0
-mock==5.1.0
+mock==5.2.0
 multidict==6.1.0
 multiprocess==0.70.16
 mypy-extensions==1.0.0
 nest-asyncio==1.6.0
 numpy==1.24.4
-openai==1.52.0
+openai==1.65.3
 opentracing==2.4.0
-orjson==3.10.7
-packaging==24.1
+orjson==3.10.15
+packaging==24.2
 pandas==2.0.3
 pluggy==1.5.0
 propcache==0.2.0
 pyarrow==17.0.0
-pydantic==2.9.2
-pydantic-core==2.23.4
+pydantic==2.10.6
+pydantic-core==2.27.2
 pysbd==0.3.4
-pytest==8.3.3
+pytest==8.3.5
 pytest-asyncio==0.21.1
 pytest-cov==5.0.0
 pytest-mock==3.14.0
 python-dateutil==2.9.0.post0
-pytz==2024.2
+pytz==2025.1
 pyyaml==6.0.2
 ragas==0.1.21
-regex==2024.9.11
+regex==2024.11.6
 requests==2.32.3
 requests-toolbelt==1.0.0
-six==1.16.0
+six==1.17.0
 sniffio==1.3.1
 sortedcontainers==2.4.0
-sqlalchemy==2.0.36
+sqlalchemy==2.0.38
 tenacity==8.5.0
 tiktoken==0.7.0
-tomli==2.0.2
-tqdm==4.66.5
+tomli==2.2.1
+tqdm==4.67.1
 typing-extensions==4.12.2
 typing-inspect==0.9.0
-tzdata==2024.2
+tzdata==2025.1
 urllib3==1.26.20
 vcrpy==6.0.2
-wrapt==1.16.0
+wrapt==1.17.2
 xxhash==3.5.0
 yarl==1.15.2

.riot/requirements/1687eab.txt

@@ -2,82 +2,82 @@
 # This file is autogenerated by pip-compile with Python 3.11
 # by the following command:
 #
-#    pip-compile --no-annotate .riot/requirements/1687eab.in
+#    pip-compile --allow-unsafe --no-annotate --resolver=backtracking .riot/requirements/1687eab.in
 #
-aiohappyeyeballs==2.4.3
-aiohttp==3.10.10
-aiosignal==1.3.1
+aiohappyeyeballs==2.4.8
+aiohttp==3.11.13
+aiosignal==1.3.2
 annotated-types==0.7.0
-anyio==4.6.2.post1
+anyio==4.8.0
 appdirs==1.4.4
-attrs==24.2.0
-certifi==2024.8.30
-charset-normalizer==3.4.0
-coverage[toml]==7.6.3
+attrs==25.1.0
+certifi==2025.1.31
+charset-normalizer==3.4.1
+coverage[toml]==7.6.12
 dataclasses-json==0.6.7
-datasets==3.0.1
+datasets==3.3.2
 dill==0.3.8
 distro==1.9.0
-filelock==3.16.1
-frozenlist==1.4.1
-fsspec[http]==2024.6.1
+filelock==3.17.0
+frozenlist==1.5.0
+fsspec[http]==2024.12.0
 h11==0.14.0
-httpcore==1.0.6
-httpx==0.27.2
-huggingface-hub==0.26.0
+httpcore==1.0.7
+httpx==0.28.1
+huggingface-hub==0.29.2
 hypothesis==6.45.0
 idna==3.10
 iniconfig==2.0.0
-jiter==0.6.1
+jiter==0.8.2
 jsonpatch==1.33
 jsonpointer==3.0.0
-langchain==0.2.16
-langchain-community==0.2.17
-langchain-core==0.2.41
+langchain==0.2.17
+langchain-community==0.2.19
+langchain-core==0.2.43
 langchain-openai==0.1.25
 langchain-text-splitters==0.2.4
-langsmith==0.1.136
-marshmallow==3.23.0
-mock==5.1.0
+langsmith==0.1.147
+marshmallow==3.26.1
+mock==5.2.0
 multidict==6.1.0
 multiprocess==0.70.16
 mypy-extensions==1.0.0
 nest-asyncio==1.6.0
 numpy==1.26.4
-openai==1.52.0
+openai==1.65.3
 opentracing==2.4.0
-orjson==3.10.7
-packaging==24.1
+orjson==3.10.15
+packaging==24.2
 pandas==2.2.3
 pluggy==1.5.0
-propcache==0.2.0
-pyarrow==17.0.0
-pydantic==2.9.2
-pydantic-core==2.23.4
+propcache==0.3.0
+pyarrow==19.0.1
+pydantic==2.10.6
+pydantic-core==2.27.2
 pysbd==0.3.4
-pytest==8.3.3
+pytest==8.3.5
 pytest-asyncio==0.21.1
-pytest-cov==5.0.0
+pytest-cov==6.0.0
 pytest-mock==3.14.0
 python-dateutil==2.9.0.post0
-pytz==2024.2
+pytz==2025.1
 pyyaml==6.0.2
 ragas==0.1.21
-regex==2024.9.11
+regex==2024.11.6
 requests==2.32.3
 requests-toolbelt==1.0.0
-six==1.16.0
+six==1.17.0
 sniffio==1.3.1
 sortedcontainers==2.4.0
-sqlalchemy==2.0.36
+sqlalchemy==2.0.38
 tenacity==8.5.0
-tiktoken==0.8.0
-tqdm==4.66.5
+tiktoken==0.9.0
+tqdm==4.67.1
 typing-extensions==4.12.2
 typing-inspect==0.9.0
-tzdata==2024.2
-urllib3==2.2.3
-vcrpy==6.0.2
-wrapt==1.16.0
+tzdata==2025.1
+urllib3==2.3.0
+vcrpy==7.0.0
+wrapt==1.17.2
 xxhash==3.5.0
-yarl==1.15.5
+yarl==1.18.3