fix(core): prevent RuntimeError in Serializable.to_json() under thread concurrency

[Balaji Seshadri (gitbalaji)]

Summary

• Root cause: Serializable.to_json() iterated over self (which calls Pydantic __iter__, i.e. self.__dict__.items() live). A concurrent thread that accesses a @cached_property for the first time (e.g. _serialized on BasePromptTemplate.invoke()) writes a new key into self.__dict__, causing RuntimeError: dictionary changed size during iteration.
• Fix: Copy __dict__ before iterating — self.__dict__.copy().items(). dict.copy() is atomic under CPython's GIL, so it produces a consistent snapshot that cannot race with a concurrent @cached_property write. Keeps the original iteration logic intact with minimal diff.
• Regression test: test_to_json_thread_safe — spawns 4 threads running concurrent dumpd() and .invoke() calls against the same prompt instance. Reliably reproduced the crash without the fix.

Fixes #34887.

Areas requiring careful review

• self.__dict__.copy() is a shallow copy taken atomically under CPython's GIL. Once the copy exists, we iterate over a separate dict object, so any concurrent @cached_property write to the original __dict__ is harmless.
• Non-model-field keys that may appear in __dict__ (e.g. cached properties like _serialized) are already filtered out by _is_field_useful, so the final lc_kwargs is identical to before.
• No behaviour change for single-threaded callers.

Test plan

• test_to_json_thread_safe passes and would fail on the unfixed code
• Full tests/unit_tests/load/test_serializable.py suite passes (44 tests)
• ruff check and ruff format clean
• mypy clean

---

This PR was developed with the assistance of Claude Code (AI agent).

🤖 Generated with Claude Code

[codspeed-hq]

Merging this PR will improve performance by 34.02%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 13 improved benchmarks
⏩ 23 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	test_import_time[InMemoryRateLimiter]	184 ms	161.6 ms	+13.91%
⚡	WallTime	test_import_time[Document]	201 ms	168.7 ms	+19.12%
⚡	WallTime	test_import_time[CallbackManager]	344.5 ms	292.4 ms	+17.81%
⚡	WallTime	test_import_time[ChatPromptTemplate]	680.4 ms	557.9 ms	+21.98%
⚡	WallTime	test_import_time[InMemoryVectorStore]	659.2 ms	530.1 ms	+24.36%
⚡	WallTime	test_import_time[LangChainTracer]	487.6 ms	416.8 ms	+16.98%
⚡	WallTime	test_import_time[BaseChatModel]	580.6 ms	484.8 ms	+19.76%
⚡	WallTime	test_async_callbacks_in_sync	24.5 ms	18.3 ms	+34.02%
⚡	WallTime	test_import_time[Runnable]	533.9 ms	443.3 ms	+20.45%
⚡	WallTime	test_import_time[RunnableLambda]	540.3 ms	446.2 ms	+21.09%
⚡	WallTime	test_import_time[HumanMessage]	287.6 ms	239.6 ms	+20.04%
⚡	WallTime	test_import_time[tool]	601.5 ms	478.3 ms	+25.76%
⚡	WallTime	test_import_time[PydanticOutputParser]	583.2 ms	478.4 ms	+21.91%

---

_{Comparing gitbalaji:fix/core-serializable-thread-safe (5134cb4) with master (d6e46bb)}

Footnotes

1. 23 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩