Added support to handle nested Pydantic types

[AdilFayyaz]

Summary

• Fix _get_element_type() in the type engine to correctly resolve nested and recursive Pydantic types that were previously flattened to str
• Add $ref resolution so nested models or enums inside containers (List, Dict) are resolved via $defs lookup instead of defaulting to str
• Rewrite anyOf handling to support $ref variants (e.g. Optional[Inner]), fixing KeyError: 'type' crashes
• Add recursive array/object branches so nested containers (List[List[int]], Dict[str, Dict[str, int]]) resolve to their correct Python types
• Handle null JSON schema type, returning NoneType instead of str
• Add example demonstrating all nested Pydantic type patterns running as parallel Flyte tasks

Test plan

• pytest tests/flyte/type_engine/pydantic/test_nested_structs_in_pydantic.py : 16 new tests covering -
  • Nested arrays: List[List[int]] schema resolution + roundtrip, List[List[Inner]] roundtrip
  • Nested dicts: List[Dict[str, int]] schema + roundtrip, Dict[str, Dict[str, int]] schema + roundtrip, List[Dict[str, Inner]] roundtrip, Dict[str, Dict[str, Inner]] roundtrip
  • anyOf with $ref: List[Optional[Inner]] schema resolution (no KeyError), List[Optional[str]] schema resolution, List[Optional[Inner]] roundtrip via guess_python_type
  • null type: {"type": "null"} resolves to NoneType
  • Enums in containers: List[ModelWithEnum] roundtrip, Dict[str, ModelWithEnum] roundtrip
  • Complex combined: model with List[List[Inner]], Dict[str, Inner], List[Dict[str, int]], List[ModelWithEnum], Optional[Inner] roundtrip with data and with None optional

All previous tests pass

[pingsutw]

LGTM, thanks!

[pingsutw]

For the follow-up, let’s make sure that FlyteFile, FlyteDirectory, and DataFrame also work in the nested model

[AdilFayyaz]

Yeah will work on that next.