Proposal: Enable multi-provider support in `ash_ai` using `lang_schema`

[nallwhy]

Is your feature request related to a problem? Please describe.

Currently, ash_ai uses ash_json_api to generate structured outputs and function parameters. This works well for OpenAI’s API, but the generated schema doesn’t comply with the requirements of other providers like Gemini. As a result, ash_ai cannot be easily used with non-OpenAI providers.

Describe the solution you'd like

I’d like to propose integrating lang_schema, a library I created to abstract schema definitions and generate provider-specific JSON schemas. It defines an abstract schema format and provides converters for each provider (e.g., OpenAI, Gemini).
If we introduce a new library, ash_lang_schema, to convert Ash resources into this abstract schema format, then ash_ai could support multiple AI providers seamlessly—not just OpenAI.

Describe alternatives you've considered

• Extending ash_json_api to support Gemini or other providers, but this would likely lead to increasing complexity and diverging logic.
• Manually writing separate schema generators per provider, which would duplicate effort and reduce maintainability.

Additional context

This proposal builds on langchain#278, where I first introduced the idea of abstracting JSON schemas for cross-provider AI integration.

[zachdaniel]

🤔 hmm....I'm interested, but it would be a pretty big shift from what it currently uses, and I'd have to vet lang_schema, and on my initial read it seems fairly limited in terms of the available types to express etc.

I'd suggest that we start by making the resource/type -> schema generation pluggable, and then you can prove out the usage of lang_schema independently?

[zachdaniel]

I do see the issue you're expressing though, and we want cross LLM compatibility, I'd just like to have a way to prove out the pattern before integrating things.

[nallwhy]

I agree with your suggestion to validate the approach in a pluggable way first.

One of the reasons I created this library is because many AI providers’ schema specs diverge significantly from standard JSON Schema. For example, some rely on the order of map entries or use non-standard fields like propertyOrdering, which can’t be handled properly using only ash_json_api.

To make ash_ai more AI-friendly, we might also need to consider allowing things like attribute ordering when defining return values. The order of fields can have a significant impact on the quality of responses generated by LLMs.

[zachdaniel]

@nallwhy I think a first step would be to vendor the existing code from ash_json_api into ash_ai, into a module that can be swapped. We would remove the OpenApiSpex related code and make it all raw JSON. Would you be interested in taking on that work?

Once that is done, we can investigate introducing your cross-language schema tool?

[nallwhy]

Sounds good. Here’s a quick summary of the main responsibilities currently handled by AshJsonApi:

• Attribute to JSON Schema Conversion: Converting resource fields to JSON schemas via AshJsonApi.OpenApi.resource_write_attribute_type/4.
• Filter Schema Generation: Automatically generating filter JSON schemas for fields using AshJsonApi.OpenApi.raw_filter_type/2.
• Data Serialization: Serializing resources into clean data structures with AshJsonApi.Serializer.serialize_value/4.

For now, we can probably skip bringing over the serialize_value logic.

For the migration, my plan would be to bring the remaining logic over to ash_ai and replace the OpenApiSpex structures with raw JSON, initially targeting OpenAI’s schema conventions to ensure backward compatibility.

Once that part is stable, we can then work on adding the abstract schema -> converter process to support multiple providers. Does that sound like a reasonable approach?

[zachdaniel]

I'd say let's bring over serialize_value too since it's all pretty much the same thing under the hood.

[TwistingTwists]

Hi @zachdaniel , @nallwhy is someone working on this already?

[zachdaniel]

I don't think anyone is doing step 1 yet, which is vendoring the ash_json_api serialization logic into ash_ai.

[nallwhy]

I’m currently working on it, but I won’t be working on it until this Sunday. If anyone wants to pick it up, feel free.

[KasparKipp]

I’m currently working on it, but I won’t be working on it until this Sunday. If anyone wants to pick it up, feel free.

Looked into it a bit.
I vendored the AshJsonApi.OpenApi & AshJsonApi.Serializer modules for what was needed. Didn't get to replacing OpenApiSpex structures with raw JSON.

If it is any help then here's what I did:
https://github.com/KasparKipp/ash_ai/tree/feat--multi-provider-support-prerequisite---eliminate-OpenApiSpex-reliance

Tried to make commit messages as explanatory as possible.
The refactoring got a bit too complicated for today, to get rid of the OpenApiSpex.Schema nested usages.

[zachdaniel]

I would be fine if we first made a commit to vendor it, and then later made a commit to factor out open api spex 🙂

[KasparKipp]

Could clean it up and make it so once I understand what is actually going on.

For now, discovered that AshAi.Actions.Prompt called out to AshJsonApi.OpenApi as well. But some tests are still dead so will pick it up on Tuesday unless someone else has...