Deprecating inference API in favor of OpenAI endpoints

[ashwinb]

We have declared the intent to do this, no issues there. I want to talk about the specifics. How should we go about doing this?

There are some clear action items (for example, Agents API references some types which are related to the about to be deprecated APIs.) Once these action items are done, when should we perform the deprecation? I'd like to hear from folks who have production deployments and are likely to be most affected.

@leseb @bbrowning @mattf @terrytangyuan @cdoern

[mattf]

related: #2365

[bbrowning]

I think @mattf outlined a reasonable plan in the linked issue, where we update all our official content to use the newer APIs, let users know the older inference APIs are deprecated (via client warnings, server warnings, server docs), help users migrate to the newer APIs, and bump from 0.2.x to 0.3.x when we actually remove those deprecated endpoints.

Where things get a bit more complicated are for example around the Agent API, as we haven't committed to deprecating it in favor of Responses API. I'm not sure we have a clear path to make breaking changes to the Agents API itself while allowing both the old and new versions to coexist like we do with the Inference APIs. We could choose to start nudging people towards the Responses API before 0.3.x to give them a chance to migrate without immediately breaking, assuming we also choose to break the Agents API at 0.3.x to align with OpenAI API style inference parameters.

[cdoern]

Yeah, I pretty much agree with the path for this that Matt put forward. I think it'd be a useful exercise to put together a list of different adjacent APIs, providers, and usages in LLS we intend to deprecate in 0.3.0 (or beyond if we don't think we can do it all by 0.3.0) and at a high level follow the same protocol for all of these APIs:

1. client warnings both in the Library Client and in the CLI for at least 1 release (preferably more)
2. server warning for admins so they know these errors before a client reports it to them for at least one release
3. break in 0.3.0 and provide notes in the documentation for a suitable upgrade made (switch to using endpoint x instead of endpoint y)

This maps pretty well to the linked issue, but I think the addition of server warning and ensuring the library client has these warning as well is pretty important to ensure stuff like this is caught early.

One thing I'd like to bring up too is the idea of standardizing/versioning our overall API during this process. Having some sort of standard "contract" we publish in documentation and/or datatypes we promise not to break (inputs and outputs of different APIs, and things like StackRunConfig, DistributionConfig) could be a useful thing to include here as well, though tangential.

[ashwinb]

Thank you @mattf - I missed it. I did search for the word "deprecation" which didn't find any issue 😡 I will close this discussion in lieu of that one.

@bbrowning re: Agents API, modulo getting some input from the users it is hard to tell. Personally, I feel less strongly about keeping it. If we do decide to keep it, it is best that we make all breaking changes in the last release going from 0.2.x to 0.3.0 -- there is simply no good way to migrate incrementally without adding un-ergonomic parameter names ("messages_new", "response_format_new" and such)