should we have a default max tokens for anthropic

[alexagriffith]

Description:

Anthropic's max tokens is required https://docs.anthropic.com/en/api/messages#body-max-tokens

while openai's is optional https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_completion_tokens

should we have some default max token setting for translation from openai to anthropic?

[hustxiayang]

liteLLM used 4K as default max_tokens, which I do not think it's good. In our previous implementation, we used the model's output limit as the default. @yuzisun

[alexagriffith]

I think that something like 4096 could be a safe default

if the stop reason is max tokens, we can return an error that informs the user that there is a default if its not set in the open ai request.

Can you elaborate on why 4k isn't good? we can use max output, it varies per model. from 4096 - 32000 https://docs.anthropic.com/en/docs/about-claude/models/overview#model-comparison-table

[alexagriffith]

The problem with having a per model number is that we would have to configure it / hardcode it I think - I dont see a way to not do that so far. How did you use the model's output limit? @hustxiayang

[hustxiayang]

@alexagriffith I think for users, when not setting a limit, it means that the users allow the model to use up the output limit. Thus, putting a hard-coded limit is not reasonable and might as expected.

we can use max output, it varies per model. from 4096 - 32000. This is what I suggested. We store such model information separately. May need some efforts for ai-gateway.

[missBerg]

Should we convert this to a discussion or is there an action item to be generated to close this issue?