Bedrock: workaround LiteLLM passthrough issues

[jr]

Explicitly setting requestTimeout seems to be required for it to be possible to use the Bedrock provider with LiteLLM
Bedrock passthrough.

Example litellm config: https://gist.github.com/jr/bb8188c29719adb27565cffe45e0b5c3 Without this change trying to connect to LiteLLM using the bedrock provider with Bedrock API Key "sk-my_special_key" and custom VPC endpoint http://localhost:4000/bedrock will quickly fail. Seems like the requests are failing immediately for some reason.

I would love it if someone understood this more deeply, because it seems crazy.

---

Important

Adds requestTimeout: 0 to AwsBedrockHandler in bedrock.ts to fix LiteLLM passthrough issues.

• Behavior:
  • Adds requestTimeout: 0 to clientConfig.requestHandler in AwsBedrockHandler in bedrock.ts to ensure compatibility with LiteLLM passthrough.
• Context:
  • Without this change, requests to LiteLLM using the Bedrock provider fail immediately.

^{This description was created by for 9228a91. You can customize this summary. It will automatically update as commits are pushed.}

[roomote]

Thank you for your contribution! I've reviewed the changes and have some suggestions for improvement. The workaround appears functional, but understanding the root cause better and considering potential side effects would strengthen this fix.

[roomote]

The comment mentions "This should be the default anyway" but it's unclear why explicitly setting requestTimeout: 0 fixes the LiteLLM issue. Could you provide more context about what was happening without this setting? Was LiteLLM timing out immediately, or was there a different default being applied?

Also, setting requestTimeout: 0 means no timeout limit. While this fixes the immediate issue, could this potentially cause requests to hang indefinitely in certain failure scenarios? Would a reasonable timeout value (e.g., 300000 for 5 minutes) work instead?

[roomote]

Since this is a workaround for a specific LiteLLM passthrough issue, consider adding a test case that verifies the requestHandler configuration is properly set when using API key authentication. This would prevent regression if the code is refactored in the future.

Also, would it make sense to add a configuration option like awsLiteLLMCompatibilityMode so users can explicitly enable this workaround only when needed?