Adding a custom http client causes stall-stream protection to trigger if request takes too long

[aaronvg]

Describe the bug

I added a custom http client like this:
https://github.com/BoundaryML/baml/pull/1827/files#top

but then my requests to bedrock stopped failing if they took longer than 5s with:

DispatchFailure(
    DispatchFailure {
        source: ConnectorError {
            kind: Timeout,
            source: ThroughputBelowMinimum {
                expected: Throughput {
                    bytes_read: 1,
                    per_time_elapsed: 1s,
                },
                actual: Throughput {
                    bytes_read: 0,
                    per_time_elapsed: 1s,
                },
            },
            connection: Unknown,
        },
    },
)

To fix it I had to disable streamstall protection:
https://github.com/BoundaryML/baml/pull/1877/files

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Don't fail with a timeout

Current Behavior

Fails with an error

Reproduction Steps

Add a custom http client like in: https://github.com/BoundaryML/baml/pull/1827/files#top

Possible Solution

No response

Additional Information/Context

No response

Version

1.85.0

Environment details (OS name and version, etc.)

macos

Logs

No response

[aajtodd]

Interesting, stalled stream protection shouldn't even be enabled for this operation I don't think given it has an event stream.

Event stream operations should automatically disable registering the interceptor.

Indeed I don't see one registered for this operation here, for comparison here is the invoke_model runtime_components.

Will have to look closer at this.

[aaronvg]

we use the Converse and ConverseStream endpoints. The requests ran into the streamprotection error on Converse (i didnt test conversestream, or i dont remember if it failed there too)

[aajtodd]

If it was Converse and not ConverseStream then that makes more sense as it does have stalled stream protection applied.

Disabling it is a reasonable workaround. We probably ought to consider if this should be add it to the existing customization or not as the default behavior.

[aaronvg]

yeah definitely a bit weird that it only happens with the custom http client, but not with the default one. LLM api calls regularly take > 5s so it just triggered that protection due to no bytes received. Thanks for the response.