[Inference API] Remove worst-case additional 50ms latency for non-rate limited requests#136167

Open

timgrein wants to merge 29 commits intoelastic:mainfrom

timgrein:es-eis-latency-issue

Commits on Oct 8, 2025

Remove worst-case additional 50ms latency for non-rate limited requests
timgrein
committed
Update docs/changelog/136167.yaml
timgrein
committed
Merge branch 'main' into es-eis-latency-issue
timgrein
authored
Do not use forbidden API
timgrein
committed
Merge remote-tracking branch 'origin/es-eis-latency-issue' into es-eis-latency-issue
timgrein
committed
Merge branch 'main' into es-eis-latency-issue
timgrein
authored
Move startRequestQueueTask before start signal
timgrein
committed
Merge remote-tracking branch 'origin/es-eis-latency-issue' into es-eis-latency-issue
timgrein
committed
Cleanup in finally block
timgrein
committed
Reject request on shutdown
timgrein
committed
Reuse rateLimitSettingsEnabled check
timgrein
committed

Commits on Oct 9, 2025

Add NoopTask to wake up queue on shutdown
timgrein
committed
Only add non-rate-limited tasks to fast-path request queue
timgrein
committed
Extract rejection logic to common static method
timgrein
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Inference API] Remove worst-case additional 50ms latency for non-rate limited requests#136167

[Inference API] Remove worst-case additional 50ms latency for non-rate limited requests#136167
timgrein wants to merge 29 commits intoelastic:mainfrom
timgrein:es-eis-latency-issue

Commits on Oct 8, 2025

Remove worst-case additional 50ms latency for non-rate limited requests

Update docs/changelog/136167.yaml

Merge branch 'main' into es-eis-latency-issue

Do not use forbidden API

Merge remote-tracking branch 'origin/es-eis-latency-issue' into es-eis-latency-issue

Merge branch 'main' into es-eis-latency-issue

Move startRequestQueueTask before start signal

Merge remote-tracking branch 'origin/es-eis-latency-issue' into es-eis-latency-issue

Cleanup in finally block

Reject request on shutdown

Reuse rateLimitSettingsEnabled check

Commits on Oct 9, 2025

Add NoopTask to wake up queue on shutdown

Only add non-rate-limited tasks to fast-path request queue

Extract rejection logic to common static method

Commits on Oct 10, 2025

Remove unnecessary cast

Use string placeholder in assertion

Adjust test to check that a throwing task does not terminate the service

Adjust error message in general exception handler

Adjust warn to error

Commits on Oct 13, 2025

Adjust error message when request gets rejected

Rename id in RateLimitingEndpointHandler to rateLimitGroupingId

Use Strings.format(...) in assertion

Use thenAnswer instead of suppression

Only reject requests of the respective execution path (rate-limited vs. not rate-limited)

Commits on Oct 14, 2025

Merge branch 'main' into es-eis-latency-issue

Submit only ingest embeddings requests to rate-limited execution path

Merge remote-tracking branch 'origin/es-eis-latency-issue' into es-eis-latency-issue

Add rate limiting check

Make NoopTask all caps

[Inference API] Remove worst-case additional 50ms latency for non-rate limited requests#136167timgrein wants to merge 29 commits intoelastic:mainelastic/elasticsearch:mainfrom timgrein:es-eis-latency-issuetimgrein/elasticsearch:es-eis-latency-issueCopy head branch name to clipboard

Commits

Commits on Oct 8, 2025

Commits on Oct 9, 2025

Commits on Oct 10, 2025

Commits on Oct 13, 2025

Commits on Oct 14, 2025

[Inference API] Remove worst-case additional 50ms latency for non-rate limited requests#136167
timgrein wants to merge 29 commits intoelastic:mainfrom
timgrein:es-eis-latency-issue