Gemini on Vertex AI aborts half-way through completions

[JeongJuhyeon]

Error

{"level":"error","message":"[handleAbortError] AI response error; aborting request: Aborted","stack":"Error: Aborted\n    at EventTarget.listener (/app/node_modules/@langchain/google-common/node_modules/@langchain/core/dist/utils/signal.cjs:20:24)\n    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)\n    at EventTarget.dispatchEvent (node:internal/event_target:755:26)\n    at abortSignal (node:internal/abort_controller:370:10)\n    at node:internal/abort_controller:373:20\n    at Set.forEach (<anonymous>)\n    at abortSignal (node:internal/abort_controller:371:30)\n    at Timeout._onTimeout (node:internal/abort_controller:127:7)\n    at listOnTimeout (node:internal/timers:581:17)\n    at process.processTimers (node:internal/timers:519:7)"}
{"code":"ERR_HTTP_HEADERS_SENT","level":"error","message":"[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client","stack":"Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at ServerResponse.setHeader (node:_http_outgoing:659:11)\n    at ServerResponse.header (/app/node_modules/express/lib/response.js:794:10)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:174:12)\n    at ServerResponse.json (/app/node_modules/express/lib/response.js:278:15)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:162:21)\n    at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n    at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n    at AskController (/app/api/server/controllers/AskController.js:175:5)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /app/api/server/routes/ask/google.js:23:5"}

Reproduction steps

1. Give it any medium-sized prompt
2. Observe error
3. Give it a very short prompt, e.g. "What is 1+2"
4. No error

Background info

Because I also see issues using Bedrock (#4282) this might suggest a server setup error on our side (e.g. response size/duration limits), but out of the 6 endpoints we have installed, 4 of them (Anthropic, OpenAI, Fireworks and Groq) all work well with zero such issues when using the same prompts.

[ierezs]

@JeongJuhyeon did you find solution for that ?

[danny-avila]

Closing in favor of #4755