HTTP 431 (Request Headers too long) being caused by x-goog-api-client becoming longer and longer

[eecarus]

Hi everyone!

I have a strange case of getting sporadic HTTP 431 errors using the langchain-google-vertexai integration. Specifically, a portion of the wiretap identifies the following header in particular. Only a snippet is shown since that is all was captured in the log, but the total length exceeded 8K (without me knowing how long it actually is). 8K was the point at which errors began to be registered.

x-goog-api-client: model-builder/1.76.0+top_google_constructor_method+vertexai.generative_models.GenerativeModel.generate_content_async+tools+langchain-google-vertexai/2.0.13-VertexAI_gemini-2.0-flash-001+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+langchain-google-vertex|(log truncated)

Broken down, it looks like

x-goog-api-client: 
model-builder/1.76.0+top_google_constructor_method+vertexai.generative_models.GenerativeModel.generate_content_async+tools+
langchain-google-vertexai/2.0.13-VertexAI_gemini-2.0-flash-001+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-004+
langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005+
langchain-google-vertex(log truncated)

I'm not sure how to resolve this issue (or if there is a resolution). There have been longstanding reports of x-goog-api-client getting longer and longer (especially in the nodejs versions) but those reports have been a few years old.

Below are some details that I have discovered:

1. This is where VertexAIBase obtains the user-agent. Based on this, the general format would fit langchain-google-vertexai/2.0.13-VertexAI_gemini-2.0-flash-001 or langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005. Similarly, the VertexAIEmbeddings initializes it's user-agent here.

2. The async function does not use a LRU cache but the non-async one does.

3. For the async client, this line calls the get_client_info() function and passes the full user-agent as the module argument. Because the function calls get_user_agent() again, an original user-agent of langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005`` would become langchain-google-vertexai/2.0.13-langchain-google-vertexai/2.0.13-VertexAIEmbeddings_text-embedding-005`

4. This code eventually uses the clientInfo object to generate the x-goog-api-client header.

5. It seems the code that actually generates a string similar to what we're seeing could be here, but I'm not certain at this point where the telemetry._tool_names_to_append is coming from other than here, which in turn leads to this.

---

My questions are two-fold:

a. How does this impact a long-running application that is using the same VertexAI / ChatVertexAI BaseModel instance? If the headers keep getting longer, we can only increase the allowed size so much before it will eventually be exceeded.

b. We use asyncio heavily. In the snippet above, the final call that exceeded the allowed header length was gemini-2.0-flash, but seeing the text-embedding signatures makes me think that maybe some threadlocal variable is being updated repeatedly inside the asyncio loop. However, I don't understand how Google expected such as string to be a legitimate header for their purposes.

I'm hoping someone might be able to point me towards the right direction. Thank you in advance.

[onestardao]

It looks like what you’re hitting isn’t just a header-length bug, it’s a systemic accumulation problem.
The repeating x-goog-api-client string in your trace is a sign that each call keeps stacking metadata without cleanup. That pattern usually maps to a failure type I’ve seen documented before (we call it “bootstrap-order drift” in my notes).

The tricky part is that patching it at the HTTP layer doesn’t solve the underlying leak — the fix has to be applied at the orchestration layer so the client doesn’t keep multiplying state.

If you want, I can point you to the full failure catalog where this category is broken down with permanent fixes. Let me know and I’ll share it.

[eecarus]

Thanks @onestardao ! I appreciate the response, and I agree with your conclusion as well. I would definitely appreciate the link.

[onestardao]

got it. here’s the catalog link. this case usually maps to No.14 “Bootstrap ordering” in my notes, where a long-lived client keeps accumulating per-call metadata and the fix must be at the orchestration layer, not at HTTP.

Link: [WFGY Problem Map — README]

quick checks you can try right now:

• re-init the Vertex client per worker or add a reset hook so request middleware does not append twice.
• assert the x-goog-api-client length is stable across 10 calls. if it grows, strip or rebuild the header each call.
• avoid module-level mutable user-agent builders. pass a fresh copy of headers into each async task.
• if you pool clients, add a guard that dedupes the header before send.

if you want I can point you to the exact sub-page once you confirm how you construct and reuse the client.

^_____^ Tthanks

[eecarus]

This is great! Thank you so much! I will mark your post as the answer when I get back after the holiday! 🙏

[onestardao]

Thank you for your reply ^____^ If any Q , just ask me, also note my Global Fix Map. you can see it now .