Incomplete Logprobs When Using Structured Output in Google Vertex AI

[hienphan161]

I faced an issue when trying to get full logprobs while using Structured Output with Google Vertex AI. I followed the instructions from this link and used the include_raw=True flag with the Google VertexAI model. However, the returned logprobs don't seem to cover all the tokens in the response.

Here’s my code:

from typing import Optional
from langchain.chat_models import init_chat_model
from pydantic import BaseModel, Field

# Pydantic
class Joke(BaseModel):
    """Joke to tell user."""
    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")
    rating: Optional[int] = Field(
        default=None, description="How funny the joke is, from 1 to 10"
    )

llm = init_chat_model("gemini-2.5-flash", model_provider="google_vertexai", logprobs=True)
structured_llm = llm.with_structured_output(Joke, include_raw=True, method="json_mode")
resp = structured_llm.invoke("Tell me a joke about cats")

print(resp["parsed"])
print(resp["raw"].response_metadata["logprobs_result"])

And here’s the response I got:

setup='What do you call a cat that can sing?' punchline='A meowsician!' rating=7

The logprobs_result looks like this:

[{'token': 'H', 'logprob': -0.2684961259365082, 'log_topprobs': []}, {'token': 'ere', 'logprob': -0.02346515655517578, 'log_topprobs': []}, {'token': ' the', 'logprob': -0.0009606147650629282, 'log_topprobs': []}, {'token': ' JSON', 'logprob': -1.0013011888077017e-05, 'log_topprobs': []}, {'token': ' requested', 'logprob': -2.4487404823303223, 'log_topprobs': []}, {'token': '```', 'logprob': -4.2197159928036854e-05, 'log_topprobs': []}, {'token': '{"', 'logprob': -0.013238185085356236, 'log_topprobs': []}, {'token': 'setup', 'logprob': -5.960258704362786e-07, 'log_topprobs': []}, {'token': '":"', 'logprob': -0.10070745646953583, 'log_topprobs': []}, {'token': 'What', 'logprob': -0.03174138069152832, 'log_topprobs': []}, {'token': ' do', 'logprob': -0.008807279169559479, 'log_topprobs': []}, {'token': ' you', 'logprob': -0.0003854265669360757, 'log_topprobs': []}, {'token': ' call', 'logprob': -7.962695235619321e-05, 'log_topprobs': []}, {'token': ' a', 'logprob': -0.00012873455125372857, 'log_topprobs': []}, {'token': ' cat', 'logprob': -0.0014946687733754516, 'log_topprobs': []}, {'token': ' that', 'logprob': -0.02148868516087532, 'log_topprobs': []}, {'token': ' can', 'logprob': -0.048124898225069046, 'log_topprobs': []}, {'token': ' sing', 'logprob': -0.003144164802506566, 'log_topprobs': []}, {'token': '?', 'logprob': -0.0014611022779718041, 'log_topprobs': []}, {'token': '","', 'logprob': -7.271794856933411e-06, 'log_topprobs': []}, {'token': 'punch', 'logprob': -7.152119110287458e-07, 'log_topprobs': []}, {'token': '":"', 'logprob': -3.731144170160405e-05, 'log_topprobs': []}, {'token': 'A', 'logprob': -0.00027581251924857497, 'log_topprobs': []}, {'token': ' me', 'logprob': -0.011680936440825462, 'log_topprobs': []}, {'token': 'ows', 'logprob': -0.008941029198467731, 'log_topprobs': []}, {'token': 'ician', 'logprob': -8.010359306354076e-05, 'log_topprobs': []}, {'token': '!', 'logprob': -0.0037019907031208277, 'log_topprobs': []}, {'token': '","', 'logprob': -6.436950116039952e-06, 'log_topprobs': []}, {'token': '":', 'logprob': -1.1922384146600962e-07, 'log_topprobs': []}, {'token': '7', 'logprob': -0.00581785012036562, 'log_topprobs': []}, {'token': '}', 'logprob': -5.960258704362786e-07, 'log_topprobs': []}, {'token': '\n', 'logprob': -5.960258704362786e-07, 'log_topprobs': []}]

If I combine all the tokens from the logprobs, I get:

Here is the JSON requested```{"setup":"Why did the cat cross the road?","punch":"To get to the other feline!","":7}```

You can see that the line in punchline disappears, and also the rating.