Feature Request: Allow Assistant Messages to Act as Generation Prefix (Prefill)

[fonfonya]

This issue proposes adding support for assistant prefill: allowing an assistant message to be provided as a prefix that the model should continue generating from, rather than treating it as a completed assistant turn.

This capability enables deterministic continuation, structured output anchoring, and server-controlled tool or schema-guided generation.

Current Behavior

When passing messages like:

messages = [
    {"role": "user", "content": "Write a short apology email to a customer for a delayed shipment."},
    {"role": "assistant", "content": "Hi John,\n\nI'm sorry for the delay with your order. "}
]

the model server currently serializes the assistant message as a completed turn (constexpr bool add_generation_prompt = true;) and then starts a new assistant turn:

Pipeline input text: <|im_start|>user
Write a short apology email to a customer for a delayed shipment.<|im_end|>
<|im_start|>assistant
Hi John,

I'm sorry for the delay with your order. <|im_end|>
<|im_start|>assistant

As a result, the provided assistant content cannot be used as the active generation prefix.

Expected Behavior

The assistant message should be treated as a partial prefix, and generation should continue immediately after it:

<|im_start|>user
Write a short apology email to a customer for a delayed shipment.<|im_end|>
<|im_start|>assistant
Hi John,

I'm sorry for the delay with your order. 

Other Use Cases

Structured Output Prefill

messages = [
    {
        "role": "user",
        "content": "Is the customer satisfied? Respond in JSON with fields \"reasoning\" and \"answer\"."
    },
    {
        "role": "assistant",
        # Prefills the JSON shape and anchors a concise reasoning style.
        "content": '{\n  "reasoning": "Based on the user\'s tone and wording, '
    }
]

This allows the server to enforce output structure while still letting the model complete the response naturally.

Tool-Guided Generation

messages = [
    {
        "role": "user",
        "content": "Tell the user that their package is delayed by 2 days."
    },
    {
        "role": "assistant",
        # Tool name and immutable arguments are injected by the system.
        # The model only needs to generate the remaining message content.
        "content": '{"tool": "send_notification", "args": {"user_id": "uid_541", "message": "'
    }
]

This pattern enables server-controlled tool configuration, while allowing the model to complete the remaining schema-constrained fields.

[mzegla]

Would configuring add_generation_prompt in the request be the solution here?

Also, regarding mentioned use cases, note that for structured output in general we have response_format parameter:
https://github.com/openvinotoolkit/model_server/tree/main/demos/continuous_batching/structured_output

Tool guided generation is strictly connected to agentic cases and it's generally implemented for a few models that we support in terms of tool calling:
https://github.com/openvinotoolkit/model_server/blob/main/docs/llm/reference.md#output-parsing-settings
https://github.com/openvinotoolkit/model_server/tree/main/demos/continuous_batching/agentic_ai

[fonfonya]

Thanks for the clarification.

For general assistant prefilling, enabling add_generation_prompt can be helpful. It appears that frameworks such as transformers and vLLM expose two separate controls:

• whether to add a new generation prompt (add_generation_prompt), and
• whether the final assistant message should be treated as an active prefix (continue_final_message).

For example, using the transformers Tokenizer for Qwen3/Qwen3-4B-Instruct-2507:

tokenizer.apply_chat_template(
    [
        {"role": "user", "content": "Who are you?"},
        {"role": "assistant", "content": "I'm"}
    ],
    tokenize=False,
    add_generation_prompt=True,
    continue_final_message=False,
)
# <|im_start|>user
# Who are you?<|im_end|>
# <|im_start|>assistant
# I'm<|im_end|>
# <|im_start|>assistant

tokenizer.apply_chat_template(
    [
        {"role": "user", "content": "Who are you?"},
        {"role": "assistant", "content": "I'm"}
    ],
    tokenize=False,
    add_generation_prompt=False,
    continue_final_message=True,
)
# <|im_start|>user
# Who are you?<|im_end|>
# <|im_start|>assistant
# I'm

tokenizer.apply_chat_template(
    [
        {"role": "user", "content": "Who are you?"},
        {"role": "assistant", "content": "I'm"}
    ],
    tokenize=False,
    add_generation_prompt=False,
    continue_final_message=False,
)
# <|im_start|>user
# Who are you?<|im_end|>
# <|im_start|>assistant
# I'm<|im_end|>

The transformers documentation also notes the following behavior:

TextGenerationPipeline sets add_generation_prompt to True by default to start a new message. However, if the final message in the chat has the assistant role, it assumes the message is a prefill and switches to continue_final_message=True. This is because most models don’t support multiple consecutive assistant messages. To override this behavior, explicitly pass the continue_final_message argument to the pipeline.

---

Regarding structured model outputs and tool-guided generation: while response_format and existing agentic tooling cover many scenarios, there are cases where part of the JSON prefix, or the tool name and a subset of arguments, are already determined by the server. Being able to prefill those prefixes and let the model continue generation from them may reduce latency (when the prefix has non-trivial length), while avoiding the need for the model to generate all content from scratch.

In this sense, assistant prefill can be seen as a general primitive that complements existing structured output and tool-calling mechanisms.

References:

• transformers chat templating: https://huggingface.co/docs/transformers/main/chat_templating#addgenerationprompt
• vLLM OpenAI-compatible chat API: https://docs.vllm.ai/en/latest/serving/openai_compatible_server/#chat-api

[mzegla]

Makes sense. I think it's worth looking into.

@Wovchena, @apaniukov I know add_generation_prompt is already configurable, but maybe continue_final_message is also something that could be useful on GenAI level? Just an idea.

[apaniukov]

The continue_final_message feature is in the backlog. The actual implementation is quite simple, so external contributions to OpenVINO GenAI are welcome in that regard.