What is best practice for evaluating a base model with inspect?

[kdu4108]

Hi, what is the recommended best practice for evaluating a base model with inspect on, for example, gsm8k?

For example, I'd like to evaluate Qwen/Qwen3-1.7B-Base. It appears that the execution will eventually reach

        if self.tokenizer.chat_template is not None:
            chat = self.tokenizer.apply_chat_template(
                hf_messages,
                add_generation_prompt=True,
                tokenize=False,
                tools=tools_list if len(tools_list) > 0 else None,
                enable_thinking=self.enable_thinking,  # not all models use this, check if it is supported
            )
        else:
            chat = ""
            for message in hf_messages:
                chat += f"{message.role}: {message.content}\n"
        # return
        return cast(str, chat)

at

inspect_ai/src/inspect_ai/model/_providers/hf.py

Line 281 in 694d3ed

if self.tokenizer.chat_template is not None:

if using the hf provider.

However, since the tokenizer of the Qwen/Qwen3-1.7B-Base base model does have a chat template, it will format the text nonetheless using the chat template (which results in gibberish).

Similarly, for vllm, it seems the text will always get formatted into chat messages if I'm reading this execution correctly:

inspect_ai/src/inspect_ai/model/_providers/openai_compatible.py

Line 146 in 694d3ed

async def generate(

.

Is there a recommended workaround for generating an input text without a chat template for base models with these two providers? I suppose you could check if "base" is in the name but this feels rather hacky (especially for different model families having different conventions for whether excluding "base" from the name is the base model or instruct, e.g., Qwen2.5 vs Qwen3). Thanks for the help!

[jjallaire]

There isn't currently a clean way to bypass the chat template. You could t hough introduce model args that make this work (so would be less hacky than looking for "base")

[kdu4108]

Thanks for the quick response. Would this be something of interest to the team maintaining the library/the community of inspect as a PR?

[jjallaire]

Yes, would absolutely take that as a PR. I noticed BTW when looking at the code that we already appear to have a chat_template model arg that is unused! (unless I'm missing something)

[kdu4108]

Hey an update here: I believe that when using vllm, it already will handle base models if you pass the chat_template arg as like here:

inspect_ai/tests/model/providers/test_vllm.py

Line 28 in 7edbf64

chat_template="{% for message in messages %}{{ message.content }}{% endfor %}",

.

Basically, at some point in the flow when resolve_models is called (

inspect_ai/src/inspect_ai/model/_model.py

Line 1325 in 7edbf64

**model_args,

), it passes the chat_template as a model_arg which is directly part of the vllm server start command like vllm serve allenai/Olmo-3-1025-7B --host 0.0.0.0 --api-key inspectai --max-model-len 16384 --tensor-parallel-size 1 --port 37718 --chat_template '{% for message in messages %}{{ message.content }}{% endfor %}'

However, I am having some difficulty validating this behavior as the actual input as transformed by the vllm server does not seem obviously exposed to the client when making a request to the vllm server. Is there an easy way to inspect the full rendered input string (after chat template) in inspect for the vllm provider, do you know?

[jjallaire]

I would just add print debugging in that spot and see what is there.