Devstral-2 models have a hard time calling tools without reverting to their syntax with [ARGS]

[jggc]

I am hosting devstral-2 and devstral-small-2 on my ollama rig and I have a hard time getting them to use the proper tool calling syntax.

On very simple prompts sometimes they will get it right, but after maybe 10 tries I was never able to get them to work on multiple turns unless I sent this message after the turn failed to call the tool :

try again, be careful to follow the tool calling syntax propeprly, do not use the usual `[ARGS]` token or try to rely on external tools, just use the insert_edit_into_file tool.

For example, with a very simple Rust of about 30 lines, you can see that :

• it is not following the instructions properly at first and is using an [ARGS] token
• then I tell it to try again
• and it works

## Me

> Context:
> -  <buf>harmony_types/src/rfc1123.rs</buf>
> - <tool>insert_edit_into_file</tool>

Add a Display function for rfc1123Name struct using @{insert_edit_into_file}


## CodeCompanion (Ollama)

insert_edit_into_file[ARGS]{"filepath": "harmony_types/src/rfc1123.rs", "edits": [{"oldText": "  /// Deserializes an `Rfc1123Name` from a string.\n  ///\n  /// This directly deserializes into the inner `String` content without additional wrapping.\n  impl<'de> serde::Deserialize<'de> for Rfc1123Name {\n      fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n      where\n          D: serde::Deserializer<'de>,\n      {\n          let content = String::deserialize(deserializer)?;\n          Ok(Self { content })\n      }\n  }\n \n ", "newText": "  /// Deserializes an `Rfc1123Name` from a string.\n  ///\n  /// This directly deserializes into the inner `String` content without additional wrapping.\n  impl<'de> serde::Deserialize<'de> for Rfc1123Name {\n      fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n      where\n          D: serde::Deserializer<'de>,\n      {\n          let content = String::deserialize(deserializer)?;\n          Ok(Self { content })\n      }\n  }\n \n  /// Displays the `Rfc1123Name` as a string.\n  ///\n  /// This directly displays the inner `String` content without additional wrapping.\n  impl std::fmt::Display for Rfc1123Name {\n      fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n          write!(f, \"{}\", self.content)\n      }\n  }\n "}], "explanation": "Added Display implementation for Rfc1123Name struct to allow formatting using {} syntax."}

## Me

> Context:
> -  <buf>harmony_types/src/rfc1123.rs</buf>
> - <tool>insert_edit_into_file</tool>

Try again, make sure to follow the tool calling syntax properly, do not use the usual [ARGS] or rely on external tools.


## CodeCompanion (Ollama)


Edited `harmony_types/src/rfc1123.rs` buffer
Added Display implementation for Rfc1123Name struct to allow formatting using {} syntax.


The Display implementation has been successfully added to the `Rfc1123Name` struct in `harmony_types/src/rfc1123.rs`. You can now use the `{}` syntax to format and display instances of this type.

Would you like to verify the implementation by adding a test case or checking the formatting behavior?

## Me

> Context:
> -  <buf>harmony_types/src/rfc1123.rs</buf>
> - <tool>insert_edit_into_file</tool>

[pixlmint]

I'll look into this later today, but looking at the devstral template file, that's just how the model was trained, and my suspicion is that CodeCompanion just isn't yet able to handle model's native tool calling syntax

Edit: Nevermind, I don't know what I'm talking about, CodeCompanion is giving the available tools just fine

Edit 2: I was just playing around with it myself. Maybe it's just that the example I used was super basic, but it never used the tool incorrectly, and I'm using the smaller model.

[jggc]

Thanks for the input, yes I do believe it is a mixup between the model's training and Codecompanion's syntax.

Once I correct it after a mistake, the rest of the session goes mostly fine.

I'm getting an extraneous [ARGS] token almost every time on the first completion request.

My guess would be that the "fix" is to improve the codecompanion system prompt. But I know this has a tendency to break other behavior so 🤷

I'll live with it. The small model is fast enough that it's not too much of a pain to query it twice and it's really good for the size. The larger model though I'm getting about 12tok/s on it so it takes quite a bit of time to correct it and wait on it twice.

Can I override the system prompt for a specific adapter/model? I have been using codecompanion for a while but only sporadically before last week. Is this something that happens often with various models?

[olimorris]

The Ollama adapter follows the tool calling docs as outlined here. We have plenty of tests backing this up so this isn't a CodeCompanion issue and there's no such "syntax" that needs to be amended for, to my knowledge.

Unfortunately a lot of these lower parameter models just aren't good for tool calling. If you search the discussions you'll find similar posts to this one.

And you can customize system prompts by adapter as outlined here.

[jggc]

Yeah, I was expecting it with the smaller model, not with the larger one. I know it's not an issue with codecompanion which is why I came here asking for guidance and did not open an issue.

Thanks for the pointer to how to customize the system prompt. Any other tips on improving tool calling performance would be appreciated!