o1-preview model not working on chat

[perinm]

It gives buttons to click but don't actually execute like in gpt-4o.

Test bring-your-own-model

with both o1-preview, and gpt-4o set, where the latter works fine.

[perinm]

This is probably not a bug, but a new feature, because o1-preview changes a lot of requirements.

[kgilpin]

Thanks @perinm. There are differences in the API for o1, such as - it doesn't allow System messages. So, it's not working with Navie - yet!

[dustinbyrne]

Hi @perinm, thanks for the report.

You're right that o1-preview is not currently functional due to changes in the model's requirements. Though, there are a few things worth mentioning here:

1. Work on initial support for o1-preview in Navie has been completed, but it has not yet been shipped. I imagine this will be included in the next release, though I don't have a hard date for that.
2. The buttons work using a separate model, which by default is gpt-4o-mini if you're using OpenAI endpoints. This is configurable by setting the APPMAP_NAVIE_MINI_MODEL environment variable. I only mention this because it's not (yet) documented.
3. We've done some initial testing with o1-preview on SWE Bench, though our findings are that it currently benchmarks lower than gpt-4o due to changes and limitations of the model.

In regard to using o1-preview, is there a particular aspect of the model that you feel would be beneficial to bring into Navie?

[perinm]

@dustinbyrne excited for that.

Actually, I was having great success copying and pasting specific files into o1-preview, and briefly explaining what I need to code to do.

Apply changes suggested, and modify a little, copy and paste all files and repeat.
like 10x more productive than normal using gpt for helping. And surprisingly low content of bugs.

Answers can go up to 32k tokens, and context goes up 128k tokens.

Then I searched solutions to streamline the process and found this project you guys built with great passion and talent.
And feel like they are head-to-head but feel I prefer o1-preview overall, less inconsistencies on my tests, but different playgounds.

gpt-4o I tested using Navie.
o1-preview I tested using open-webui

Will keep testing on gpt-4o with Navie because this is such a great gem!