[Feature request] integration with DeepConf

[itsmeknt]

You've probably seen it already, but the DeepConf paper was released earlier this week. It seems to be the only inference-time scaling solution that achieved 99.9% AIME score with open weight models (GPT-OSS 120B).

For reference, here is a llama.cpp fork that supports DeepConf

Do you have any plans to add this to optillm anytime soon?

If not, I'm happy to give a go at implementing this and submitting a PR. Will be my first time hacking optillm, and I also have limited free time, so this might take ~2 weeks. I just want to confirm if it's on your plans or not, as to avoid duplicating efforts.

[codelion]

I added an initial implementation here - https://github.com/codelion/optillm/tree/main/optillm/deepconf you can help test and evaluate it.

[itsmeknt]

Happy to test it! Would it be helpful if I run it against some benchmarks and share the results too? I can only run small models that can fit in a single 3090, but I'm curious about the improvement

[codelion]

Sure, you can test with existing benchmarks that are already there in the repo AIME-24, MATH-500, OptiLLMBench and ArenaHardAuto, the evaluators for those are already in the scripts folder - https://github.com/codelion/optillm/tree/main/scripts

[itsmeknt]

I should have time to run the three benchmarks on my rig later this week! I will be using Qwen3 Coder 30B A3B Instruct AWQ 4-bit model, hope that is still useful to you guys.

I was diving into the deepconf implementation code. It seems like it first uses pytorch to compute the logits, and then passes them into the confidence calculator which converts them into logprobs and then calculates the confidence scores.

Are there any plans to have deepconf support proxy-only mode? deepconf seems like a perfect match for SGLang's radix attention mechanism and I'd feel like it could speed up the process a lot using SGLang as the backend. Since deepconf takes a very long time, I feel like a batching approach + radix attention would be such a nice QoL. I believe all the OpenAI-compatible engines exposes logprobs directly so integration should be simple.

[codelion]

Someone will need to implement it in SGLang, because it requires token by token generation and then intervention based on confidence scores

[itsmeknt]

I dont think I can get this to run on my machine with the models I want to test unfortunately (Qwen Coder 30b a3b and Qwen 30b a3b instruct 2507). Since it requires using transformers library, and the models I want is AWQ 4-bit quantized, I am having trouble getting it to load on my GPU.

With Sglang and vllm the model takes 19GB VRAM (same size as disk) but with AutoModelForCausalLM.from_pretrained() or AutoAWQForCausalLM.from_quantized() it takes >24GB and ooms.

Is there some way to load AWQ quantized models more efficiently with transformers? If not, I'll close this issue since deepconf is already implemented

[codelion]

I think with a 4-bit quantized model it may not work as well since we need to look at the log probs at the very least it will require a lot of tuning of config etc.