Status of Offline Knowledge Distillation

[hyeobiiii]

Hi everyone,
I understand that full online distillation support is still a work-in-progress.

In the meantime, could someone clarify the current status of offline distillation? Specifically, are the loss functions and related components considered complete and stable?

I’m training an 8B student model on v0.9.1 (the latest release that works smoothly in my python 3.10 environment) and have run several experiments. While I do see performance gains, they’re still below what I get from SFT-only (with teacher generation) checkpoints.

Before I dig further, I’d like to verify whether the training code is fully implemented—just in case any unfinished parts might be influencing the results. Any insights, known issues, or example configurations that delivered solid outcomes would be greatly appreciated.

Thanks in advance, and thanks for all your hard work on this project! 🙏

[ehartford]

There's no Axolotl feature around this.

I recommend distillkit though it's opinionated.

https://github.com/arcee-ai/DistillKit

Or vibe coding a custom trainer that does what you want.

[hyeobiiii]

Thanks for the reply!

Just to clarify—what exactly did you mean by:

"There's no Axolotl feature around this."

As far as I understand, there is a feature for offline distillation, like in this module:
https://github.com/axolotl-ai-cloud/axolotl/tree/v0.9.1/src/axolotl/integrations/kd

That’s actually what I’ve been using in my experiments.

Appreciate the recommendation of other kits as well :)