Achieving result consistency with Huggingface's implementation with manually patched kernels

[dibbla]

Hi guys!

First, thank you for creating and sharing FlashInfer. I've experienced its impressive performance benefits firsthand through frameworks like vLLM and SGLang.

For my research, I have a specific requirement to maintain output consistency with the standard Huggingface transformers implementation. As expected with highly optimized kernels, I've observed generation differences between the results from inference engines and the baseline.

My goal is to leverage the kernel speed-ups while ensuring the generation is perfectly reproducible with the Huggingface reference. I understand this is a significant challenge.

Could you offer any guidance on this? For example, are there specific kernels, compilation flags, or configurations within FlashInfer that are designed to prioritize numerical precision and consistency with standard libraries over absolute maximum performance? My (very naive) plan is to find a way to manully patch my LLM and test out different kernels.