Speculative Decoding + ktransformers

[kingyaaayaaa]

ktransformers has a low utilization rate on GPU. It is enough to store and run a small 7B model on a 24G GPU and use speculative decoding to achieve inference acceleration. Is this idea feasible? Currently, the ktransformers backend does not support batch inference, which will pose a challenge to the parallel verification of large models, but I don’t think this is an unsolvable problem.