[Feature] Batch Decoding #477

Open

AnaRhisT94 wants to merge 2 commits intostate-spaces:mainfrom

AnaRhisT94:main

Commits on Jul 18, 2024

Implemented decoding N tokens, instead of one token after another.
ilanaizelman
committed
added code which shows that chunked decoding works, and also latency test that it's much faster.
ilanaizelman
committed