[WIP] support paged attention#710

Draft

jslhcl wants to merge 18 commits intomainfrom

leca/pagedAttention

Commits on May 6, 2024

Commits on May 7, 2024

add kernel functions and build successfully
Lei Cao
committed

Commits on May 8, 2024

move checkInput() to paged_attention.h
Lei Cao
committed
call flash attention code for prompt mode
Lei Cao
committed

Commits on May 20, 2024

runtime error in RunMultiHeadAttention
jslhcl
committed

Commits on May 21, 2024

UT can run now
jslhcl
committed

Commits on May 29, 2024

new UT with q,k and v whose shape is 381x512, and not use GetScratchBufferUnderMultiStream
jslhcl
committed

Commits on Jun 2, 2024

make context_lens CPU input, change scale in test, change batch_size and seq_len
jslhcl
committed

Commits on Jun 4, 2024

change window_size and causal to make ut pass
jslhcl
committed

Commits on Jun 7, 2024

enhance flash attention lib to support page attention, and UT
jslhcl
committed

Commits on Jun 10, 2024

runtime error in gpu_data_transfer.cc when copying result back to cpu for test test_cuda_paged_attention_prompt_decoding
jslhcl
committed

Commits on Jun 19, 2024

sync ort-genai changes
jslhcl
committed

Commits on Jun 22, 2024

sync flash attn 2.5.9 kernel code. test_cuda_paged_attention3 is good, test_cuda_paged_attention_decoding runtime error
jslhcl
committed

Commits on Jun 23, 2024

fix runtime error and add new test case
jslhcl
committed

Commits on Jun 25, 2024

UT for paged attention decoding case works
jslhcl
committed

Commits on Jun 26, 2024

fix block_tables and context_lens to avoid unknown random error
jslhcl
committed

Commits on Jul 1, 2024

sync with ort-genai and add document
jslhcl
committed