请教block selection处的理解

[cafeii]

作者你好，我想请问下关于此处l'!=l的general selection公式您如何理解的

对于l'>l的情况，我可以理解，但是对于l'<l的情况，例如2l'=l，感觉无论如何推得到的结果都不make sence

[lucidrains]

@cafeii hey Chen, using google translate as my ability to read mandarin is limited

so i'm not sure i understand this section either, and welcome the input from anyone who does

my plan is to simply expand / interpolate the compression attention scores to the full sequence length and then redivide by the fine attention block sizes and average for a new score

does that sound reasonable to you?

[lucidrains]

dc15660#diff-35145855180157a1b610845934a5105b8307b6573caf4aec837ce2042e0f5d78R273

@cafeii let me know what you think

[cafeii]

My bad... someone shared your repository on a Chinese forum, so I simply opened this issue using Chinese without thinking...

I'm simply curious about the general formula for block selection. As the paper written, l'=l=d is the default situation, where $p^{slc}=p^{cmp}$. When l'>l, e.g. l'=2l=2d, $p^{slc}[j]=p^{cmp}[2j]+p^{cmp}[2j+1]$, which can be understood as merging two $K_{cmp}$ into one (since the size of selection block is 2x of the size of compression block). However, when l'<l (the selection block size < the compression block size), I cannot find a case to make this formula look reasonable.

I think your code gives a numerical explanation that fixes my confusion. I just never thought about it from this aspect. Great thanks

[lucidrains]

@cafeii nice! glad to hear the code makes sense from your end

what's the sentiment for the paper on the chinese forums? i found it to be missing a lot of detail

[lancerts]

@lucidrains Nice work. I'm also confused about Equation 9. +1 on it's missing details

[Wesley-Jzy]

@lucidrains Also see your great work in chinese forums. May you have a plan to use triton?

[lancerts]

@Wesley-Jzy there will be some efforts to make NSA into https://github.com/linkedin/Liger-Kernel via triton :P

[cafeii]

@lucidrains Yes I think it contains not so much details as MoBA, which is another great work about sparse attetion published by MoonShot (they released Kimi-K1.5). The missing details in NSA really confused me before, especially when it came to the block selection part. Many people on the forums are saying it is a great work, and some blogs are trying to interpret it, but they actually don't care and won't dig into these details.

[LZ-QWQ]

@cafeii hey Chen, using google translate as my ability to read mandarin is limited

so i'm not sure i understand this section either, and welcome the input from anyone who does

my plan is to simply expand / interpolate the compression attention scores to the full sequence length and then redivide by the fine attention block sizes and average for a new score

does that sound reasonable to you?

Thank you for nice sharing. A problem is seems you implementation is not support sliding_stride < compress_block_size (d < l) now ?
i.e. This paper experiment setup, d=16, l = 32, l' = 64

[lucidrains]

@LZ-QWQ could you give an example of what you would expect?

say sequence length is 4, and compressed length is 2 with compressed scores [2., 4.], then with sliding length of 1., selecting for top 2 would yield the latter two chunks [4., 4.]?

[lucidrains]

@cafeii hey, noticed you closed this, so this means everything checks out?

[cafeii]

@lucidrains well I think we can reopen it after deepseek has fixed the equation 9., perhaps they need several days to make this update

or just reopen it now, anyway. there's still some details that i haven't figured out

[Cui-Peng-624]

I would like to ask another question, assuming l' != nl, that is, the selection block size is not an integer multiple of block length, how should I do this? Is this a case?