Is the FA3 backend for prefill batch-invariant when the number of tokens is fixed?

[rajagond]

[yzh119]

Yes

[rajagond]

Thanks.

Could you please explain why the FA2 backend is not deterministic in the following scenario?

Suppose I run a request with 2048 tokens. In one case, I compute 1024 + 1024 tokens. In the second case, I compute 1024 + (513 + 511 = 1024). However, due to memory pressure, I discard the KV cache for the last 511 tokens and later recompute those 511 tokens together with 513 tokens of padding.

The results do not match.

Even when I disable KV splitting, the results are still different.