Fix: Increase N for statistically significant performance benchmarking#203
Fix: Increase N for statistically significant performance benchmarking#2030x822a5b87 wants to merge 1 commit intoAlphaGPU:mainfrom
Conversation
|
Unfortunately, since there has already been many submissions for this challenge, we cannot change the performance test case size. We can rename this to be dot product small and create another duplicate challenge that has a reasonable performance test case size |
There was a problem hiding this comment.
Verified the fix is correct and all tests pass.
What was checked:
- : N changed from 5 → 4,194,304 (2²², ~32 MB for two float32 tensors — well within 16 GB VRAM)
- : Updated to match, both files are consistent
- Ran a CUDA dot product solution against the updated challenge — all functional and performance tests pass
Minor note: The CLAUDE.md guide recommends 10M–100M elements for 1D performance tests (N=4,194,304 is ~4M, slightly below). A value like N=33,554,432 (2²⁵) would better align with guidelines, but this fix is a huge improvement over N=5 and the change is valid as-is.
Approving.
There was a problem hiding this comment.
Verified the fix is correct and all tests pass.
Checked: N changed from 5 to 4,194,304 (2^22), challenge.html updated to match, memory footprint ~32 MB (well within 16 GB VRAM). Ran a CUDA solution against the updated challenge - all functional and performance tests pass.
Minor note: CLAUDE.md guide recommends 10M-100M elements for 1D performance tests; 4,194,304 is slightly below that range, but this is a massive improvement over N=5 and is valid as-is.
Approving.
The former value of N for performance benchmarking is 5, and it results in many ridiculous benchmarking results.