Skip to content

Fix: Increase N for statistically significant performance benchmarking#203

Open
0x822a5b87 wants to merge 1 commit intoAlphaGPU:mainfrom
0x822a5b87:main
Open

Fix: Increase N for statistically significant performance benchmarking#203
0x822a5b87 wants to merge 1 commit intoAlphaGPU:mainfrom
0x822a5b87:main

Conversation

@0x822a5b87
Copy link

The former value of N for performance benchmarking is 5, and it results in many ridiculous benchmarking results.

@kunal-mansukhani
Copy link
Contributor

Unfortunately, since there has already been many submissions for this challenge, we cannot change the performance test case size. We can rename this to be dot product small and create another duplicate challenge that has a reasonable performance test case size

Copy link
Contributor

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the fix is correct and all tests pass.

What was checked:

  • : N changed from 5 → 4,194,304 (2²², ~32 MB for two float32 tensors — well within 16 GB VRAM)
  • : Updated to match, both files are consistent
  • Ran a CUDA dot product solution against the updated challenge — all functional and performance tests pass

Minor note: The CLAUDE.md guide recommends 10M–100M elements for 1D performance tests (N=4,194,304 is ~4M, slightly below). A value like N=33,554,432 (2²⁵) would better align with guidelines, but this fix is a huge improvement over N=5 and the change is valid as-is.

Approving.

Copy link
Contributor

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the fix is correct and all tests pass.

Checked: N changed from 5 to 4,194,304 (2^22), challenge.html updated to match, memory footprint ~32 MB (well within 16 GB VRAM). Ran a CUDA solution against the updated challenge - all functional and performance tests pass.

Minor note: CLAUDE.md guide recommends 10M-100M elements for 1D performance tests; 4,194,304 is slightly below that range, but this is a massive improvement over N=5 and is valid as-is.

Approving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants