Fix Triton GEMM + PostOp (add matrix) kernel benchmark int8 failure BMG #5595
+1
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dev-tomek
changed the title
anmyachev
approved these changes
Dec 4, 2025
etiotto
reviewed
Dec 4, 2025
| # Maximum across onednn=600, triton=1000 | ||
| # For onednn and triton: Some configs increase performance with warmup as a step function, but some | ||
| # slowly decrease with saturation. Performance is best at 150-200ms range, but we want stable, not just best | ||
| torch.xpu.empty_cache() |
Contributor
There was a problem hiding this comment.
what about the other benchmarks we run, shouldn't we do the same ?
Contributor
There was a problem hiding this comment.
Should we do it in a common place, maybe get_empty_cache_for_benchmark?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #5481.
Fixes
RuntimeError: Native API failed. Native API returns: 20 (UR_RESULT_ERROR_DEVICE_LOST)on Triton GEMM + PostOp (add matrix) kernel benchmark int8 BMG.The memory reservation would rise between each configuration ran within the benchmark, finally resulting in oom under the hood.
The issue visible on only a single bmg runner due to more system RAM on that runner, which makes it pass runtime checks and run an additional test case.
Passing Triton GEMM + PostOp (add matrix) kernel benchmark int8 BMG on that runner.
Similar error message is visible also on FlexAttention (batch_size=16) Causal Mask fwd, however the same fix does not apply indicating a different issue. This will be continued here: #5603.