Skip to content

Conversation

@makslevental
Copy link
Contributor

@makslevental makslevental commented Jan 12, 2025

I have "bisected" the current CUDA regression (mentioned here) on A100:

git bisect start
# bad: [f5a35a31bfe6cbc16bec0c130f2bb3632dbf1fbf] [LV] Add test cases with incorrect IV live-outs.
git bisect bad f5a35a31bfe6cbc16bec0c130f2bb3632dbf1fbf
# good: [86b69c31642e98f8357df62c09d118ad1da4e16a] [SPIR-V] Fix SPIR-V extension SPV_INTEL_function_pointers: 
...
# first bad commit: [b8703369daf777706196ff914c0376c27adde3cf] [SLP] Match poison as instruction with the same opcode

The failure presents as :

Traceback (most recent call last):
  File "/home/mleventa/dev_projects/repro/repro.py", line 135, in <module>
    torch.cuda.synchronize()
  File "/home/mleventa/dev_projects/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 987, in synchronize
    return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered

The failing commit is b870336 and the last working commit is 2fe947b4. It's not clear to me the relationship but will investigate.

This PR to llvm-head is to verify that my repro on A100 matches the internal regression/fail.

@makslevental makslevental requested a review from ptillet as a code owner January 12, 2025 23:39
@makslevental makslevental force-pushed the makslevental/most-recent-llvm branch from 7780ad4 to 36ccf02 Compare January 12, 2025 23:40
@makslevental makslevental changed the base branch from main to llvm-head January 12, 2025 23:40
@makslevental makslevental force-pushed the makslevental/most-recent-llvm branch from 36ccf02 to 7780ad4 Compare January 12, 2025 23:40
@makslevental
Copy link
Contributor Author

FYI this is the reproducer used https://gist.github.com/jataylo/5ebe20d3fac7a84cb24e69c0b7b449f5 (credit to @jataylo)

@makslevental
Copy link
Contributor Author

Sanity check: putting tuningOptions.SLPVectorization = false; "unblocks" the regression (I.e., the reproducer passes on LLVM HEAD).

@antiagainst antiagainst merged commit 0d88501 into triton-lang:llvm-head Jan 13, 2025
2 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants