Running ROCM dataset setup results in a Python crash

[vaenyr]

https://github.com/AMD-AIG-AIMA/GEAK-eval/blob/master/geak_eval/run.py#L63 ?

I uncommented this line and also had to manually install pytest.
Is there a reason why it was disabled? Is it expected to work ok?

[vaenyr]

Well, I tried running and it crashed hard (segmentation fault?).

I would appreciate if someone could comment on the status of the ROCM dataset.

[iraj465]

Hi @vaenyr
Thanks for pointing this out, this feature is to bootstrap ROCm setup performance baselines on AMD GPUs (we tested ROCm setup on MI300X ) but now it should support other architectures as well. We have add a PR for this. We advise to run the geak-eval setup -ds rocm on the updated codebase to bootstrap on your specific GPUs for best numerical perf approximations.

NOTE: The ROCM triton kernel tests were primarily written keeping in mind the MI300X architecture, so you might face the eval being slower on lower-end AMD GPUs.

[vaenyr]

I ran it on MI300X

[vaenyr]

It still crashes.

Stdout:

(geak) lukasz@mia1-gpu-115:~/dev/GEAK-eval$ geak-eval setup -ds rocm                                                                                                                                       [83/1919]
Creating performance evaluation folder at /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/performance/golden_results                                                                                            
Running ROCm performance analysis for /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1                                                                                                                   
Removing existing performance folder: /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/performance/golden_results                                                                                                
Ensured gen_perf_folder exists at: /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/performance/golden_results                                                                                                   
Found .py files to test: ['gemm.py', 'layernorm.py', 'moe_gemm.py', 'multreduce_matmul_dot_kernel.py', 'rmsnorm_bwd.py', 'rmsnorm_fwd.py', 'softmax.py', 'test_add_kernel.py', 'test_batched_vecmat.py', 'test_block
_copy.py', 'test_block_pointer_matmul.py', 'test_cast_matmul.py', 'test_chained_dot_fp8.py', 'test_chained_matmul.py', 'test_flashattention_fwd.py', 'test_gemm_fusion.py', 'test_gemm_no_scf.py', 'test_iv_dependen
t_matmul.py', 'test_kernel_dot.py', 'test_kernel_sub.py', 'test_load_reduce.py', 'test_matmul_MXFP.py', 'test_randn.py', 'test_random_int.py', 'test_reverse_range.py', 'test_tma_store_gemm.py', 'test_triton_flip.
py', 'test_triton_sort.py', 'test_triton_swizzle2d.py', 'triton_multreduce_matmul_kernel.py', 'naive_softmax.py']                                                                                                   
Running pytest for /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/gemm.py (test_performance then test_save_performance_results)...                                                                     
Pytest run for /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/gemm.py (test_performance & test_save_performance_results) status: True                                                                  
Stdout: ============================= test session starts ==============================                                                                                                                            
platform linux -- Python 3.10.12, pytest-8.4.1, pluggy-1.6.0                                                                                                                                                        
rootdir: /home/lukasz/dev/GEAK-eval                                                                                                                                                                                 
collected 34 items                                                                                                                                                                                                  
                                                                                                                                                                                                                    
data/ROCm/data/ROCm_v1/gemm.py ..................................        [100%]                                                                                                                                     
                                                                                                                                                                                                                    
=============================== warnings summary ===============================                                                                                                                                    
../../../envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py:108                                                                                                                                     
  /home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py:108: DeprecationWarning: warmup, rep, and use_cuda_graph parameters are deprecated. See https://github.com/triton-lang/triton/pull
/4496 for details.                                                                                                                                                                                                  
    warnings.warn(("warmup, rep, and use_cuda_graph parameters are deprecated. See "                                                                                                                                
                                                                                                                                                                                                                    
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html                                                                                                                                             
================== 34 passed, 1 warning in 187.60s (0:03:07) ===================                                                                                                                                    
Running pytest for /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py (test_performance then test_save_performance_results)...                                                                
Pytest run for /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py (test_performance & test_save_performance_results) status: False                                                            
Stdout: ============================= test session starts ==============================                                                                                                                            
platform linux -- Python 3.10.12, pytest-8.4.1, pluggy-1.6.0                                                                                                                                                        
rootdir: /home/lukasz/dev/GEAK-eval                                                                                                                                                                                 
collected 25 items                                                                                                                                                                                                  
                                                                                                                                                                                                                    
data/ROCm/data/ROCm_v1/layernorm.py .............
Stderr: Fatal Python error: Aborted

Thread 0x00007fcb293e2000 (most recent call first):
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/torch/cuda/__init__.py", line 1040 in synchronize
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/testing.py", line 96 in do_bench_cudagraph
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 113 in <lambda>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 170 in _bench
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 192 in <dictcomp>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 192 in run
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/jit.py", line 347 in <lambda>
  File "/home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py", line 252 in forward
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/torch/autograd/function.py", line 575 in apply 
  File "/home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py", line 427 in <lambda>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/testing.py", line 145 in do_bench
  File "/home/lukasz/dev/GEAK-eval/geak_eval/perf/ROCm/performance_utils_pytest.py", line 69 in run_benchmark
  File "/home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py", line 441 in test_performance
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/python.py", line 157 in pytest_pyfunc_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/python.py", line 1671 in runtest
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 178 in pytest_runtest_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 246 in <lambda>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 344 in from_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 245 in call_and_report
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 136 in runtestprotocol
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 117 in pytest_runtest_protocol
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/python.py", line 1671 in runtest
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 178 in pytest_runtest_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 246 in <lambda>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 344 in from_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 245 in call_and_report
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 136 in runtestprotocol
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 117 in pytest_runtest_protocol
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 367 in pytest_runtestloop
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 343 in _main
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 289 in wrap_session
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 336 in pytest_cmdline_main
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/home/lukasz/envs/geak/bin/pytest", line 7 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_in
tegers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_fra
me, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, hip_utils, __triton_launcher (total: 26)
Aborted (core dumped)
Traceback (most recent call last):
  File "/home/lukasz/envs/geak/bin/geak-eval", line 7, in <module>
    sys.exit(main())
  File "/home/lukasz/dev/GEAK-eval/geak_eval/run.py", line 57, in main
    return args.func(args)
  File "/home/lukasz/dev/GEAK-eval/geak_eval/run.py", line 63, in setup
    initialize_performance_eval_rocm()
  File "/home/lukasz/dev/GEAK-eval/geak_eval/initializations.py", line 16, in initialize_performance_eval_rocm
    perf_evaluator(exec_folder=ROCm_DATA_ROOT, gen_perf_folder=ROCM_PERF_GOLD_DATA_ROOT)
  File "/home/lukasz/dev/GEAK-eval/geak_eval/perf/base.py", line 21, in __call__
    return self.evaluate(*args, **kwargs)
  File "/home/lukasz/dev/GEAK-eval/geak_eval/perf/efficiency.py", line 186, in evaluate
    assert False, f"Pytest execution failed for {py_file_path} (test_performance & test_save_performance_results): {pytest_stderr}"
AssertionError: Pytest execution failed for /home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py (test_performance & test_save_performance_results): Fatal Python error: Aborted

Thread 0x00007fcb293e2000 (most recent call first):
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/torch/cuda/__init__.py", line 1040 in synchronize
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/testing.py", line 96 in do_bench_cudagraph
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 113 in <lambda>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 170 in _bench
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 192 in <dictcomp>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 192 in run
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/runtime/jit.py", line 347 in <lambda>
  File "/home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py", line 252 in forward
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/torch/autograd/function.py", line 575 in apply 
  File "/home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py", line 427 in <lambda>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/triton/testing.py", line 145 in do_bench
  File "/home/lukasz/dev/GEAK-eval/geak_eval/perf/ROCm/performance_utils_pytest.py", line 69 in run_benchmark
  File "/home/lukasz/dev/GEAK-eval/geak_eval/data/ROCm/data/ROCm_v1/layernorm.py", line 441 in test_performance
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/python.py", line 157 in pytest_pyfunc_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/python.py", line 1671 in runtest
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 178 in pytest_runtest_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 246 in <lambda>
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 344 in from_call
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 245 in call_and_report
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 136 in runtestprotocol
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/runner.py", line 117 in pytest_runtest_protocol
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 367 in pytest_runtestloop
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 343 in _main
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 289 in wrap_session
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/main.py", line 336 in pytest_cmdline_main
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/home/lukasz/envs/geak/lib/python3.10/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/home/lukasz/envs/geak/bin/pytest", line 7 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_in
tegers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_fra
me, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, hip_utils, __triton_launcher (total: 26)
Aborted (core dumped)
(geak) lukasz@mia1-gpu-115:~/dev/GEAK-eval$ 

Environment (the same as in #6 ):
OS: Ubuntu 22.04
ROCM: 6.4.1
amdgpu: 6.10.5
GPUs: 8xMI300X
Commit: 5761d21
Python: 3.10.12

Pip:

Package             Version        Editable project location
------------------- -------------- --------------------------
aiohappyeyeballs    2.6.1
aiohttp             3.12.15
aiosignal           1.4.0
async-timeout       5.0.1
attrs               25.3.0
certifi             2025.8.3
charset-normalizer  3.4.2
filelock            3.13.1
frozenlist          1.7.0
fsspec              2024.6.1
geak-eval           0.1.0
idna                3.10
Jinja2              3.1.4
MarkupSafe          2.1.5
mpmath              1.3.0
multidict           6.6.3
networkx            3.3
numpy               1.26.0
openai              0.28.0
pandas              2.3.1
parse_llm_code      0.1.31
pillow              11.0.0
pip                 25.2
propcache           0.3.2
python-dateutil     2.9.0.post0
pytorch-triton-rocm 3.3.1
pytz                2025.2
requests            2.32.4
setuptools          59.6.0
six                 1.17.0
sympy               1.13.3
torch               2.7.1+rocm6.3
torchaudio          2.7.1+rocm6.3
torchvision         0.22.1+rocm6.3
tqdm                4.67.1
triton              3.3.0
typing_extensions   4.14.1
tzdata              2025.2
urllib3             2.5.0
yarl                1.20.1

rocminfo (one GPU only):

*******                                                                                                                                                                                                             
Agent 10                                                                                                                                                                                                            
*******                                                                                                                                                                                                             
  Name:                    gfx942                                                                                                                                                                                   
  Uuid:                    GPU-b5754425b981d2a7                                                                                                                                                                     
  Marketing Name:          AMD Instinct MI300X                                                                                                                                                                      
  Vendor Name:             AMD                                                                                                                                                                                      
  Feature:                 KERNEL_DISPATCH                                                                                                                                                                          
  Profile:                 BASE_PROFILE                                                                                                                                                                             
  Float Round Mode:        NEAR                                                                                                                                                                                     
  Max Queue Number:        128(0x80)                                                                                                                                                                                
  Queue Min Size:          64(0x40)                                                                                                                                                                                 
  Queue Max Size:          131072(0x20000)                                                                                                                                                                          
  Queue Type:              MULTI                                                                                                                                                                                    
  Node:                    9                                   
  Device Type:             GPU                                 
  Cache Info:              
    L1:                      32(0x20) KB                         
    L2:                      4096(0x1000) KB                     
    L3:                      262144(0x40000) KB                  
  Chip ID:                 29857(0x74a1)                       
  ASIC Revision:           1(0x1)                              
  Cacheline Size:          64(0x40)                            
  Max Clock Freq. (MHz):   2100                                
  BDFID:                   42240                               
  Internal Node ID:        9                                   
  Compute Unit:            304                                 
  SIMDs per CU:            4                                   
  Shader Engines:          32                                  
  Shader Arrs. per Eng.:   1                                   
  WatchPts on Addr. Ranges:4                                   
  Coherent Host Access:    FALSE                               
  Memory Properties:       
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                                
  Wavefront Size:          64(0x40)                            
  Workgroup Max Size:      1024(0x400)                         
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                         
    y                        1024(0x400)                         
    z                        1024(0x400)                         
  Max Waves Per CU:        32(0x20)                            
  Max Work-item Per CU:    2048(0x800)                         
  Grid Max Size:           4294967295(0xffffffff)              
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)              
    y                        4294967295(0xffffffff)              
    z                        4294967295(0xffffffff)              
  Max fbarriers/Workgrp:   32                                  
  Packet Processor uCode:: 166                                 
  SDMA engine uCode::      22                                  
  IOMMU Support::          None                                
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    201310208(0xbffc000) KB            
      Allocatable:             TRUE                                
      Alloc Granule:           4KB                                 
      Alloc Recommended Granule:2048KB                              
      Alloc Alignment:         4KB                                 
      Accessible by all:       FALSE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB            
      Allocatable:             TRUE                                
      Alloc Granule:           4KB                                 
      Alloc Recommended Granule:2048KB                              
      Alloc Alignment:         4KB                                 
      Accessible by all:       FALSE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    201310208(0xbffc000) KB            
      Allocatable:             TRUE                                
      Alloc Granule:           4KB                                 
      Alloc Recommended Granule:2048KB                              
      Alloc Alignment:         4KB                                 
      Accessible by all:       FALSE                               
    Pool 4                   
      Segment:                 GROUP                               
      Size:                    64(0x40) KB                         
      Allocatable:             FALSE                               
      Alloc Granule:           0KB                                 
      Alloc Recommended Granule:0KB                                 
      Alloc Alignment:         0KB                                 
      Accessible by all:       FALSE                               
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                    
      Default Rounding Mode:   NEAR                                
      Default Rounding Mode:   NEAR                                
      Fast f16:                TRUE                                
      Workgroup Max Size:      1024(0x400)                         
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                         
        y                        1024(0x400)                         
        z                        1024(0x400)                         
      Grid Max Size:           4294967295(0xffffffff)              
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                  
    ISA 2                    
      Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                    
      Default Rounding Mode:   NEAR                                
      Default Rounding Mode:   NEAR                                
      Fast f16:                TRUE                                
      Workgroup Max Size:      1024(0x400)                         
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                         
        y                        1024(0x400)                         
        z                        1024(0x400)                         
      Grid Max Size:           4294967295(0xffffffff)              
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                  

[iraj465]

Hi @vaenyr,

Can you please try to your eval runs in this standard ROCm docker setup and report to us if the issue persists? Additionally, please add the errors if it does persist, it will then be reproducible from our side too. (keep triton build as is =3.2.0)

docker run -it -d --name geak-rocm-eval --device /dev/kfd \ --device /dev/dri \ --group-add video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ --shm-size=64g \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ -v <your-volume>:<your-volume> \ -v /etc/passwd:/etc/passwd:ro \ -v /etc/group:/etc/group:ro \ rocm/pytorch:latest

[vaenyr]

I've checked and it works with triton 3.2.0, 3.3.0 fails with the same error as mentioned earlier.

I'd recommend updating your requirements...