Fail to run the tutorial example. #17
Description
Fail to run the tutorial example shown at the document.
Neutrino has been install successful.
~/g/n/e/probe (main)> neutrino --help
usage: neutrino [options] command
NOTE: Probes must be given via -p (--probe) option. Buit-in tools: ('program_progress', '__init__', 'gmem_bytes_analysis', 'gmem_bytes', 'dmat', 'block_sched', 'tensorop',
'dmat_callback', 'block_sched_callback')
positional arguments:
command
options:
-h, --help show this help message and exit
-p PROBE, --probe PROBE
probe in form of path to the file (default: None)
--tracedir TRACEDIR parent folder of traces (default: ./trace)
--driver DRIVER path to the real cuda/hip driver shared library (default: /lib/x86_64-linux-gnu/libcuda.so.535.261.03)
--python PYTHON path to python executable used (default: ~/git/neutrino/venv/bin/python3)
--filter FILTER filter OUT buggy kernels by (part of) name, split by : (default: None)
-k KERNEL, --kernel KERNEL
filter the kernel by (part of) name, split by : (default: None)
--callback CALLBACK attach callback for trace analysis (default: None)
--benchmark enable benchmark mode to evaluate overhead w.r.t. the original kernel (default: False)
--memusage prevent the profiling and only measure the memory usage (default: False)
Examples: `neutrino -t gmem_bytes python test/zero_.py`. Open issue(s) in https://github.com/neutrino-gpu/neutrino if encountered problemsBug Traces
The probe code I use.
from neutrino import probe, Map
import neutrino.language as nl
# declare maps for persistence
@Map(level="warp", type="array", size=16, cap=1)
class BlockSched:
start: nl.u64
elapsed: nl.u64
cuid: nl.u32
# declare probe registers shared across probes
start: nl.u64 = 0 # starting clock
# define probes with decorator
@probe(pos="kernel", level="warp", before=True)
def thread_start():
start = nl.clock()
@probe(pos="kernel", level="warp")
def thread_end():
# elapsed = nl.clock() - start
BlockSched.save(start, start, nl.cuid())The command I to measure the profile of torch.
neutrino -p probe.py python -c "import torch; torch.zeros((4096, 4096), dtype=torch.float16)"I cannot see neither any output from the bash nor offline traces at ./traces
Why I use a dummy probe?
This dummy probe code does not measure the elapsed time because neutrino will complain the elapsed does not exist.
Here are the probe code that can cause error.
from neutrino import probe, Map
import neutrino.language as nl
# declare maps for persistence
@Map(level="warp", type="array", size=16, cap=1)
class BlockSched:
start: nl.u64
elapsed: nl.u64
cuid: nl.u32
# declare probe registers shared across probes
start: nl.u64 = 0 # starting clock
# define probes with decorator
@probe(pos="kernel", level="warp", before=True)
def thread_start():
start = nl.clock()
@probe(pos="kernel", level="warp")
def thread_end():
elapsed = nl.clock() - start
BlockSched.save(start, elapsed, nl.cuid())The full bug traces
Traceback (most recent call last):
File "~/git/neutrino/venv/bin/neutrino", line 33, in <module>
sys.exit(load_entry_point('neutrino==0.1.0', 'console_scripts', 'neutrino')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/git/neutrino/venv/lib/python3.11/site-packages/neutrino/cli.py", line 82, in main
NEUTRINO_PROBE = compile(NEUTRINO_MODE, source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/git/neutrino/venv/lib/python3.11/site-packages/neutrino/language/compiler.py", line 7, in compile
regs, probes, maps, callback = parse(source)
^^^^^^^^^^^^^
File "~/git/neutrino/venv/lib/python3.11/site-packages/neutrino/language/frontend.py", line 229, in parse
visitor.visit(ast.Module(body=node.body)) # Take it as independent code
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/ast.py", line 418, in visit
return visitor(node)
^^^^^^^^^^^^^
File "~/git/neutrino/venv/lib/python3.11/site-packages/neutrino/language/frontend.py", line 165, in generic_visit
super().generic_visit(node)
File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
self.visit(item)
File "/usr/lib/python3.11/ast.py", line 418, in visit
return visitor(node)
^^^^^^^^^^^^^
File "~/git/neutrino/venv/lib/python3.11/site-packages/neutrino/language/frontend.py", line 49, in visit_Assign
name = self.reg_map[node.targets[0].id]
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
KeyError: 'elapsed'