-
Notifications
You must be signed in to change notification settings - Fork 26
PTX Not Found in CUBIN #16
Copy link
Copy link
Open
Description
Command:
neutrino -p block_sched --tracedir block_sched python block_sched/memset.py
memset.py:
import torch
torch.zeros((4096,4096), dtype=torch.float16, device="cuda")
Error log:
in trace/kernel/process.log
None
None
Traceback (most recent call last):
File "/home/srcx/miniconda/envs/zz_neutrino/lib/python3.11/site-packages/neutrino/probe/cuda.py", line 698, in <module>
ptx = dump(workdir)
^^^^^^^^^^^^^
File "/home/srcx/miniconda/envs/zz_neutrino/lib/python3.11/site-packages/neutrino/probe/cuda.py", line 87, in dump
raise ValueError("PTX Not Found in CUBIN")
ValueError: PTX Not Found in CUBIN
GPU:
NVIDIA H100
CUDA Version: 13.0
sm_90
cuda: /usr/local/cuda-12.8/bin/nvcc
I'm not sure neutrino support H100?
Torch:
can't find the torch version in /neutrino/artifact/prepare_env.py
wget --quiet https://pub-eef24bf0aa5b4950860ea28dfbe39d8c.r2.dev/sm_90/torch/torch-2.5.0-cp311-cp311-linux_x86_64.whl
not only for sm_90 but also for sm_89
Thus we load torch-2.5.0-cp311-cp311-manylinux1_x86_64.whl from official torch web: https://pypi.org/project/torch/2.5.0/#files
Another tips:
Install:
git clone -b artifact https://github.com/open-neutrino/neutrino.git
cd neutrino/
python setup.py install
There's many warning: I'm not sure it has relation with PTX issue
[WARNING] can't resolve cuCheckpointProcessCheckpoint
[WARNING] can't resolve cuCheckpointProcessGetRestoreThreadId
[WARNING] can't resolve cuCheckpointProcessGetState
[WARNING] can't resolve cuCheckpointProcessLock
[WARNING] can't resolve cuCheckpointProcessRestore
[WARNING] can't resolve cuCheckpointProcessUnlock
[WARNING] can't resolve cudbgApiAttach
[WARNING] can't resolve cudbgApiClientPid
[WARNING] can't resolve cudbgApiClientRevision
[WARNING] can't resolve cudbgApiDetach
[WARNING] can't resolve cudbgApiInit
[WARNING] can't resolve cudbgAttachHandlerAvailable
[WARNING] can't resolve cudbgDebuggerCapabilities
[WARNING] can't resolve cudbgDebuggerInitialized
[WARNING] can't resolve cudbgDetachSuspendedDevicesMask
[WARNING] can't resolve cudbgEnableIntegratedMemcheck
[WARNING] can't resolve cudbgEnableLaunchBlocking
[WARNING] can't resolve cudbgEnablePreemptionDebugging
[WARNING] can't resolve cudbgGetAPI
[WARNING] can't resolve cudbgGetAPIVersion
[WARNING] can't resolve cudbgInjectionPath
[WARNING] can't resolve cudbgIpcFlag
[WARNING] can't resolve cudbgMain
[WARNING] can't resolve cudbgReportDriverApiError
[WARNING] can't resolve cudbgReportDriverApiErrorFlags
[WARNING] can't resolve cudbgReportDriverInternalError
[WARNING] can't resolve cudbgReportedDriverApiErrorCode
[WARNING] can't resolve cudbgReportedDriverApiErrorFuncNameAddr
[WARNING] can't resolve cudbgReportedDriverApiErrorFuncNameSize
[WARNING] can't resolve cudbgReportedDriverApiErrorNameAddr
[WARNING] can't resolve cudbgReportedDriverApiErrorNameSize
[WARNING] can't resolve cudbgReportedDriverApiErrorSource
[WARNING] can't resolve cudbgReportedDriverApiErrorStringAddr
[WARNING] can't resolve cudbgReportedDriverApiErrorStringSize
[WARNING] can't resolve cudbgReportedDriverInternalErrorCode
[WARNING] can't resolve cudbgResumeForAttachDetach
[WARNING] can't resolve cudbgRpcEnabled
[WARNING] can't resolve cudbgSessionId
[WARNING] can't resolve cudbgUseExternalDebugger
[WARNING] can't resolve cuDeviceGetHostAtomicCapabilities
[WARNING] can't resolve cuDeviceGetP2PAtomicCapabilities
[WARNING] can't resolve cuEGLApiInit
[WARNING] can't resolve cuEGLStreamConsumerAcquireFrame
[WARNING] can't resolve cuEGLStreamConsumerConnect
[WARNING] can't resolve cuEGLStreamConsumerConnectWithFlags
[WARNING] can't resolve cuEGLStreamConsumerDisconnect
[WARNING] can't resolve cuEGLStreamConsumerReleaseFrame
[WARNING] can't resolve cuEGLStreamProducerConnect
[WARNING] can't resolve cuEGLStreamProducerDisconnect
[WARNING] can't resolve cuEGLStreamProducerPresentFrame
[WARNING] can't resolve cuEGLStreamProducerReturnFrame
[WARNING] can't resolve cuGLCtxCreate
[WARNING] can't resolve cuGLCtxCreate_v2
[WARNING] can't resolve cuGLGetDevices
[WARNING] can't resolve cuGLGetDevices_v2
[WARNING] can't resolve cuGLInit
[WARNING] can't resolve cuGLMapBufferObject
[WARNING] can't resolve cuGLMapBufferObjectAsync
[WARNING] can't resolve cuGLMapBufferObjectAsync_v2
[WARNING] can't resolve cuGLMapBufferObjectAsync_v2_ptsz
[WARNING] can't resolve cuGLMapBufferObject_v2
[WARNING] can't resolve cuGLMapBufferObject_v2_ptds
[WARNING] can't resolve cuGLRegisterBufferObject
[WARNING] can't resolve cuGLSetBufferObjectMapFlags
[WARNING] can't resolve cuGLUnmapBufferObject
[WARNING] can't resolve cuGLUnmapBufferObjectAsync
[WARNING] can't resolve cuGLUnregisterBufferObject
[WARNING] can't resolve cuGraphicsEGLRegisterImage
[WARNING] can't resolve cuGraphicsGLRegisterBuffer
[WARNING] can't resolve cuGraphicsGLRegisterImage
[WARNING] can't resolve cuGraphicsResourceGetMappedEglFrame
[WARNING] can't resolve cuGraphicsVDPAURegisterOutputSurface
[WARNING] can't resolve cuGraphicsVDPAURegisterVideoSurface
[WARNING] can't resolve cuGraphInstantiateWithFlags
[WARNING] can't resolve cuGreenCtxGetId
[WARNING] can't resolve cuLogsCurrent
[WARNING] can't resolve cuLogsDumpToFile
[WARNING] can't resolve cuLogsDumpToMemory
[WARNING] can't resolve cuLogsRegisterCallback
[WARNING] can't resolve cuLogsUnregisterCallback
[WARNING] can't resolve cuMemDiscardAndPrefetchBatchAsync
[WARNING] can't resolve cuMemDiscardAndPrefetchBatchAsync_ptsz
[WARNING] can't resolve cuMemDiscardBatchAsync
[WARNING] can't resolve cuMemDiscardBatchAsync_ptsz
[WARNING] can't resolve cuMemGetAttribute
[WARNING] can't resolve cuMemGetAttribute_v2
[WARNING] can't resolve cuMemGetDefaultMemPool
[WARNING] can't resolve cuMemGetMemPool
[WARNING] can't resolve cuMemPrefetchBatchAsync
[WARNING] can't resolve cuMemPrefetchBatchAsync_ptsz
[WARNING] can't resolve cuMemSetMemPool
[WARNING] can't resolve cuProfilerInitialize
[WARNING] can't resolve cuProfilerStart
[WARNING] can't resolve cuProfilerStop
[WARNING] can't resolve cuVDPAUCtxCreate
[WARNING] can't resolve cuVDPAUCtxCreate_v2
[WARNING] can't resolve cuVDPAUGetDevice
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels