Skip to content

PTX Not Found in CUBIN #16

@WeiMa01

Description

@WeiMa01

Command
neutrino -p block_sched --tracedir block_sched python block_sched/memset.py
memset.py:

import torch
torch.zeros((4096,4096), dtype=torch.float16, device="cuda")

Error log
in trace/kernel/process.log

None
None
Traceback (most recent call last):
  File "/home/srcx/miniconda/envs/zz_neutrino/lib/python3.11/site-packages/neutrino/probe/cuda.py", line 698, in <module>
    ptx = dump(workdir)
          ^^^^^^^^^^^^^
  File "/home/srcx/miniconda/envs/zz_neutrino/lib/python3.11/site-packages/neutrino/probe/cuda.py", line 87, in dump
    raise ValueError("PTX Not Found in CUBIN")
ValueError: PTX Not Found in CUBIN

GPU:
NVIDIA H100
CUDA Version: 13.0
sm_90
cuda: /usr/local/cuda-12.8/bin/nvcc

I'm not sure neutrino support H100?

Torch:
can't find the torch version in /neutrino/artifact/prepare_env.py
wget --quiet https://pub-eef24bf0aa5b4950860ea28dfbe39d8c.r2.dev/sm_90/torch/torch-2.5.0-cp311-cp311-linux_x86_64.whl
not only for sm_90 but also for sm_89
Thus we load torch-2.5.0-cp311-cp311-manylinux1_x86_64.whl from official torch web: https://pypi.org/project/torch/2.5.0/#files

Another tips:
Install:

git clone -b artifact https://github.com/open-neutrino/neutrino.git
cd neutrino/
python setup.py install

There's many warning: I'm not sure it has relation with PTX issue

[WARNING] can't resolve cuCheckpointProcessCheckpoint
[WARNING] can't resolve cuCheckpointProcessGetRestoreThreadId
[WARNING] can't resolve cuCheckpointProcessGetState
[WARNING] can't resolve cuCheckpointProcessLock
[WARNING] can't resolve cuCheckpointProcessRestore
[WARNING] can't resolve cuCheckpointProcessUnlock
[WARNING] can't resolve cudbgApiAttach
[WARNING] can't resolve cudbgApiClientPid
[WARNING] can't resolve cudbgApiClientRevision
[WARNING] can't resolve cudbgApiDetach
[WARNING] can't resolve cudbgApiInit
[WARNING] can't resolve cudbgAttachHandlerAvailable
[WARNING] can't resolve cudbgDebuggerCapabilities
[WARNING] can't resolve cudbgDebuggerInitialized
[WARNING] can't resolve cudbgDetachSuspendedDevicesMask
[WARNING] can't resolve cudbgEnableIntegratedMemcheck
[WARNING] can't resolve cudbgEnableLaunchBlocking
[WARNING] can't resolve cudbgEnablePreemptionDebugging
[WARNING] can't resolve cudbgGetAPI
[WARNING] can't resolve cudbgGetAPIVersion
[WARNING] can't resolve cudbgInjectionPath
[WARNING] can't resolve cudbgIpcFlag
[WARNING] can't resolve cudbgMain
[WARNING] can't resolve cudbgReportDriverApiError
[WARNING] can't resolve cudbgReportDriverApiErrorFlags
[WARNING] can't resolve cudbgReportDriverInternalError
[WARNING] can't resolve cudbgReportedDriverApiErrorCode
[WARNING] can't resolve cudbgReportedDriverApiErrorFuncNameAddr
[WARNING] can't resolve cudbgReportedDriverApiErrorFuncNameSize
[WARNING] can't resolve cudbgReportedDriverApiErrorNameAddr
[WARNING] can't resolve cudbgReportedDriverApiErrorNameSize
[WARNING] can't resolve cudbgReportedDriverApiErrorSource
[WARNING] can't resolve cudbgReportedDriverApiErrorStringAddr
[WARNING] can't resolve cudbgReportedDriverApiErrorStringSize
[WARNING] can't resolve cudbgReportedDriverInternalErrorCode
[WARNING] can't resolve cudbgResumeForAttachDetach
[WARNING] can't resolve cudbgRpcEnabled
[WARNING] can't resolve cudbgSessionId
[WARNING] can't resolve cudbgUseExternalDebugger
[WARNING] can't resolve cuDeviceGetHostAtomicCapabilities
[WARNING] can't resolve cuDeviceGetP2PAtomicCapabilities
[WARNING] can't resolve cuEGLApiInit
[WARNING] can't resolve cuEGLStreamConsumerAcquireFrame
[WARNING] can't resolve cuEGLStreamConsumerConnect
[WARNING] can't resolve cuEGLStreamConsumerConnectWithFlags
[WARNING] can't resolve cuEGLStreamConsumerDisconnect
[WARNING] can't resolve cuEGLStreamConsumerReleaseFrame
[WARNING] can't resolve cuEGLStreamProducerConnect
[WARNING] can't resolve cuEGLStreamProducerDisconnect
[WARNING] can't resolve cuEGLStreamProducerPresentFrame
[WARNING] can't resolve cuEGLStreamProducerReturnFrame
[WARNING] can't resolve cuGLCtxCreate
[WARNING] can't resolve cuGLCtxCreate_v2
[WARNING] can't resolve cuGLGetDevices
[WARNING] can't resolve cuGLGetDevices_v2
[WARNING] can't resolve cuGLInit
[WARNING] can't resolve cuGLMapBufferObject
[WARNING] can't resolve cuGLMapBufferObjectAsync
[WARNING] can't resolve cuGLMapBufferObjectAsync_v2
[WARNING] can't resolve cuGLMapBufferObjectAsync_v2_ptsz
[WARNING] can't resolve cuGLMapBufferObject_v2
[WARNING] can't resolve cuGLMapBufferObject_v2_ptds
[WARNING] can't resolve cuGLRegisterBufferObject
[WARNING] can't resolve cuGLSetBufferObjectMapFlags
[WARNING] can't resolve cuGLUnmapBufferObject
[WARNING] can't resolve cuGLUnmapBufferObjectAsync
[WARNING] can't resolve cuGLUnregisterBufferObject
[WARNING] can't resolve cuGraphicsEGLRegisterImage
[WARNING] can't resolve cuGraphicsGLRegisterBuffer
[WARNING] can't resolve cuGraphicsGLRegisterImage
[WARNING] can't resolve cuGraphicsResourceGetMappedEglFrame
[WARNING] can't resolve cuGraphicsVDPAURegisterOutputSurface
[WARNING] can't resolve cuGraphicsVDPAURegisterVideoSurface
[WARNING] can't resolve cuGraphInstantiateWithFlags
[WARNING] can't resolve cuGreenCtxGetId
[WARNING] can't resolve cuLogsCurrent
[WARNING] can't resolve cuLogsDumpToFile
[WARNING] can't resolve cuLogsDumpToMemory
[WARNING] can't resolve cuLogsRegisterCallback
[WARNING] can't resolve cuLogsUnregisterCallback
[WARNING] can't resolve cuMemDiscardAndPrefetchBatchAsync
[WARNING] can't resolve cuMemDiscardAndPrefetchBatchAsync_ptsz
[WARNING] can't resolve cuMemDiscardBatchAsync
[WARNING] can't resolve cuMemDiscardBatchAsync_ptsz
[WARNING] can't resolve cuMemGetAttribute
[WARNING] can't resolve cuMemGetAttribute_v2
[WARNING] can't resolve cuMemGetDefaultMemPool
[WARNING] can't resolve cuMemGetMemPool
[WARNING] can't resolve cuMemPrefetchBatchAsync
[WARNING] can't resolve cuMemPrefetchBatchAsync_ptsz
[WARNING] can't resolve cuMemSetMemPool
[WARNING] can't resolve cuProfilerInitialize
[WARNING] can't resolve cuProfilerStart
[WARNING] can't resolve cuProfilerStop
[WARNING] can't resolve cuVDPAUCtxCreate
[WARNING] can't resolve cuVDPAUCtxCreate_v2
[WARNING] can't resolve cuVDPAUGetDevice

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions