Fix cuvidCreateDecoder segfault on Linux (tcu_ulong layout) - #1
Open
devnen wants to merge 1 commit into
Open
Conversation
`tcu_ulong` is `unsigned long` in cuviddec.h: 4 bytes on Windows LLP64, 8 bytes on Linux LP64. The affected fields in CUVIDDECODECREATEINFO and CUVIDSOURCEDATAPACKET were declared as c_uint32, which is correct on Windows but truncates the struct by ~68 bytes on Linux. The driver reads past the buffer and segfaults inside cuvidCreateDecoder. Patch adds a tcu_ulong alias (c_uint32 on Windows, c_ulong on Linux/macOS) and switches the 13 affected fields in CUVIDDECODECREATEINFO plus flags and payload_size in CUVIDSOURCEDATAPACKET. Enum fields stay c_uint32. sizeof on Linux x86_64: - CUVIDDECODECREATEINFO: 108 -> 176 (matches nv-codec-headers layout) - CUVIDSOURCEDATAPACKET: 24 -> 32 Both unchanged on Windows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cuvidCreateDecodersegfaults on Linux becausetcu_ulongfields inCUVIDDECODECREATEINFOandCUVIDSOURCEDATAPACKETare declared asc_uint32. That's correct on Windows LLP64 (4 bytes) but wrong on Linux LP64 (unsigned longis 8 bytes). Result: the struct is short by ~68 bytes and the driver reads garbage.Patch:
tcu_ulongalias at the top ofdecoder.py:c_uint32on Windows,c_ulongon Linux/macOS.CUVIDDECODECREATEINFO(13 fields plus theReserved2array element type) andCUVIDSOURCEDATAPACKET(flags,payload_size) totcu_ulong.CodecType,ChromaFormat,OutputFormat,DeinterlaceMode) asc_uint32since those are 4 bytes on both platforms in the C header.CUVIDPARSERPARAMS,CUVIDEOFORMAT,CUVIDPROCPARAMS,CUVIDPICPARAMS(they useunsigned int, which is 4 bytes on both platforms).Sanity check:
ctypes.sizeof(CUVIDDECODECREATEINFO)on Linux x86_64: 108 -> 176 (matchesnv-codec-headers/dynlink_cuviddec.hlayout)ctypes.sizeof(CUVIDSOURCEDATAPACKET)on Linux x86_64: 24 -> 32Verified on:
c_uint32on Windows by definition, so it should be a no-op on that platform. Someone with a Windows setup confirming the existing tests still pass would close that gap.Caveat: I tested only NV12 4:2:0 on this hardware because the 3090's NVDEC doesn't support HEVC 4:4:4 decode (Lovelace+ feature). The 4:4:4 path through this fix should be unaffected since the struct fields are the same; someone on a 40-series or 50-series card could confirm.
The file's own docstring at lines 22-23 already flagged the Windows-only assumption, so this is the documented follow-up.