Can't run nvidia-smi on 64-bit Ubuntu 13.04

[mrj10]

OS: Ubuntu 13.04, 64-bit
Kernel: 3.8.0-27
NVIDIA Driver: 319.32

I can build libnvidia-ml.so.1 just fine, but when I install it and try to run nvidia-smi, I get:

Mismatch in versions between nvidia-smi and NVML.
Are you sure you are using nvidia-smi provided with the driver?
Failed to properly shut down NVML: Function Not Found

I have tried:

• Compiling nvml_fix using gcc-4.6 and gcc-4.7 (the default).
• Using the newer version of nvml.h (NVML_API_VERSION 4, vs. 3 which is included with nvml_fix).

Curiously, I am able to compile and run http://cfsworks.com/files/downloads/nvml_bug.c successfully.

Any insight would be greatly appreciated. Thanks!

[CFSworks]

Hi! Sorry for the delay; I was on vacation until today.

Could you show me the output of ls -l /usr/lib/libnvidia-ml*?

[mrj10]

No worries :)

Before nvml_fix installation:

$ ls -l /usr/lib/libnvidia-ml*
lrwxrwxrwx 1 root root 17 Aug 4 14:45 /usr/lib/libnvidia-ml.so ->
libnvidia-ml.so.1
lrwxrwxrwx 1 root root 22 Aug 4 14:52 /usr/lib/libnvidia-ml.so.1 ->
libnvidia-ml.so.319.32
-rwxr-xr-x 1 root root 549936 Aug 4 14:45 /usr/lib/libnvidia-ml.so.319.32

Installation:

$ sudo make install TARGET_VER=319.32 PREFIX=/usr
/usr/bin/install -D -Dm755 libnvidia-ml.so.1 /usr/lib/libnvidia-ml.so.1

After:

$ ls -l libnvidia-ml*
lrwxrwxrwx 1 root root 17 Aug 4 14:45 libnvidia-ml.so ->
libnvidia-ml.so.1
-rwxr-xr-x 1 root root 12965 Aug 8 22:15 libnvidia-ml.so.1
-rwxr-xr-x 1 root root 549936 Aug 4 14:45 libnvidia-ml.so.319.32

Try to use nvidia-smi:

$ nvidia-smi
Mismatch in versions between nvidia-smi and NVML.
Are you sure you are using nvidia-smi provided with the driver?
Failed to properly shut down NVML: Function Not Found

In case it helps, a trace of the relevant syscalls when opening the library:

$ strace nvidia-smi 2>&1 | grep -n -A10 nvidia-ml
127:open("tls/x86_64/libnvidia-ml.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
128:open("tls/libnvidia-ml.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No
such file or directory)
129:open("x86_64/libnvidia-ml.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No
such file or directory)
130:open("libnvidia-ml.so.1", O_RDONLY|O_CLOEXEC) = 3
131-read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\10\0\0\0\0\0\0"...,
832) = 832
132-fstat(3, {st_mode=S_IFREG|0755, st_size=12965, ...}) = 0
133-getcwd("/usr/lib", 128) = 9
134-mmap(NULL, 2105488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0x7ffd900f8000
135-mprotect(0x7ffd900fa000, 2093056, PROT_NONE) = 0
136-mmap(0x7ffd902f9000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7ffd902f9000
137-close(3) = 0
138-mprotect(0x7ffd902f9000, 4096, PROT_READ) = 0
139:open("tls/x86_64/libnvidia-ml.so.319.32", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
140:open("tls/libnvidia-ml.so.319.32", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
141:open("x86_64/libnvidia-ml.so.319.32", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
142:open("libnvidia-ml.so.319.32", O_RDONLY|O_CLOEXEC) = 3
143-read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300d\0\0\0\0\0\0"...,
832) = 832
144-fstat(3, {st_mode=S_IFREG|0755, st_size=549936, ...}) = 0
145-getcwd("/usr/lib", 128) = 9
146-mmap(NULL, 2694976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0x7ffd8fe66000
147-mprotect(0x7ffd8fee4000, 2097152, PROT_NONE) = 0
148-mmap(0x7ffd900e4000, 32768, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7e000) = 0x7ffd900e4000
149-mmap(0x7ffd900ec000, 48960, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ffd900ec000
150-close(3) = 0

On 08/08/2013 10:13 PM, CFSworks wrote:

Hi! Sorry for the delay; I was on vacation until today.

Could you show me the output of |ls -l /usr/lib/libnvidia-ml*|?

—
Reply to this email directly or view it on GitHub
#2 (comment).

[arich82]

@mrj10:
I'm afraid I'm getting the same error with .so.319.60 on Ubuntu 13.04 (64-bit). Did you ever get this to work?

@CFSworks:
I've also tried using the new "nvml.h" file from August 1st version of the Tesla Deployment Kit (NVML Version 5). In the new header, there's a #define to replace 'nvmlInit', 'nvmlDeviceGetHandleByIndex', and 'nvmlDeviceGetHandleByPciBusId' with their "_v2" counterparts, so I simply commented out the lines in nvml_fix.c which seemed to serve the same purpose (i.e. lines 7; 9, 11, 15; 23, 25, 29; 35; 70, 74). This compiles fine with the new header (GCC 4.7), but the "Mismatch" error persists.

I see on the other thread ("New driver version 325.15") that several other people are reporting the same problem. Is it only with 64-bit Linux kernels?

I'd really appreciate any help in getting this to work.

[stiobhan]

I had the same issue on Ubuntu 12.04 x86_64. Install gcc-4.4 and compile with that version instead. Here's a list of gcc versions I tried that didn't work (with the compiled library I get the "Mismatch in versions between nvidia-smi and NVML." error):
gcc (Ubuntu/Linaro 4.6.4-1ubuntu112.04) 4.6.4
gcc-4.7 (Ubuntu/Linaro 4.7.3-2ubuntu112.04) 4.7.3
gcc-4.8 (Ubuntu 4.8.1-2ubuntu1~12.04) 4.8.1

And finally the one that made it work as expected:
gcc-4.4 (Ubuntu/Linaro 4.4.7-1ubuntu2) 4.4.7

See my other comment on #3 for how to fix this with newer gcc versions.