SIGSEGV and waveform artefacts when using 16-bit samples with Siglent Oscilloscopes

[andreasmatz]

Hi,

I spent the evening trying to debug a problem where selecting a data width of 16 bits in Setup > Preferences > Drivers > Siglent SDS HD causes ngscopeclient to crash with an access violation when selecting a memory depth 10 k or above.

I traced the error back to Oscilloscope::Convert16BitSamplesFMA, so I assumed there was a bug in the vector arithmetic, but it prevails when I patched out all vector optimizations and instead used Oscilloscope::Convert16BitSamplesGeneric.

Here is a gdb backtrace in case it helps:

Thread 40 "ngscopeclient" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff8ffef6c0 (LWP 26025)]
0x00007ffff649ab30 in Oscilloscope::Convert16BitSamplesFMA(float*, short const*, float, float, unsigned long) () from /lib/x86_64-linux-gnu/libscopehal.so
(gdb) bt
#0  0x00007ffff649ab30 in Oscilloscope::Convert16BitSamplesFMA(float*, short const*, float, float, unsigned long) () at /lib/x86_64-linux-gnu/libscopehal.so
ngscopeclient/scopehal-apps#1  0x00007ffff649b285 in ??? () at /lib/x86_64-linux-gnu/libscopehal.so
ngscopeclient/scopehal-apps#2  0x00007ffff71f241f in ??? () at /lib/x86_64-linux-gnu/libgomp.so.1
ngscopeclient/scopehal-apps#3  0x00007ffff4ea3d64 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:448
ngscopeclient/scopehal-apps#4  0x00007ffff4f373bc in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

For memory depth 2k, it usually works. In both modes after drawing the waveform, there is sometimes large-scale noise artefacts of the same length as the real data; not sure if those are related.

The issue does not occur when switching to 8-bit data width (but that negates the advantage of a 12 bit oscilloscope).

Let me know if you need more info.

Best regards
Andreas

[azonenberg]

What's the offset in the buffer that it complains about? Try building in DebugNoOpt configuration to get full symbols and no optimizations. Are the noise artifacts only in the second half of the waveform or the whole thing, and can you screenshot them so I can get a better idea of what they look like? What scope firmware version are you on?

My first thought is that this is related to a bug in some Siglent HD scope firmwares where the device would incorrectly report the waveform length in packet headers in samples rather than bytes; we added a workaround for this but if that workaround is triggering wrong (or broken) we might be corrupting memory or leaving some stuff uninitialized.

[andreasmatz]

Hi Andrew,
I tried to build it in the way you suggested but I think I did something wrong, now it always crashes with SIGABRT no matter if I select 8 bit or 16 bit.

/usr/include/c++/15/bits/basic_string.h:1369: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::reference std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::operator[](size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; reference = char&; size_type = long unsigned int]: Assertion '__pos <= size()' failed.

Thread 12 "ngscopeclient" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffbd5ba6c0 (LWP 87871)]
__pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
Warnung: 44	./nptl/pthread_kill.c: Datei oder Verzeichnis nicht gefunden

The noise artifacts are only in the second half of the waveform; I will rebuild and try to get a screenshot.

[azonenberg]

Moving this ticket over to scopehal as this is looking more and more like a backend bug rather than something on the GUI side.

[azonenberg]

Corruption in the second half of the waveform sounds very much like we have some code where a buffer is sized/filled with data assuming int8 rather than int16, so we either read off the end or read uninitialized garbage.

Different behavior in release vs debug builds is also consistent with memory safety bugs.

From a quick glance, the abort error in your previous comment is also consistent with that (trying to access a string with an index greater than the length). Can you backtrace and print some more data about where it is and exactly which buffer/offset it's complaining about?

[andreasmatz]

Hi,
sure - sorry forgot the bt.

(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:100
#3  0x00007ffff4845e2e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff4828888 in __GI_abort () at ./stdlib/abort.c:77
#5  0x00007ffff4cad3d8 in std::__glibcxx_assert_fail(char const*, int, char const*, char const*) () at /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00005555557c6eca in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator[](unsigned long) ()
#7  0x00007ffff62b5509 in SiglentSCPIOscilloscope::AcquireData() () at /lib/x86_64-linux-gnu/libscopehal.so
#8  0x00005555558a38d3 in ??? ()
#9  0x0000555555aa61c9 in ??? ()
#10 0x0000555555aa5f97 in ??? ()
#11 0x0000555555aa5e51 in ??? ()
#12 0x0000555555aa579c in ??? ()
#13 0x0000555555aa4724 in ??? ()
#14 0x00007ffff4cf2584 in ??? () at /lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007ffff48a3d64 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:448
#16 0x00007ffff49373bc in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Seems like I was not successful in getting full debug info. I followed the CI workflow at [1] which allowed me to successfully build. In the configure step, I modified -DCMAKE_BUILD_TYPE=Release to -DCMAKE_BUILD_TYPE=DebugNoOpt
[1] https://github.com/ngscopeclient/scopehal-apps/actions/runs/19043465206/job/54385937406

(gdb) list
End of the file was already reached, use "list ." to list the current location again
(gdb) list .
Insufficient debug info for showing source lines at current PC (0x7ffff62b5509).
(gdb) info line *$rip
Keine Zeilennummerninformationen verfügbar for address 0x7ffff62b5509 <_ZN23SiglentSCPIOscilloscope11AcquireDataEv+3117>

[azonenberg]

Hmm, I'm not sure why there's no debug symbols for libscopehal. @d235j any ideas?

We can try figuring this out with old school printf debugging as well worst case, I suppose, but it'll take a lot longer.

The first things I'd be interested in seeing are in SiglentSCPIOscilloscope.cpp around 2125 what the values of hdWorkaround, acqPoints, pageSize, pages, and acqBytes are.

[azonenberg]

You can also try just doing CMAKE_BUIILD_TYPE=DEBUG and see if that works any better

[d235j]

You can also try just doing CMAKE_BUIILD_TYPE=DEBUG and see if that works any better

You'll need either this or RelWithDebInfo. Maybe we should make RelWithDebInfo the default?

[azonenberg]

You can also try just doing CMAKE_BUIILD_TYPE=DEBUG and see if that works any better

You'll need either this or RelWithDebInfo. Maybe we should make RelWithDebInfo the default?

DebugNoOpt should be -O0 -D_DEBUG -g so I'm not sure why it's not working.

RelWithDebInfo is not the default because the binaries are enormous (libscopeprotocols is half a gigabyte) which both wastes disk space and increases application load time, so we don't want to build binaries with debug info unless you're a developer.

[andreasmatz]

Ok so DEBUG and RelWithDebInfo give me the release behavior.

(gdb) bt
#0  0x00007ffff6c1dd40 in Oscilloscope::Convert16BitSamplesGeneric(float*, short const*, float, float, unsigned long) () at /lib/x86_64-linux-gnu/libscopehal.so
#1  0x00007ffff6c1de1d in ??? () at /lib/x86_64-linux-gnu/libscopehal.so
#2  0x00007ffff677e41f in ??? () at /lib/x86_64-linux-gnu/libgomp.so.1
#3  0x00007ffff54a3d64 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:448
#4  0x00007ffff55373bc in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) list
Warnung: 1	./nptl/<built-in>: Datei oder Verzeichnis nicht gefunden
(gdb) list .
Insufficient debug info for showing source lines at current PC (0x7ffff6c1dd40).
(gdb) info line *$rip
Keine Zeilennummerninformationen verfügbar for address 0x7ffff6c1dd40 <_ZN12Oscilloscope26Convert16BitSamplesGenericEPfPKsffm+40>

Just a sanity check - I do the following, not seeing an obvious error:

  mkdir build
  cd build
  cmake \
    -DCMAKE_BUILD_TYPE=DEBUG \
    -DDISABLE_PCH=ON \
    -DBUILD_TESTING=ON \
    -GNinja \
    -DCPACK_GENERATOR=DEB \
    -DCMAKE_INSTALL_PREFIX=/usr \
    ..
ninja
ninja package

Here is the requested screenshot:

In the meantime, I will printf, see if that gets us somewhere.
PS: I think DCMAKE_BUILD_TYPE is not applied correctly - package is 19M total, nowhere nere 500MB

[azonenberg]

Oh are you installing the package rather than running out of the build directory? That would explain it, I'm pretty sure CPack is stripping. We don't have the packaging infrastructure set up for debug builds since it's assumed that developers are building from source; packages are intended for end users who aren't touching code.

Try just running build/src/ngscopeclient/ngscopeclient directly.

[andreasmatz]

Now we are getting there. Let me know what you need. Still the vars listed above?

Thread 40 "ngscopeclient" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff8c39d6c0 (LWP 112701)]
Oscilloscope::Convert16BitSamplesGeneric (pout=0x7fff9b412700, pin=0x7fffa411a390, gain=gain@entry=2.60416673e-05, offset=offset@entry=0, count=624960) at /home/andreas/Code/scopehal-apps/lib/scopehal/Oscilloscope.cpp:1286
1286			pout[j] = gain*pin[j] - offset;
(gdb) bt
#0  Oscilloscope::Convert16BitSamplesGeneric (pout=0x7fff9b412700, pin=0x7fffa411a390, gain=gain@entry=2.60416673e-05, offset=offset@entry=0, count=624960) at /home/andreas/Code/scopehal-apps/lib/scopehal/Oscilloscope.cpp:1286
#1  0x00007ffff649a966 in _ZN12Oscilloscope19Convert16BitSamplesEPfPKsffm._omp_fn.0(void) () at /home/andreas/Code/scopehal-apps/lib/scopehal/Oscilloscope.cpp:1260
#2  0x00007ffff71f241f in ??? () at /lib/x86_64-linux-gnu/libgomp.so.1
#3  0x00007ffff4ea3d64 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:448
#4  0x00007ffff4f373bc in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) l
1281		@brief Converts raw ADC samples to floating point
1282	 */
1283	void Oscilloscope::Convert16BitSamplesGeneric(float* pout, const int16_t* pin, float gain, float offset, size_t count)
1284	{
1285		for(size_t j=0; j<count; j++)
1286			pout[j] = gain*pin[j] - offset;
1287	}
1288	
1289	#ifdef __x86_64__
1290	__attribute__((target("avx2")))