error: "cysignals must be compiled without _FORTIFY_SOURCE"

[orlitzky]

#73 is back for the third time. When building with clang on Gentoo,

$ CC=clang python setup.py build
running build
running build_py
...
In file included from build/src/cysignals/signals.c:2365:
build/src/cysignals/implementation.c:27:2: error: "cysignals must be compiled without _FORTIFY_SOURCE"
#error "cysignals must be compiled without _FORTIFY_SOURCE"

The constant __USE_FORTIFY_LEVEL=3 is built in to the compiler, so the two different hacks that we have to work around this are ineffective (and yes, the result still crashes if I remove the guard). I have yet to find a third workaround.

This is also Gentoo bug 918934.

[user202729]

For what it's worth I use the workaround explained in #80 and it works.

$ CC=clang CFLAGS="-Wp,-U_FORTIFY_SOURCE" pip install cysignals

Not sure how to specify compiler flags in pyproject.toml or meson.build etc.

[orlitzky]

Does it fail without those CFLAGS? You would need a clang where the source fortification constant is built in to the compiler to reproduce it. (You also need glibc, in case you are on Void, Alpine, Gentoo, or some other distro that offers musl.)

longjmp seems to be causing us a lot of problems:

• this
• Incompatibility with signal handling on Windows (@culler mentioned this on the mailing list last month)
• Random segfaults in Sage (discussion):
  • Segfault testing src/sage/libs/gap/element.pyx on python 3.12 sage#37026
  • Segfaults in sage.libs.gap.element sage#37289
  • Random Segfaults in matrix_gap sage#37295

[orlitzky]

With newer compilers 🤦:

 * QA Notice: Package triggers severe warnings which indicate that it
 *            may exhibit random runtime failures.
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:19608:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:19267:13: warning: variable '__pyx_v_atexit' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:19272:13: warning: variable '__pyx_t_1' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:19273:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:19033:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:18198:13: warning: variable '__pyx_t_1' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:18199:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:18200:13: warning: variable '__pyx_t_3' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:24449:23: warning: variable 'exc_info' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:17813:13: warning: variable '__pyx_r' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:17815:13: warning: variable '__pyx_t_1' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:17816:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:24449:23: warning: variable 'exc_info' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:17348:13: warning: variable '__pyx_t_1' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:17349:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:17350:13: warning: variable '__pyx_t_3' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:24449:23: warning: variable 'exc_info' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:16134:66: warning: variable '__pyx_v_x' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:16015:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:16017:13: warning: variable '__pyx_t_4' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:14482:8: warning: variable '__pyx_v_i' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:14488:8: warning: variable '__pyx_t_3' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:24449:23: warning: variable 'exc_info' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:13691:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:13193:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:12783:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:12659:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:12407:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:12155:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:11903:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:11651:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:11399:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:10822:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:10092:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:9788:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:9642:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:8278:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:8026:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:7783:22: warning: variable '_save' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:16943:13: warning: variable '__pyx_t_1' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:16944:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:16945:13: warning: variable '__pyx_t_3' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:24449:23: warning: variable 'exc_info' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:14189:13: warning: variable '__pyx_t_1' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:14190:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:14191:13: warning: variable '__pyx_t_3' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:24449:23: warning: variable 'exc_info' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:15659:13: warning: variable '__pyx_t_1' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:15660:13: warning: variable '__pyx_t_2' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:15675:13: warning: variable '__pyx_t_17' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:15676:13: warning: variable '__pyx_t_18' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:15677:13: warning: variable '__pyx_t_19' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:24449:23: warning: variable 'exc_info' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]
 * src/cysignals/tests.cpython-312-riscv64-linux-musl.so.p/src/cysignals/tests.pyx.c:10445:13: warning: variable '__pyx_r' might be clobbered by 'longjmp' or 'vfork' [-Wclobbered]

[dimpase]

are these compiler warnings meaningful? I see gcc having a long-standing issue with them
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21161

[dimpase]

At least a part of these random bugs in libgap interface are due to missing volatile declarations. Newer compilers optimise more of function calls, passing arguments in registers, which get clobbered on longjumps. But they may hold a reference to a freshly created GAP object, oops...

[orlitzky]

I think they are meaningful in this case and illustrate nicely the fundamental problem. All cython functions that eventually call sig_on() and sig_off() get some cython boilerplate like,

static PyObject *blahblahblah(...) // the C implementation of test_signal_during_malloc()
  PyObject *__pyx_r = NULL;
  __Pyx_RefNannyDeclarations
  PyObject *__pyx_t_1 = NULL;
  PyObject *__pyx_t_2 = NULL;
  PyObject *__pyx_t_3 = NULL;
  int __pyx_t_4;
  int __pyx_lineno = 0;
  const char *__pyx_filename = NULL;
  int __pyx_clineno = 0;
  ...

before they eventually get to the sig_on(). And this is for a function that declares no local variables! How will we ever declare those volatile? There's no special data flow that invalidates the warning. I think those variables can really be clobbered, and the warning is emphasizing how hard it will be to properly use setjmp/longjmp from cython.

[user202729]

Actually it is not necessary to declare everything volatile, only the local variables that are changed between setjmp and longjmp need to be declared volatile.

I don't know what these temporary _t_ variables are used for (probably for temporary variables), but worst case their values are indeterminate.

But, yes, ideally the whole toolchain (cython, gcc) would be setjmp/longjmp-aware.

[dimpase]

Actually it is not necessary to declare everything volatile, only the local variables that are changed between setjmp and longjmp need to be declared volatile.

I don't know what these temporary _t_ variables are used for (probably for temporary variables), but worst case their values are indeterminate.

But, yes, ideally the whole toolchain (cython, gcc) would be setjmp/longjmp-aware.

the point is that with this problem, things like
libgap.Foo(libgap.Baz()) are not safe, as Foo() may throw a GAP error, and this will clobber the pointer to the result of Baz() kept in a register.

[user202729]

But if you call libgap.Foo(libgap.Baz()) and Foo throws an error then the result of Baz is discarded anyway?

Are there guards to delete them afterwards somewhere?

[orlitzky]

But if you call libgap.Foo(libgap.Baz()) and Foo throws an error then the result of Baz is discarded anyway?

It isn't that we need the value of Baz(), but that if you clobber the pointer to it with random data, you'll get a segfault when you try to free() it. (The fact that a pointer is used here should be an internal cython detail.) In a more complicated example you might actually try to use the return value, which could dereference the nonsense pointer. In python you should be able to catch the error and respond intelligently with something like,

x = libgap.Baz()
try:
    y = libgap.Foo(x)
except:
    # x might be garbage!
    print("Foo error with x=", x)

There are actually two closely-related problems here, since libgap uses setjmp/longjmp internally too. In either case, the situation really sucks for the caller because he has to know where and how setjmp/longjmp are used in these third-party libraries in order to use them safely. And even beyond that, he has to know what C code cython is going to generate, to make sure that its internal details can't be clobbered by longjmp, too.

[dimpase]

these (needing volatile) only started manifest themselves as segfaults (rather than just memory leaks - or perhaps these weren't even memory leaks, and were garbage-collectable by GAP? Or perhaps this still was leaving refcounted "handles" meant to be pointing to GAP objects created in libgap Cython interface behind) since Python 3.12, which changed the codepath for Pythnon error handling.

[culler]

This discussion does not match my understanding of how sig_on and sig_off are used by sage.

The primary use case is making it possible for Sage to exit cleanly after a segfault occurs in an external library function.

In particular, Sage's longjmp call should never occur in any sort of normal error processing or exception handling by, say, libgap. The call to longjmp and subsequent variable clobbering should only happen if there is a signal which is not handled by any code in the external library (usually SIGSEGV).

Is there an example code snippet that is known to generate the segfault with libgap that is being discussed here? I suspect that real problem is a seqfault in libgap, not some design flaw in sig_on or sig_off. Of course compiler errors would be a different story.

[dimpase]

GAP uses longjmp to handle its errors.
Handling nested setjmp/longjmp pairs is fun.

See e.g.
sagemath/sage#37026
and
sagemath/sage#37951

[culler]

I am still not seeing any evidence that sig_on and sig_off are involved in these crashes. A setjmp with no corresponding longjmp should be harmless. And Sage only calls longjmp from signal handlers installed by sig_on. So Sage's longjmp should only be able to cause trouble in the case when a signal is being generated for which sig_on had installed a handler. If GAP is calling longjmp from its own signal handler then it would have had to install that handler on top of Sage's, and Sage's handler would not be invoked. If GAP is calling longjmp in some other way then Sage's longjmp should not get called. The crash suggests to me that there is a SIGSEGV signal being generated by the GAP code. The fact that this SIGSEGV is being handled by the handler installed by sig_on only means that sig_on is working as it was designed to work.