`_PyEval_SetTrace` is no longer available in C API since 3.13

[dalazx]

Bug report

Bug description:

Hello,

Prior 3.13 it was possible to set a trace in a specific thread from another thread by leveraging _PyEval_SetTrace, with pure Python. Now one can either set a trace for the current thread or all the threads using PyEval_SetTrace or PyEval_SetTraceAllThreads.
The change:
https://github.com/python/cpython/pull/108433/files#diff-9e7eb7e81667528dfa3a35bbf32fe622d4721c3c71385ea0ea2250abbd15d545R21
Interestingly, the profiling counterpart seems to be still available in the C API.

The use case I am interested in is a dedicated monitoring thread connected to a distributed observability system. This thread itself should not be traced as it manages tracing for the rest of the process.

Would it be possible to reinstate _PyEval_SetTrace back to the private C API?

Also, the new sys.monitoring module seems to offer a more high-level interface for debuggers and profilers, but it is unclear whether it is possible to limit the events to a certain thread. (unless of course one checks every event which does not seem efficient).

Alternatively, would writing an extension to expose this function be a better approach?

I would greatly appreciate any comments or suggestions regarding this. Thank you!

cc @vstinner

CPython versions tested on:

3.13

Operating systems tested on:

macOS

[ZeroIntensity]

Calling _PyEval_SetTrace on a non-attached thread state is probably a thread safety headache because the fields are protected by the GIL (or by thread-localness on the free-threaded build), so I'm not sure you want to do that. If you have the pointer to the thread state that you want to mess with, you should attach to it with PyThreadState_Swap and then just call PyEval_SetTrace. Something like this should work:

static void
PyEval_SetTrace_ForThread(PyThreadState *tstate, Py_tracefunc func, PyObject *obj)
{
    // Attach to target thread
    PyThreadState *old_tstate = PyThreadState_Swap(tstate);
    PyEval_SetTrace(func, obj)
    // Restore old thread
    PyThreadState_Swap(old_tstate);
}

[gaogaotiantian]

I can answer the sys.monitoring part - no there isn't a mechanism to limit the events on a specific thread at this point. The fundamental of sys.monitoring is dynamic opcodes, which changes the code object directly so it's code object specific. Even the current sys.settrace and sys.setprofile is based on sys.monitoring, and they have an explicit check on thread-specific trace/profile funcs (the underlying sys.monitoring event will always be triggered).

[dalazx]

@ZeroIntensity thank you for your reply.

_PyEval_SetTrace does check the GIL:

/* The caller must hold the GIL */
assert(PyGILState_Check());

and prior 3.13 it seemed quite simple to manage this using ctypes:

def _settrace_thread(
    tstate: ctypes._Pointer[PyThreadState], arg: TraceFunction | None
) -> None:
    func: ctypes.c_void_p
    argument: t.Any
    if arg is not None:
        func = _c_trace_trampoline
        argument = arg
    else:
        func = ctypes.cast(0, ctypes.c_void_p)
        argument = PyObjectPtr()
    _PyEval_SetTrace(tstate, func, argument)

I presume for PyThreadState_Swap I'll need to have an extension though.

[dalazx]

@gaogaotiantian thanks. I'll study the current settrace function closely.

[ZeroIntensity]

PyGILState_Check isn't extensive. On the free-threaded build and with subintepreters, accessing another thread state is non-atomic.

I presume for PyThreadState_Swap I'll need to have an extension though.

Yeah, changing the thread state between Python calls isn't safe. Is there any reason you can't use an extension? (In my experience, ctypes is significantly more painful than just using an extension.)

[dalazx]

Is there any reason you can't use an extension?

I thought it might slightly complicate package distribution with platform-specific wheels etc.

[ZeroIntensity]

I guess it would, but ctypes will open up a new complicated can of worms for debugging. IMO, if we really want to use ctypes, the best way to proceed here is not to reinstate _PyEval_SetTrace, but to just add some sort of mechanism into ctypes for changing the thread state. I might play around with developing that later today if I have time.

[dalazx]

thanks @ZeroIntensity.

do you mean trying to set the fields below directly using ctypes?

cpython/Python/legacy_tracing.c

Lines 577 to 582 in 5c89adf

	tstate->c_tracefunc = func;
	*old_traceobj = tstate->c_traceobj;
	tstate->c_traceobj = Py_XNewRef(arg);
	tstate->interp->sys_tracing_threads += delta;
	assert(tstate->interp->sys_tracing_threads >= 0);
	return tstate->interp->sys_tracing_threads;

[ZeroIntensity]

Nah, I mean actually trying to change Python's thread state form ctypes. It's not possible right now because the eval loop relies on it, but in theory some sort of wrapper that changes the thread state before calling into C could work. If I can come up with a reference implementation, I'll bring it to DPO.