Python's Tcl interpreter does not link shared objects with Tcl library by default

[MayeulC-ST]

Bug report

Bug description:

Hello,

I am using Python's Tcl interpreter roughly as suggested in https://wiki.tcl-lang.org/page/Python-Tcl-Interactions

import tkinter
interpreter = tkinter.Tcl()
interpreter.eval('puts "hello from Tcl"')

And everything works fine. However, this is a large Tcl legacy codebase, and some procedures are provided as shared objects that are loaded with Tcl's load procedure:

load -global -- /path/to/shared_object.so myso

This works for some shared objects, but not all: some imports fail with

E   _tkinter.TclError: couldn't load file "/path/to/shared_object.so": /path/to/shared_object.so: undefined symbol: Tcl_NewStringObj

I have reproduced this error by compiling a sample shared object and omitting -ltcl when compiling it:

// Example inspired from https://www.tcl-lang.org/man/tcl/TclCmd/load.htm
// Compile with: gcc -shared -o tclext.so -fPIC tclext.c -ltcl
// load in tclsh with: load -- tclext.so Foo
// Then you can call the command foo
#include <tcl.h>
static int fooCmd(void *clientData,
        Tcl_Interp *interp, int objc, Tcl_Obj *const objv[]) {
    // Returns a Tcl string "called with %d arguments\n"
    Tcl_Obj *result = Tcl_NewStringObj("called with ", -1);
    Tcl_AppendPrintfToObj(result, "%d", objc);
    Tcl_AppendToObj(result, " arguments\n", -1);
    Tcl_SetObjResult(interp, result);
    return TCL_OK;
}
int Foo_Init(Tcl_Interp *interp) {
    if (Tcl_InitStubs(interp, "8.1", 0) == NULL) {
        return TCL_ERROR;
    }
    // Create foo command
    Tcl_CreateObjCommand(interp, "foo", fooCmd, NULL, NULL);
    return TCL_OK;
}

In python, the above can be loaded just fine with:

import tkinter as tk
tcl = tk.Tcl()
tcl.eval('''
load ./tclext_str.so Foo
set abc [foo]
puts "abc contains ``$abc''"
''')

After compiling the .so with:

gcc -shared -o tclext_str.so -fPIC tclext_str.c -ltcl 

However, Python fails to load the .so (with the aforementioned error) when it is compiled without -ltcl:

gcc -shared -o tclext_str.so -fPIC tclext_str.c

The same load statement works perfectly from tclsh.

My suspicion is that tclsh implicitly links everything with the tcl library, while this is not the case for Python's Tcl interpreter. I think this should be fixed, if possible.

As a workaround, I can patchelf --add-needed libtcl.so tclext_str.so but I am unsure about the drawbacks, and this difference in behavior remains problematic for sharing a unified codebase.

CPython versions tested on:

3.9, 3.10, 3.11, 3.12

Operating systems tested on:

Linux

[ned-deily]

Python and Tcl are both general-purposes languages with large legacy environments with many overlaps in functionality. The only reason Python provides access to Tcl is to support Python's tkinter use of Tk as a GUI. We make no claims to provide access to a full-fledged Tcl installation through Python nor do we test that. And what versions of Tcl and Tk, and what parts of them, are available with a particular instance of Python is up to the distributor or builder of Python. So, in general, if your goal is, as you state, to run a large Tcl legacy codebase, it would be safer and likely less problematic to run such code directly via a full Tcl installation and not indirectly through tkinter, for example, by "shelling out" to tclsh from within Python.

[ned-deily]

Of course, both Python and Tcl are open-source projects so another option might be to build a dedicated Python / Tcl environment from source that provides all of the components and linkage you need to run the codebase.

[MayeulC-ST]

Thank you for the quick answer, I understand your reservations. Of course, Python has no obligation to provide an environment that is 100% compatible with existing Tcl codebases. At least this present issue will now come up in search results to help document that caveat, so feel free to close this ticket if you do not want to investigate further.

It may however lead to unexpected behavior and slowness (falling back to pure-Tcl instead of compiled functions) when loading existing tk extensions.

---

As for my use-case, that is a bit specific, but I:

• Developed a pytest extension for Tcl (which I hope to open source at some point) -- this is how I came across this issue
• Would like to fully migrate to Python eventually, which is helped by the two-way interoperability provided by tkinter.

I can imagine compiling my own tclsh embedding CPython for better interoperability, or some other approach, but I don't think I have enough bandwidth for this. I think improving tkinter instead would also benefit more people.

As an aside, do you deliberately limit the scope of tkinter (for instance by not providing convenience interfaces to transfer lists, dicts, arrays, etc from/to Python), and would you accept contributions to expand it?

It would be safer and likely less problematic to run such code directly via a full Tcl installation

I didn't have high expectations at first, but it turns out the Python-provided Tcl interpreter was able to handle everything I threw at it except for this issue (admittedly, I might find more later).

And what versions of Tcl and Tk, and what parts of them, are available with a particular instance of Python is up to the distributor or builder of Python

Interesting, that may indeed be an issue for distribution in some cases (though not in controlled environments). And the fix may well belong to these parts.

---

For now, I think I will go with patchelf regardless. If I ever find a simple fix for this behavior, would you accept it? Though I am not sure where to start looking.

[ned-deily]

For now, I think I will go with patchelf regardless. If I ever find a simple fix for this behavior, would you accept it?

It's hard to say upfront without a concrete proposal. Factors would include things like maintenance impact, testability, applicability across the various platforms Python (and Tk) supports (i.e. Linux, Windows, macOS, etc). But I think we would consider a pull request if submitted. By the way, for Unix-y builds, the places to start looking for how Python decides to link things would be configure.ac, which is used to produce configure, and Makefile.pre.in, which is used to produce Makefile. Windows builds use a different build system. There is some information in the Python Developer's Guide about all this that might be of use. Good luck!

[MayeulC-ST]

I found a nicer workaround: unconditionally ask the interpreter to:

load -global libtcl.so

After instantiating it. This easy to carry independently, and does not require patching shared objects, so I can live with this.

I am still puzzled as to why this is needed at all: these symbols are obviously already available to _tkinter, which uses them:

cpython/Modules/_tkinter.c

Line 615 in e62e1ca

value = Tcl_NewStringObj("true", -1);

On my system, it seems that the corresponding lib-dynload/_tkinter.cpython-310-x86_64-linux-gnu.so is dynamically linked with the system's libtcl.so, so loading it unconditionally on initialization should not cause an issue (if tkinter can be imported, it means that libtcl.so is present on the system).

Feel free to close this issue if you don't think this is an avenue worth exploring. At least, there is now a simple workaround documented somewhere :)

[MayeulC-ST]

I just found another possible workaround:

sys.setdlopenflags(os.RTLD_NOW | os.RTLD_GLOBAL)
import tkinter

Setting these flags before importing tkinter also works (only setting the second flag leads to an error). It may be worth documenting, or setting these flags by default for tkinter?

[ned-deily]

As there is no guarantee that there is a workaround for this situation that is generally applicable and since it seems you have an acceptable workaround for your use case, I am going to close this issue as not planned. Feel free to reopen if there are further developments.