fix(TTk): register signal event callbacks on init to avoid exception

[slook]

TTk: fix exception when invoking mainloop() from a thread

It is necessary for me to invoke mainloop() from a different thread because the core part of my application has to run in the main thread, and these two calls (to TTkSignalDriver.init() and TTkTerm.registerResizeCb()) were the only things which prevented that arrangement from being possible due to this error message on startup:

Caught an exception: signal only works in main thread of the main interpreter (no stacktrace)

If you require further clarification or explanation about this edge case then please do let me know. I think this should be a harmless change, and is a benefit to enable more flexible coroutine invocation methods for various purposes.

Thank you very muchly for your time and efforts :)

---

Update:
With later fixes found after this workaround enabled to get a stacktrace of what was happened underneath:

user@ThinkPadX220:~/Git/slook/nicotine-plus$ ./nicotine --tui
Caught an exception: signal only works in main thread of the main interpreter <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>
Traceback (most recent call last):
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/TermTk/TTkCore/ttk.py", line 164, in _mainloop_1
    TTkSignalDriver.init()
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/TermTk/TTkCore/drivers/unix.py", line 83, in init
    signal.signal(signal.SIGTSTP, TTkSignalDriver._SIGSTOP) # Ctrl-Z
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/signal.py", line 56, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: signal only works in main thread of the main interpreter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/TermTk/TTkCore/ttk.py", line 144, in mainloop
    self._mainloop_1()
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/TermTk/TTkCore/ttk.py", line 198, in _mainloop_1
    TTkSignalDriver.exit()
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/TermTk/TTkCore/drivers/unix.py", line 88, in exit
    signal.signal(signal.SIGINT,  signal.SIG_DFL)
  File "/usr/lib/python3.11/signal.py", line 56, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: signal only works in main thread of the main interpreter
user@ThinkPadX220:~/Git/slook/nicotine-plus$ 

signal is actually nothing to do with the actual cause of the crash as you can see there was a bug in my program that the later errors in your library did hide but now they can be seen so I already fixed them so its okay only for future reference.

Of course, I had no way of knowing that before because this crash wipes out any other traceback created by my program.

[ceccopierangiolieugenio]

I think the easiest solution is to split the main loop in 2 methods,
an initialisation and a loop,
you can run the initialisation in the main thread and the loop in a separate thread.

[ceccopierangiolieugenio]

you would use the code like this:

from threading import Thread

import TermTk as ttk

root = ttk.TTk()
win = ttk.TTkWindow(parent=root, size=(50,20), layout=ttk.TTkGridLayout())
ttk.TTkLogViewer(parent=win)

root.main_init()

def main_thread():
    root.main_run()

app = Thread(target=main_thread)
app.start()

app.join()

but an exception is raised when ctrl-c is captured

[slook]

Yes you have the right idea about having the capability to instantiate the toolkit in its own thread. However your example is misleading to name the threaded target "main_thread" since that is the name by which the underlying program codebase already uses for core modules which run plainly unthreaded, so I have chosen to use the name "screen" to identify the user interface because that matches the naming convention that curses tends to use.

The exception upon pressing Ctrl-C is not so much of an issue because at least it doesn't prevent startup and that exception can be avoided by setting a sigmask if such key capturing isn't required anyway (shortcut will be used for the Quit menu item). I suppose the key capturing emits could be registered with a threaded handler callback such that you have already done for the window resizing signal.

[slook]

Your above example is somewhat similar to what I had, which shows we are thinking along the same lines, but I've subclassed TTk in a similar way to how Gtk does. Not saying this is the best or only way, just the way I ended up with a working solution in my particular case where it is necessary to "retro-fit" the toolkit onto an already existing program that can't be easily changed.

Ideally, the toolkit library would have a built-in method to handle creating a new thread for itself automatically, but that would be another matter for you consider entirely as to if that is really a good idea or not. This does get both the user interface and the program core started cleanly so that different modules can interact with eachother...

Example mock-up derived from the ttk branch of my development fork for a network client ...

from threading import Thread
from time import sleep

from TermTk.TTkCore.ttk import TTk

from core import core
from core.events import events
from core.ttktui.mainscreen import MainScreen


class MainScreen(Thread):
"""Threaded root widget that can be invoked with start() and then updated afterwards."""

    def __init__(self, application):
        super().__init__(target=application.mainloop, daemon=True)

        from TermTk.TTkCore.ttk import TTkAppTemplate, TTkGridLayout

        application.setLayout(ttk.TTkGridLayout())
        self.container = ttk.TTkAppTemplate(parent=application)
        self.root = application  # TTk()

    def destroy(self):
        raise SystemExit


class Application(TTk):
"""Text User Interface for a program with an interactive core process."""

    def __init__(self):

        # Initialize the toolkit to register its signal handlers in the main thread
        super().__init__(title="Application", sigmask=(TTkTerm.Sigmask.CTRL_C | TTkTerm.Sigmask.CTRL_Q |
                                                       TTkTerm.Sigmask.CTRL_S | TTkTerm.Sigmask.CTRL_Z))
        self.screen = MainScreen(self)

        # Receive core events emitted from the main thread
        for event_name, callback in (
            ("quit", self.quit),       # The core tells the application window when to quit
            # ("task", self.on_task),  # and meanwhile does lots of stuff in the background
            # ("...", self.on_and_so_on)
        ):
            events.connect(event_name, callback)

    def run(self):
        """Entry point for threaded application interface."""

        # Run the interface in a separate thread
        self.screen.start()

        # Start running the program in the main thread
        core.start()

        # Main loop, send events from the program's main thread 10 times per second
        while events.process_thread_events():
            sleep(0.1)

        # Shut down with exit code 0 (success)
        return 0

    def quit(self):
        self.screen.join(timeout=1)
        if self.screen.is_alive():
            self.screen.destroy()
        ttk.TTkHelper.quit()


# Invoke this from your program's core process in the main thread
root = Application()
return root.run()

As you can see, the client already has its own "main loop" and threaded event emitter so it seems like a good idea to use a terminology other than "main thread" to avoid ambiguity with core activities that need to run directly on the Python interpreter.

If you are interested about my experimental implementation attempt, you can take a look at the ttk branch of my Nicotine+ fork (a network client), which may be run as binary packages built with various Actions runners (I've not tested those yet) or directly from a Git folder with:

git clone --branch ttk --depth 1 https://github.com/slook/nicotine-plus.git
cd nicotine-plus
./nicotine --tui  # use the TTk TUI instead of the default Gtk GUI

The TermTk interface is in the "pynicotine/ttktui" folder of the project (its an interesting experiment to test your library).

It could be helpful to bring you some ideas about how developers might choose to implement your library into larger programs that take a modular approach to separate the core from the UI(s), albeit perhaps in ways that maybe weren't originally intended!

It could make sense to expose run() and start() methods to invoke threaded startup of the toolkit, if desired, and keep mainloop() for platforms such as WebAssembly where the threading module cannot be used and in ordinary cases where the program is baked into the interface.

Ideally, it should be possible to properly register the relevant signal handlers upon initializing the TTk() class (either directly or by subclassing it), without needing to manually call a main_init() method.

[ceccopierangiolieugenio]

I am interested in exploring the different ways to integrate my library with different projects.
I think I can expose a start and run methods.

where I can call the initialise routine in the start.
the problem is that in start I need to call super().start() so I need to check also if self is an instance of Thread.

Anyway, the reason I am forcing to run it in the main thread is mainly because I didn't thought about a different implementation and also because QT and GTK seems to use the same approach.

[ceccopierangiolieugenio]

I added the start run and ttk_init methods and it works except the ctr-c issue (that I will investigate)

This is something like you were thinking?

import sys, os
from threading import Thread

import TermTk as ttk

root = ttk.TTk()
win = ttk.TTkWindow(parent=root, size=(50,20), layout=ttk.TTkGridLayout())
ttk.TTkLogViewer(parent=win)

root.ttk_init()

def ttk_thread():
    root.run()

app = Thread(target=ttk_thread)
app.start()
app.join()

Threaded reimplementation:

import sys, os
from threading import Thread
from datetime import datetime

import TermTk as ttk

class ThreadedTTk(ttk.TTk, Thread):
    pass

root = ThreadedTTk()
win = ttk.TTkWindow(parent=root, size=(50,20), layout=ttk.TTkGridLayout())
ttk.TTkLogViewer(parent=win)

root.start()
root.join()

Or even better, I can introduce a ThreadedTTk class that would overcome a little hack I added to the main TTk.

can you point me out the gtk reference (in the gtk doc or forums) that you were mentioning?

[slook]

Gtk.Application() https://docs.gtk.org/gtk4/class.Application.html ...
"... it handles GTK initialization, application uniqueness, session management, provides some basic scriptability and desktop shell integration by exporting actions and menus..."

https://gnome.pages.gitlab.gnome.org/pygobject/tutorials/gtk4/introduction.html#extended-example

import gi

gi.require_version("Gtk", "4.0")
from gi.repository import Gtk


class MyWindow(Gtk.ApplicationWindow):
    def __init__(self, **kargs):
        super().__init__(**kargs, title="Hello World")

        self.button = Gtk.Button(label="Click Here")
        self.button.connect("clicked", self.on_button_clicked)
        self.set_child(self.button)

    def on_button_clicked(self, _widget):
        self.close()


def on_activate(app):
    # Create window
    win = MyWindow(application=app)
    win.present()


app = Gtk.Application(application_id="com.example.App")
app.connect("activate", on_activate)

app.run(None)

https://developer.gnome.org/documentation/tutorials/application.html
"A Gio.Application() can register a set of actions that it supports in addition to the default activate and open... calling action_group.activate_action() on the primary instance will activate the named action in the current process.."

https://gnome.pages.gitlab.gnome.org/pygobject/index.html
"PyGObject uses GLib, GObject, GIRepository, libffi and other libraries to access the C library (libgtk-4.so) in combination with the additional metadata from the accompanying typelib file (Gtk-4.0.typelib) and dynamically provides a Python interface based on that information..."

class MyApplication(Gtk.Application):
    def __init__(self):
        super().__init__(application_id="com.example.MyGtkApplication")
        GLib.set_application_name("My Gtk Application")

    def do_activate(self):
        window = Gtk.ApplicationWindow(application=self, title="Hello World")
        window.present()


app = MyApplication()
exit_status = app.run(sys.argv)
sys.exit(exit_status)

I can introduce a ThreadedTTk class

I'd suggest in future you might do a combination of other asynchronous methods depending on the platform, and so I advice against using the term "Threaded" in case that causes confusion about its purpose. It would be ideal if the working mechanics were abstracted away from the developer so they don't have to worry about managing threads or choosing from various methods invocation. In other words, the TTk() class should be able to do it all no matter what the runtime environment is like.

Since TTk seems to only be capable of making full-screen applications then I think calling it "TTkScreen()" or "TTkApp()" or "TTkApplication()" or "TTkRoot()" would be appropriate as the base root class widget, and it is reasonable to enforce TTkGridLayout() or even TTkAppTemplate() as the default so then the developer can skip straight to adding widgets on the screen.

except the ctr-c issue (that I will investigate)

It might be helpful for you to know that some (but not all) Slots and Signals are also affected by the same crash such as connecting TTkButton() and TTkKeyEvent() don't work, yet I have got the TabBar and TextEditView working fine without errors.

the problem is that in start I need to call super().start()

You don't need a start() function becauuse Python handles calling your run() function by itself when you subclass the Thread() class object then calling start() "arranges for the object’s run() method to be invoked in a separate thread of control." For example with this PR it is possible to make a top level root screen application widget like this:

https://github.com/slook/nicotine-plus/blob/ttk/pynicotine/ttktui/widgets/screen.py

from threading import Thread


class Screen(Thread):

    def __init__(self, root):
        super().__init__(target=root.mainloop, daemon=True)
        self.root = root  # TTk()

    def destroy(self):
        raise SystemExit

https://github.com/slook/nicotine-plus/blob/ttk/pynicotine/ttktui/application.py#L48

        self.screen.start()

That works nicely even though the Screen(Thread) class has no such function called start() in it...

[ceccopierangiolieugenio]

main_thread was just an example, ThreadedTTk same , it was just the idea of introducing a thread extension for this purpose I would have not used those names.
About the other signatures,
I know that many design decision are questionable,
but I could not have predicted everything when I started 5 years ago,
I used the QT api and structure as reference and I would break compatibility with older apps if I decide to change the name of the main class.

Anyway, I am not sure what you mean about my library being only able to make full screen apps.
I know that it is technically possible to create apps that occupy a partial strip of the terminal, but any TUI just keeps full control of the terminal area and draws on it, it is not possible to do something in between like spawning a window.

About the start
I need to initialise the signals in the main thread not during the TTk initialisation.
this is the reason of my previous examples:

class TTk():

    def mainloop(self) -> None:
        self.ttk_init()
        self.run()

    def ttk_init(self) -> None:
        # Signal and callbacks registration

    def start(self) -> None:
        self.ttk_init()
        # if it is an istance of Thread
        super().start()

    def run(self) -> None:
        # loop routine

[slook]

Your strategy seems sound, but I think implementing a class overloading approach would suit the rest of the library better for this use case. Essentially like you say, making a core widget to run threaded if it makes sense to do so and the system can allow it.

# if it is an istance of Thread

We can determine if the caller needs wants this case using this guard condition...

if threading.current_thread() is not threading.main_thread():
    # This is already an instance of Thread so run() it
    self.start()
else:
    # We need to launch a threaded instance for the caller
    screen = ScreenThread(self)
    screen.start()

... or maybe elif not isinstance(threading.main_thread(), TTk): (Idk, I've never tried doing this...)

https://docs.python.org/3/library/threading.html#threading.current_thread
https://docs.python.org/3/library/threading.html#threading.main_thread
"Return the main Thread object. In normal conditions, the main thread is the thread from which the Python interpreter was started."

However, the only thing that is for certain is that we're operating well outside of "normal conditions" here :D

I am not sure what you mean about my library being only able to make full screen apps.

There isn't any examples or tutorials of simple command line programs in your repo so I made an assumption that you are focused on specializing mainly in full-screen programs, whereas another library such as python-prompt-toolkit, Rich or Textual perhaps seem more suited to partial non-alternate screen REPL interfaces like the progress bars used in pip or brew and such like, but they are way too big and bloated due to having many awkwardly packaged dependencies which puts me off most of the well known ones that are around at the moment.

it is not possible to do something in between like spawning a window

Indeed. The best I have ever been able to do is modifying the last printed line but I agree that going any further up the readback is asking for trouble. In any case the fact you have the TTkTerminal widget kind of makes it irrelevant for the TermTk library to support anything other than full screen applications, and that's totally fine to focus on that because what it does do it does really well and considering it is without curses and yet has a great raw terminal style, just the way it should be. In fact I reckon it can be safer to commit to always using the alternate full screen buffer mode to take control of the entire terminal session so as to not disrupt the standard input/output content of the calling terminal at exit, or at least tidying up afterwards if possible.

I know that many design decision are questionable, but I could not have predicted everything when I started 5 years ago

Heck, fair play to you... I didn't know it is all your own project from the very beginning and implementing a full blown API that follows modern programming conventions in that time is quite a large achievement to have made so you must really know your ANSI escape sequences and have done plenty of amazing hacks like you're some kind of mathematical genius or what?!

[slook]

See related upstream issue python/cpython#139391 and PR python/cpython#139858

and python/cpython#81269 and python/cpython#83223 about signal vs threading depends when imported.

Update: signal is irrelevant only during the handling of unrelated errors see stacktrace in OP there are clues for future reference. The change in this PR is still required nonetheless because the library wouldn't allow the program to start in another thread otherwise.

Update: I pushed a related change to move the _paintEvent.set() statement into the mainloop to ensure that _drawMutex lock is always acquired by the runtime thread. This helps to avoid making garbage on the normal screen when terminating.

[slook]

I added the start run and ttk_init methods and it works except the ctr-c issue (that I will investigate)

@ceccopierangiolieugenio I have what I think is a good solution for that in this PR. Perhaps we can invoke other exit() methods always in the main thread by registering them at first with atexit like I've done in 4f6526a. I hope that is helpful for your investigations into this. I appreciate your time, many thanks.

Update: Pushed latest changes from main.

--

TTk 0.5.0

user@ThinkPadT420:~/Git/slook/nicotine-plus$ ./nicotine --tui
Using TTk executable: /home/user/GiTraceback (most recent call last):xternal/pyTermTk/TermTk
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/pyTermTk/TermTk/TTkCore/ttk.py", line 204, in _mainloop
    TTkSignalDriver.init()
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/pyTermTk/TermTk/TTkCore/drivers/unix.py", line 83, in init
    signal.signal(signal.SIGTSTP, TTkSignalDriver._SIGSTOP) # Ctrl-Z
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/signal.py", line 56, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: signal only works in main thread of the main interpreter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/pyTermTk/TermTk/TTkCore/ttk.py", line 184, in mainloop
    self._mainloop()
  File "/home/user/Git/slook/nicotine-plus/pynicotine/external/pyTermTk/TermTk/TTkCore/ttk.py", line 233, in _mainloop
    self._timer.join()
  File "/usr/lib/python3.11/threading.py", line 1107, in join
    raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started