Getting a proper LogView to avoid Client has been deleted but is still being used

[WolfgangFahl]

Question

The log handling with ui.log recently gave me trouble recently with Client has been deleted but is still being used see WolfgangFahl/nicegui_widgets#70.

I would love to properly handle this and for the time being simply try to switch off and logging via ui.log and only keep the fallback handler which logs to stderr so that i can continue debugging. This implementation seems to still not be fail safe. So how can i check whether ui.log can be pushed to? Would it be a feature request to make sure ui.log.push() checks this for itself?

File "/home/wf/source/python/ngdemo/.venv/lib/python3.10/site-packages/nicegui/client.py", line 334, in check_existence
    helpers.warn_once('Client has been deleted but is still being used. '
  File "/home/wf/source/python/ngdemo/.venv/lib/python3.10/site-packages/nicegui/helpers.py", line 20, in warn_once
    log.warning(message, stack_info=stack_info)
  File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/wf/source/python/ngdemo/.venv/lib/python3.10/site-packages/ngwidgets/log_view.py", line 106, in emit
    self.ui_log.push(formatted_msg)

log_view module from https://github.com/WolfgangFahl/nicegui_widgets/blob/main/ngwidgets/log_view.py

"""
Created on 2023-11-15

A custom logging handler that emits messages to a NiceGUI log element and
a standard logger.

@author: wf
"""

import logging
import sys
from typing import List, TextIO
from nicegui import ui

class LogView:
    """
    Handle logging by delegating to a ui.log
    """

    def __init__(
        self,
        max_lines: int = 100,
        classes: str = "w-full",
        level: int = logging.NOTSET,
        log_stream: TextIO = sys.stderr,
    ):
        self.loggers: List[logging.Logger] = []
        self.ui_log = ui.log(max_lines=max_lines).classes(classes)
        self.handler = UiLogHandler(self.ui_log, level, log_stream)

    def push(self, msg: str):
        self.ui_log.push(msg)

    def setLevel(self, level: int):
        self.handler.setLevel(level)
        for logger in self.loggers:
            logger.setLevel(level)

    def addAsHandler(self, logger: logging.Logger):
        if logger in self.loggers:
            return
        self.loggers.append(logger)
        if self.handler not in logger.handlers:
            self.handler.addAsHandler(logger)

class UiLogHandler(logging.Handler):
    """
    A logging handler that emits messages to a NiceGUI log element.
    Uses a fallback logger to make sure messages do not get lost when the log element is not available
    """

    LOG_LEVEL_ICONS = {
        logging.DEBUG: "🐞",
        logging.INFO: "ℹ️",
        logging.WARNING: "⚠️",
        logging.ERROR: "❌",
        logging.CRITICAL: "🔥",
    }

    def __init__(
        self,
        ui_log: ui.log,
        level: int = logging.NOTSET,
        log_stream: TextIO = sys.stderr,
    ):
        """
        Constructor

        Args:
            ui_log (ui.log): The NiceGUI log element to which messages will be emitted.
            level (int): The logging level for this handler. Default is logging.NOTSET.
            log_stream (TextIO): The stream to use for fallback logging. Default is sys.stderr.
        """
        super().__init__(level)
        self.ui_log = ui_log
        self.fallback_handler = logging.StreamHandler(log_stream)
        self.fallback_handler.setLevel(logging.DEBUG)

    def addAsHandler(self, logger: logging.Logger):
        """
        add me as a handler to the given logger

        Args:
            logger (logging.Logger): the source logger

        """
        if self not in logger.handlers:
            logger.addHandler(self)

    def emit(self, record: logging.LogRecord):
        """
        Emit a record to the NiceGUI log element
        and a fallback standard logging.

        Args:
            record (logging.LogRecord): The log record to be emitted.
        """
        level_icon = self.LOG_LEVEL_ICONS.get(record.levelno, "")
        if record.msg:
            record.msg = f"{level_icon}:{record.msg}"
        self.fallback_handler.emit(record)
        formatted_msg = self.format(record)

        if self.ui_log is not None:
            try:
                self.ui_log.push(formatted_msg)
            except Exception as ex:
                self.ui_log=None
                self.fallback_handler.handleError(record)
                self.fallback_handler.emit(f"ui.log failure: {str(ex)} - switching off ui log ...")
                pass

    def setLevel(self, level):
        """
        Set the logging level of this handler.

        Args:
            level (int): The logging level to set.
        """
        super().setLevel(level)

[falkoschindler]

I think the problem isn't that you're pushing to ui.log, but that you're still holding a reference to it. Your UiLogHandler seems to be a long-living object holding a reference to a UI element. This causes a memory leak, because the element and its client can't be removed by the garbage collector.

But I can't say for sure without seeing how the log handler is used.

[WolfgangFahl]

@falkoschindler thank you for looking into this what would be the proper design?
The log_view is part of my default footer.

async def setup_footer(
        self,
        with_log: bool = True,
        handle_logging: bool = True,
        max_lines: int = 20,
        log_classes: str = "w-full h-40",
    ):
        """
        setup a footer with an optional log view
        """
        if with_log:
            self.log_view = LogView(max_lines=max_lines, classes=log_classes)
            if handle_logging:
                self.log_view.addAsHandler(self.logger)
        else:
            self.log_view = None
        await super().setup_footer()
        if self.args.input:
            # await client.connected()
            with_render = self.render_on_load
            await self.read_and_optionally_render(
                self.args.input, with_render=with_render
            )
            

which is handled by a WebSolution which is effectively a wrapper for a client. The WebSolution could have proper synchronization with the Clients lifecycle. What is the official Client lifecycle and why do i get "The Client has been deleted" as an Exception right in the push action?

class InputWebSolution(WebSolution):
    """
    A WebSolution that is focused on handling a single input.

    Attributes:
        is_local (bool): Indicates if the input source is local or remote.
        input (str): The input data or path to be processed.
    """

    def __init__(self, webserver: NiceGuiWebserver, client: Client):
        """
        Initializes the InputWebSolution instance with the webserver and client.

        Args:
            webserver (NiceGuiWebserver): The webserver instance this solution is part of.
            client (Client): The client interacting with this solution.
        """
        super().__init__(webserver, client)

[falkoschindler]

what would be the proper design?

Here is an example for attaching a ui.log element to a log handler:
https://nicegui.io/documentation/log#attach_to_a_logger
Note the on_disconnect handler for removing the log handler.

What is the official Client lifecycle

There's one client for the auto-index page that lives as long as the app is running. For all other pages, there's a client created when requesting a page route. After creating and delivering the page content, the browser initiates a websocket connection, causing the client to connect. When the server-client connection disconnects, the browser has a chance to reconnect within a certain interval. If this doesn't happen in time, the client is removed from a global client dictionary. This is the crucial point: If the client (or any of its UI elements) is referenced anywhere else in the app, the instance will remain in memory and cannot be removed by the garbage collector.

[WolfgangFahl]

That means that my current assumption is wrong. I currently have WebServer and WebSolution as the core elements. A WebServer is handling the environment for a certain project and listens on a certain port. A WebSolution is for handling clients and i was under the assumption that this is stable for a "session" - but there is no real session concept here so we have to go on the Request level which means there are three levels of state handling necessary. Technical that means port/socket/request and i would love to handle this in a more abstract and generic way that does hide the technical details.