Log to stderr without rich without breaking progress bars

[xmatthias]

While this is similar to #1317 - it's actually not the same, as i'm not looking to also use rich for logging at this point.

is it possible to use the rich progressbar - with random logging to stderr - without using rich.logging/loghandlers?

import logging
from rich.progress import track
import time

logging.basicConfig(level=logging.INFO)
log = logging.getLogger()

for n in track(range(5), description="Processing..."):
    log.info(f"Working.. {n}")
    time.sleep(0.2)

Output:

Processing... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--INFO:root:Working.. 0
Processing... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--INFO:root:Working.. 1
Processing... ━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  20% -:--:--INFO:root:Working.. 2
Processing... ━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━  40% 0:00:01INFO:root:Working.. 3
Processing... ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━  60% 0:00:01INFO:root:Working.. 4
Processing... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

This is possible in progressbar2 by wrapping stdout/stderr streams - but i haven't found this possibility in rich just yet.

[saschalalala]

Not sure if I understand the question correctly. Do you want to suppress / hide the logs? Would perhaps something like

python richtest.py 2>error.log

or

python richtest.py 2>/dev/null

solve your problem?

[xmatthias]

no, unfortunately not (i could just change the log-level for that case)...
i'd want to have the progressbar rendered at the bottom - and logs should output at the top.

this seems to work with RichHandler (which has other drawbacks i've so far not been able to work around) - but not with the regular logging handler.

[willmcgugan]

Rich will actually redirect stderr. If you change your log to a print("foo", file=sys.stderr), that will work.

I think what is happening is that the logger keeps a reference to the stderr, and ignores RIch's proxy. You might want to try writing a handler that gets sys.stderr each time.

[willmcgugan]

The StreamHander is implemented in such a way that the typical redirection method will not work.

You can write a simple log handler which writes to sys.stderr without keeping a reference. Writing a custom log handler is the way to do that. It is well documented in the Python docs, and trivial. Here is an example.

[xmatthias]

The StreamHander is implemented in such a way that the typical redirection method will not work.

that's a bug then? (want me to report this as issue instead?)

Your approach (write a custom log handler) assumes control over the full logging setup - which means changes to completely unrelated parts of an application just to add a simple (pretty) progressbar.
i'm 100% certain this will work - and i did write custom log-handlers in the past (for different usecases).
But it'll assume to change the whole logging setup - which seems a bit excessive just to use a progressbar properly?

Assuming full control over logging setup (and therefore the possibility to replace the already initialized stream-handler) is not possible, what would be the approach, then?

[willmcgugan]

If you think that's a bug, then feel free to submit it to the CPython repo.

I'm afraid I can't help with your hypothetical app. But let me know if you find a satisfactory solution.

[xmatthias]

To be honest, I'd think it's a bug (or missing feature) in the Rich progressbar implementation, not in CPython.

There's no way to pass in a log-handler (or something similar) rich should respect.
it rewrites stdout/stderr - but if the prior instance is bound - then rendering breaks, without a clear reason as to why without debugging internals.
I don't really see how this would be a CPython bug. It's rich's rendering that breaks - not cpython.

---

Now don't get me wrong - i think rich is doing a great job - and the rendering it provides is great, and we can rely on it across different systems and terminals (including windows!)), without worrying about the actual details.
This is however making a gradual migration to rich quite difficult (also difficult to reason about) - as it's a change needed someplace completely different.

i did look around discussions / issues in the rich repository while researching this problem (again) - and i foundat least 2 other, still open issues with basically the same question (or facing the same problem) in #2369 and #2607, and several others which have this somehow worked around - mostly by using a console object.

we've so far used progressbar2 - which has an explicit wrap_stdout() and wrap_stderr() call, which can be called BEFORE the logging initializes - and wraps the stdout at the right time.
This seems impossible with rich (?) - and an explicit console won't really help (at least the way i understand the progressbar implementation) - as we need both stdout and stderr wrapped.

Maybe a "warning" note in the docs would help this, with possible workarounds (i'd assume this would reduce the amount of questions around progressbar displays).

[xmatthias]

I've found 2 solutions to this now, based on your help with what's actually causing this (the reference to stderr).

the obvious solution is the custom loghandler (which clearly works, if it's possible to change that application-wide).

The alternative is to wrap the Process contextmanager in a custom contextmanager, which will handle "fixing" of the actual handler (in this case, in a patchy mode fixing exactly my scenario).

@contextmanager
def CustomProgress(*args, **kwargs):
    """
    Wrapper around rich.progress.Progress to suppress output to stderr
    """
    try:
        __logger = kwargs.pop('logger', None)
        streamhandlers = [x for x in __logger.root.handlers if type(
            x) == logging.StreamHandler]

        with Progress(*args, **kwargs) as progress:
            for handler in streamhandlers:
                __prior_stderr = handler.stream
                handler.setStream(sys.stderr)

            yield progress

    finally:
        streamhandlers = [x for x in __logger.root.handlers if type(
             x) == logging.StreamHandler]
        for handler in streamhandlers:
            handler.setStream(__prior_stderr)

For my specific usecase, the below works well (we only attach handlers to the root handler, and will only have one StreamHandler) - without the need to change unrelated parts of the application.
Different scenarios may need to do this slightly differently.