showing multiple *different* progress bars

[boegel]

First of all, thanks a lot for this great library, it's a real game changer!

Is there a way to show multiple different progress bars at once, which each show the progress for a different aspect (for example, N overall tasks done + separate progress on an individual task)?

I know that a single Progress instance can show the progress for multiple tasks at once (see python -m rich.progress), but then all these tasks need to be of the same "type", and I only want to have one task active as a time (since each task is slow and can consume a lot of resources of the system).

Non-working example of what I would like to do is included below.

import os
from functools import partial
from urllib.request import urlopen

from rich.progress import BarColumn, Progress, TextColumn
from rich.progress import DownloadColumn, TransferSpeedColumn, TimeRemainingColumn

urls = [
    'https://ftp.gnu.org/gnu/gcc/gcc-10.1.0/gcc-10.1.0.tar.gz',
    'https://ftp.gnu.org/gnu/gcc/gcc-10.2.0/gcc-10.2.0.tar.gz',
    'https://ftp.gnu.org/gnu/gcc/gcc-10.3.0/gcc-10.3.0.tar.gz',
    'https://ftp.gnu.org/gnu/gcc/gcc-11.1.0/gcc-11.1.0.tar.gz',
    'https://ftp.gnu.org/gnu/gcc/gcc-11.2.0/gcc-11.2.0.tar.gz',
]


def download(url):
    """Download specified URL, in chunks, and show download progress."""
    fn = os.path.basename(url)

    progress_download = Progress(
        TextColumn('[bold yellow] Downloading {task.fields[filename]}'),
        BarColumn(),
        DownloadColumn(),
        TransferSpeedColumn(),
        TimeRemainingColumn(),
    )

    url_fh = urlopen(url)
    size = int(url_fh.info()['Content-length'])
    task_id = progress_download.add_task('', filename=fn, total=size)

    with open(fn, 'wb') as dest:
        with progress_download:
            for data in iter(partial(url_fh.read, 32*1024), b''):
                dest.write(data)
                progress_download.update(task_id, advance=len(data))


if __name__ == '__main__':
    progress_overall = Progress(
        TextColumn('[bold blue] {task.completed}/{task.total} done'),
        BarColumn(),
    )
    with progress_overall:
        main_task = progress_overall.add_task('', total=len(urls))
        for url in urls:
            download(url)
            progress_overall.update(main_task, advance=1)

This fails with rich.errors.LiveError: Only one live display may be active at once, and I understand why, but I'm not sure how to achieve what I have in mind...

I played around with using my own Live instance and using Layout, but I don't want a full screen thing, just show multiple totally different progress bars at the bottom.
I also played with Table a bit, but next to the progress bars I also want to be able to produce regular console output on top of those progress bars, not sure if using a table is the right way forward for that (it certainly feels wrong ;) ).

In my actual use case downloading stuff is actually only a small step of a single task, so in the end I may actually want three different progress bars: one overall (N tasks done), one progress bar per task (S steps of that task done), and then a temporary download progress bar (for example, there's other similar time-consuming subtasks too that may warrant a separate progress bar).

edit: I've made a little bit of progress thanks to the live_progress.py example which is referenced from #1130, seems like using a Table instance obtained via Table.grid(), and using that as a renderable could help.

But then I run into the next problem: I can add rows to a table, but there doesn't seem to be a way to remove rows? For the subtask progress bars, I want to remove them when that specific subtask (like downloading a file) is done.

Am I pushing things too far here, trying to make it too dynamic?

[boegel]

Let me answer my own question: I got things to work pretty much as I wanted by using a Live instance with a RenderGroup instance that combines a bunch of Progress instances, each of which showing the progress on a different aspect.

Full example is below...

import os
import time

from rich.console import RenderGroup
from rich.live import Live
from rich.progress import BarColumn, Progress, SpinnerColumn, TextColumn, TimeElapsedColumn
from rich.progress import DownloadColumn, TransferSpeedColumn, TimeRemainingColumn
from rich.rule import Rule

app1_urls = [
    ('https://example.org/app1-part1.tar.gz', 3401321),
    ('https://example.org/app1-part2.tar.gz', 6382217),
]
app2_urls = [
    ('https://example.org/app2-part1.tar.gz', 3929422),
    ('https://example.org/app2-part2.tar.gz', 4156304),
    ('https://example.org/app2-part3.tar.gz', 4356921),
    ('https://example.org/app2-part4.tar.gz', 4582810),
    ('https://example.org/app2-part5.tar.gz', 4731931),
    ('https://example.org/app2-part6.tar.gz', 5103109),
]

step_actions = ('configuring', 'building', 'installing')


def fake_download(url, tot_size_bytes):
    """Fake download specified URL, in chunks, and show download progress."""
    fn = os.path.basename(url)
    task_id = download_progress.add_task('', filename=fn, total=tot_size_bytes)

    chunk_size = 157*1024

    downloaded_bytes = 0
    while downloaded_bytes < tot_size_bytes:
        time.sleep(0.1)
        if downloaded_bytes + chunk_size < tot_size_bytes:
            downloaded_bytes += chunk_size
        else:
            chunk_size = tot_size_bytes - downloaded_bytes
            downloaded_bytes = tot_size_bytes
        download_progress.update(task_id, advance=chunk_size)
    download_progress.stop_task(task_id)
    download_progress.update(task_id, visible=False)


def run_steps(name, step_times, task_id):
    for idx, step_time in enumerate(step_times):
        action = step_actions[idx]
        step_task_id = step_progress_timed.add_task('', action=action, name=name)
        for _ in range(step_time):
            time.sleep(1)
            step_progress_timed.update(step_task_id, advance=1)
        step_progress_timed.stop_task(step_task_id)
        step_progress_timed.update(step_task_id, visible=False)
        task_progress.update(task_id, advance=1)


label_progress = Progress(
    TimeElapsedColumn(),
    TextColumn('{task.description}'),
)

# progress for a task step that takes a while, but we're not sure how long
step_progress_timed = Progress(
    TextColumn('  '),
    TimeElapsedColumn(),
    TextColumn('[bold purple]{task.fields[action]}'),
    SpinnerColumn('simpleDots'),
)
# progress for a task step that has a known total target (steps, bytes, ...)
step_progress = Progress(
    TextColumn('  '),
    TimeElapsedColumn(),
    TextColumn('[bold purple]{task.fields[action]}'),
    BarColumn(),
    TextColumn('({task.completed}/{task.total})'),
)
download_progress = Progress(
    TextColumn('[bold yellow]   Downloading {task.fields[filename]}'),
    BarColumn(),
    DownloadColumn(),
    TransferSpeedColumn(),
    TimeRemainingColumn(),
)
# progress for a single tasks
task_progress = Progress(
    TextColumn('[bold blue]Progress for {task.fields[name]}: {task.percentage:.0f}%'),
    BarColumn(),
    TextColumn('({task.completed} of {task.total} steps done)'),
)
overall_progress = Progress(
    TimeElapsedColumn(),
    BarColumn(),
    TextColumn('{task.description}')
)
group = RenderGroup(
    label_progress,
    step_progress_timed,
    step_progress,
    download_progress,
    Rule(style='#AAAAAA'),
    task_progress,
    overall_progress
)
live = Live(group)

apps = [
    ('app 1', app1_urls, (2, 5, 3)),
    ('app 2', app2_urls, (5, 10, 6)),
    ('app 3', [], (1, 5, 3)),
]
overall_task_id = overall_progress.add_task('', total=len(apps))
with live:
    for idx, (name, urls, step_times) in enumerate(apps):
        overall_progress.update(overall_task_id, description='[bold #AAAAAA](%d out of %d apps installed)' % (idx, len(apps)))
        label_task_id = label_progress.add_task("installing %s" % name)
        task_id = task_progress.add_task('', total=4, name=name)
        download_task_id = step_progress.add_task('', total=len(urls), action='Fetching sources', name=name)
        for (url, size_bytes) in urls:
            fake_download(url, size_bytes)
            step_progress.update(download_task_id, advance=1)
        step_progress.update(download_task_id, visible=False)
        task_progress.update(task_id, advance=1)
        run_steps(name, step_times, task_id)
        task_progress.update(task_id, visible=False)
        label_progress.stop_task(label_task_id)
        label_progress.update(label_task_id, description="[bold green]%s installed!" % name)
        overall_progress.update(overall_task_id, advance=1)

    overall_progress.update(overall_task_id, description="[bold green]%s apps installed, done!" % len(apps))

[boegel]

I understand, I'm happy to trim it down a bit if that helps in making it a good example.

The live_progress.py is close to my use case to some extent, but it too me some time to make it a bit more dynamic: some of the progress bars disappear when that (sub)task is done, which is exactly what I was after. That doesn't work with a table: although you can add rows to add additional progress bars on the fly, you can't remove those rows later on I think (you can hide the full progress bar, but that leads to blank lines).
A RenderGroup like I used works fine, but I couldn't seem to find that possibility easily in the docs/examples (partially because RenderGroup is "hidden" in rich.console, which seems a bit counterintuitive).

So perhaps this could be a dynamic_progress.py example?

[willmcgugan]

Sure. Sounds good!

[boegel]

Any specific LoC target for a good example? Less than 100 lines? Smaller?

[willmcgugan]

Not really. If you could just cut as much as possible that doesn't serve to demonstrate the features. Put yourselves in the shoes of a beginner (Python and Rich)...

[boegel]

Proposal for additional example in #1504, happy to massage it a bit further if needed!

[jfthuong]

Note that now, from rich.console import RenderGroup has been changed to from rich.console import Group