Progress bar with TQDM

[tommargar]

While updating my library, I found it annoying to have no clue what the status is.

I've asked ChatGPT to edit commands.py and add a progress bar, and as I'm not actively developing, I'm posting it as an idea with code example to adapt from.

Needs tqdm, pip install tqdm

def update_items(lib, query, album, move, pretend, fields, exclude_fields=None):
    """For all the items matched by the query, update the library to
    reflect the item's embedded tags.
    :param fields: The fields to be stored. If not specified, all fields will
    be.
    :param exclude_fields: The fields to not be stored. If not specified, all
    fields will be.
    """
    from tqdm import tqdm  # Lokaler Import von tqdm

    with lib.transaction():
        items, _ = _do_query(lib, query, album)
        if move and fields is not None and "path" not in fields:
            # Spezialfall: Wenn ein Element verschoben werden muss, muss das 'path'-Feld
            # aktualisiert werden; andernfalls wird der neue Pfad nicht in der Datenbank reflektiert.
            fields.append("path")
        if fields is None:
            # Keine Felder wurden angegeben, aktualisiere alle Medienfelder
            item_fields = fields or library.Item._media_fields
            if move and "path" not in item_fields:
                # Verschieben ist aktiviert, füge 'path' zur Liste der zu aktualisierenden Felder hinzu
                item_fields.add("path")
        else:
            # Felder wurden angegeben, aktualisiere nur diese
            item_fields = fields
        # Erhalte alle Albumfelder zur Aktualisierung
        album_fields = fields or library.Album._fields.keys()
        if exclude_fields:
            # Entferne alle ausgeschlossenen Felder aus den Item- und Album-Mengen
            item_fields = [f for f in item_fields if f not in exclude_fields]
            album_fields = [f for f in album_fields if f not in exclude_fields]

        # Durchlaufe die Items und erfasse ihre Änderungen.
        affected_albums = set()
        for item in tqdm(items, desc="Verarbeite Items", unit="Item"):
            # Item gelöscht?
            if not item.path or not os.path.exists(syspath(item.path)):
                ui.print_(format(item))
                ui.print_(ui.colorize("text_error", "  gelöscht"))
                if not pretend:
                    item.remove(True)
                affected_albums.add(item.album_id)
                continue

            # Hat sich das Item seit der letzten Überprüfung geändert?
            if item.current_mtime() <= item.mtime:
                log.debug(
                    "Überspringe {0}, da mtime aktuell ist ({1})",
                    displayable_path(item.path),
                    item.mtime,
                )
                continue

            # Lese neue Daten.
            try:
                item.read()
            except library.ReadError as exc:
                log.error(
                    "Fehler beim Lesen von {0}: {1}", displayable_path(item.path), exc
                )
                continue

            old_item = lib.get_item(item.id)

            # Spezialbehandlung für Albumkünstler, wenn er mit dem Track-Künstler übereinstimmt.
            if not item.albumartist:
                if old_item.albumartist == old_item.artist == item.artist:
                    item.albumartist = old_item.albumartist
                    item._dirty.discard("albumartist")

            if old_item.albumtype in item.albumtypes:
                item.albumtype = old_item.albumtype

            # Überprüfe und zeige Änderungen an.
            changed = ui.show_model_changes(item, fields=item_fields)

            # Speichere Änderungen.
            if not pretend:
                if changed:
                    # Verschiebe das Item, wenn es in der Bibliothek ist.
                    if move and lib.directory in ancestry(item.path):
                        item.move(store=False)

                    item.store(fields=item_fields)
                    affected_albums.add(item.album_id)
                else:
                    # Die mtime der Datei war unterschiedlich, aber es gab keine
                    # Änderungen an den Metadaten. Speichere die neue mtime,
                    # die in dem Aufruf von read() gesetzt wird, damit wir
                    # dies in Zukunft nicht erneut überprüfen.
                    item.store(fields=item_fields)

        # Überspringe Albumänderungen, wenn 'pretend' aktiv ist.
        if pretend:
            return

        # Modifiziere betroffene Alben, um Änderungen in ihren Items zu reflektieren.
        for album_id in tqdm(affected_albums, desc="Verarbeite Alben", unit="Album"):
            if album_id is None:  # Einzelstücke.
                continue
            album = lib.get_album(album_id)
            if not album:  # Leere Alben wurden bereits entfernt.
                log.debug("geleertes Album {0}", album_id)
                continue
            first_item = album.items().get()

            # Aktualisiere die Albumstruktur, um ein Item darin zu reflektieren.
            for key in library.Album.item_keys:
                album[key] = first_item[key]
            album.store(fields=album_fields)

            # Verschiebe Albumcover (und alle inkonsistenten Items).
            if move and lib.directory in ancestry(first_item.path):
                log.debug("Verschiebe Album {0}", album_id)

                # Manuelles Verschieben und Speichern des Albums.
                items = list(album.items())
                for item in items:
                    item.move(store=False, with_album=False)
                    item.store(fields=item_fields)
                album.move(store=False)
                album.store(fields=album_fields)

[peterjdolan]

Ooh, we've had the same idea!

TQDM is great, of course, but it has the nasty habit of interacting poorly with log lines or other messages sent to STDOUT. I found another, similar library called Enlighten (https://python-enlighten.readthedocs.io/en/stable/) that's really quite amazing. It handles the terminal output the way one would like it to be done - the progress bar at the bottom, and all the log messages above it with no interaction between the two.

Here's my proposed PR: #5692. Eager for feedback.

I'm not sure how to go about testing the change other than to verify that all the affected code paths are covered by other tests. In development, I definitely noticed that it's easy to accidentally override another variable's name (e.g. changed boolean and changed status bar), which typically results in an exception and crash. So, we should guard against that via testing. If we really want to test it fully, we could monkeypatch the library in the existing tests and verify that the appropriate number of calls to the progress bar were made.

[peterjdolan]

I just filed a bug to track this - #5711. My PR is almost complete, I just need to figure out how to integrate it into the import workflow.

[peterjdolan]

@sampsyo do you have any suggestions on how to best integrate a progress bar into the import operation? I'm imagining that I could initialize the progress bar in ui.commands.TerminalImportSession#run by adding an import stage following the read/query tasks stages (to initialize it with the total number of tasks), and could then update the progress bar with another stage after manipulating files.

What do you think?

We could also include sub-progress bars showing the progress of looking up candidates, as that would show that the system is making progress in the background.