What is the best way to implement progress bar for long-running migration?

[alvassin]

Describe your question
I have long runnning migration, that selects data from database using server-side cursor, modifies data and updates values in the database. I want to display progress for colleagues to make them understand how long migration would take. What is the best way for that? I tried to use tqdm, it works, perhaps there is better option?

Example (if applicable)

from alembic import op
from sqlalchemy import bindparam, text
from tqdm import tqdm


def upgrade():
    conn = op.get_bind()
    result = conn.execution_options(
        stream_results=True
    ).execute(
        text("SELECT * FROM table")
    )
    progress_bar = tqdm()
    for group in result.partitions(100):
        values = []
        for value in group:
            # Update value
            values.append(...)


        if values:
            query = example_table.update().values({
                'updated_column': bindparam('updated_column_value')
            }).where(
                example_table.columns.id == bindparam('value_primary_key')
            )
            conn.execute(query, values)
            progress_bar.update(len(values))

[zzzeek]

it looks like your progress bar is one that you can update synchronously in between statements, so that's not a very hard use case. looks fine to me? a more difficult use case is if you wanted a progress bar that would move along as the database is executing a very long blocking call of some kind.

[alvassin]

Yes, that sounds much more tricky, my example is simpler, but very common. Thank you for response, was not sure that breaking into alembic output manually is a good idea (it is using logging for output, when tqdm is just stdout).