How to print metric value every epoch ?

[richarddwang]

What I want it to look like:

What I got now:

I think I do have logged metrics, how to make it printed beautifully ?

class SortNumberModel(pl.LightningModule):
    def __init__(self, hf_config, lr):
        super().__init__()
        self.save_hyperparameters()
        self.model = AutoModelForTokenClassification.from_config(hf_config)
        self.val_acc = pl.metrics.Accuracy()

    def forward(self, batch):
        result = self.model(input_ids=batch["x"], labels=batch["y"], return_dict=True)
        return result.logits.argmax(dim=-1), result.loss

    def training_step(self, batch, batch_idx):
        return self.forward(batch)[-1]

    def validation_step(self, batch, batch_idx):
        preds, loss = self.forward(batch)
        self.val_acc(preds.view(-1), batch["y"].view(-1))
        self.log("valid_acc", self.val_acc, on_epoch=True)

    def configure_optimizers(self):
        return torch.optim.AdamW(self.parameters(), lr=self.hparams.lr, eps=1e-5)

[SkafteNicki]

In pytorch-lightning bolts we have a few extra callbacks, with one of them being the PrintTableMetricsCallback callback that should output exactly what you want. Simply install bolts as

pip install pytorch-lightning-bolts

and initialize your trainer as:

trainer = Trainer(callbacks=[PrintTableMetricsCallback()])

[richarddwang]

Thanks for your comment.
But I got this in vscode notebook for

trainer = pl.Trainer(
    max_epochs=1, gpus="0", callbacks=[PrintTableMetricsCallback()]
)
trainer.fit(model, datamodule=dm)

It seems that it even prints for valid step, any good suggestion ?

[SkafteNicki]

I can see that the callback is not doing exactly what you want.
I tried to type up another callback that should do something closer to what you want:

from pytorch_lightning.callbacks import Callback
from copy import deepcopy
import tableprint as tp

class Printer(Callback):
    def __init__(self) -> None:
        self.metrics = {}
        self.table_context = None

    def on_train_epoch_end(self, trainer, pl_module, outputs) -> None:
        metrics_dict = deepcopy(trainer.callback_metrics)
        if self.table_context is None:
            self.table_context = tp.TableContext(headers=metrics_dict.keys())
            self.table_context.__enter__()
        self.table_context([v.item() for v in metrics_dict.values()])

    def on_train_end(self, trainer, pl_module):
        self.table_context.__exit__()

this callback requires that you install https://github.com/nirum/tableprint. With this, it should output something like this to the terminal:

[richarddwang]

Thanks for following up !

I think to print validation accuracy/loss, it may be more reasonable if we add info to the table at on_validation_epoch_end.
Otherwise it seems to print metrics calculated at previous validation loop.
So I made a little modification.

class Printer(Callback):
    def __init__(self) -> None:
        self.metrics = {}
        self.table_context = None

    def on_validation_epoch_end(self, trainer, pl_module) -> None:
        if trainer.running_sanity_check:
            return
        metrics_dict = deepcopy(trainer.callback_metrics)
        if self.table_context is None:
            self.table_context = tp.TableContext(headers=metrics_dict.keys())
            self.table_context.__enter__()
        self.table_context([v.item() for v in metrics_dict.values()])

    def on_fit_end(self, trainer, pl_module):
        self.table_context.__exit__()

(Cuz sanity check also calls validation loop and thus activate on_validaton_epoch_end, I also add an if clause to skip it.)

And it now pretty fit my need.

I have tried pt for few days, it is really neat. It will be nice if someone can contribute an default(or off-the-shelf) and non-novel table printing to it.

[rudaoshi]

@SkafteNicki, Thank you for you answer.

However, I found the solution prints different metric values accross different ddp process.

I'm struggling to correctly print the metrics on the end of each epoch, but I meet a lot of problem discribed in this issue:

#6122 (comment)

Will you please look through my problems and give some suggestions?

Thank you for your help!