integration as HP metrics

[Ilykuleshov]

Is it possible to add the model's torchmetrics metrics as HP-metrics to tensorboard by default (or via a parameter)? Right now hp-metrics in pl are very inconvenient and confusing, to the point of being useless. This would deepen the pytorch lightning integration and make hp-metric a lot more usable and also add incentive to use torchmetrics.

[Borda]

Currently, PyTorch Lightning doesn’t add TorchMetrics metrics as HP metrics to TensorBoard by default. The existing HP metrics support can indeed be confusing and limited.
A good workaround is to manually log your TorchMetrics values as HP metrics inside your LightningModule like this:

from torchmetrics import Accuracy
import pytorch_lightning as pl

class LitModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.accuracy = Accuracy()

    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        acc = self.accuracy(y_hat, y)
        # Log accuracy as a hyperparameter metric
        self.log('hp/val_accuracy', acc, prog_bar=True, logger=True)

This way, your metrics will appear under the HP metrics section in TensorBoard.
This manual logging offers flexibility until there is an official parameter or deeper integration for automatic HP metric logging.