How to remove hp_metric initial -1 point and x=0 points?

[bartmch]

Hi, I don't think this is a bug but I'm doing something wrong. I want to use my val_dice as hp_metric tabular AND also see the graph on "show metric" (radio button) under the Tensorboard HPARAMS tab:

To achieve this I'm logging using self.log('hp_metric', mean_dice) (for the graph) and self.logger.log_hyperparams(params=self.hparams, metrics={'hp_metric': mean_val_dice}) (for the hparams value), both in the function validation_epoch_end
How do I get rid of the initial -1 value? How can I fix my graph so it doesn't draw any points at x=0? (zoomed in version)

[tchaton]

That's a great question ! Why do you want to see the metric under hparams tab ?

[rolanddenis]

As @bartmch, I use hp_metric to compare hyper parameters (is that not the purpose of this placeholder ?). I also have this negative initial value, but I didn't notice it before. I suppose it is not possible to remove this initial value but is there a way to customize it ?

[joe32140]

You can set default_hp_metric in TensorBoardLogger to False to prevent the initial hp_metirc score.
TensorBoardLogger(save_dir='tb_logs', default_hp_metric=False)

[rolanddenis]

Yes but then the hp_metric is not correctly tracked in the hparams (#2974)...

[rolanddenis]

I already use your first way but setting default_hp_metric to False makes hp_metric be removed from "hparams" tab (this tab isn't there at all even if I have set some hyper parameters). Adding the final log_hyperparams creates the hparams tab but the graph of hp_metric gets a final value at iteration 0 instead of final iteration), and this step will also be skipped if the job is killed.

Here are what I've tried so far:

• default_hp_metric=True: hparams tab visible in Tensorboad with hp_metric updated during training, hp_metric wrong initial value that makes the corresponding graph unsuitable with log scale and smoothing activated.
• default_hp_metric=False: no hparams tab in TensorBoard, hp_metric graph OK
• default_hp_metric=False and final log_hyperparams: no hparams tab in TensorBoard during training, hp_metric graph OK until the end of the training where final point is associated to iteration #0. No hparams tab at all if job is killed.

I will try to calculate metric before training start to appropriately populate it, or to delay log_hyperparams at the first validation...

[FranzKnut]

Did you find any solution? Best, Julian

[rolanddenis]

More or less : I finally use the following trainer to delay the call to TensorBoardLogger.log_hyperparams just before fitting starts.

import pytorch_lightning as pl
from pytorch_lightning.loggers import TensorBoardLogger

class Trainer(pl.Trainer):
    
    def __init__(self, version=None, name=None, **kwargs):

        # Logger
        if kwargs.get('logger', True) is True:
            default_root_dir = kwargs.get('default_root_dir', None)
            kwargs['logger'] = TensorBoardLogger(
                default_root_dir if default_root_dir is not None else "logs",
                name=name,
                version=version,
                default_hp_metric=False # hp_metric will be declared before training, see below
            )

        # Create trainer
        super().__init__(**kwargs)
    
    def fit(self, model, *args, **kwargs):
        # Disabing TensorBoardLogger.log_metrics since log_hyperparams
        # call log_metrics with step always set to 0, leading to a zigzag
        # in the graph
        def do_nothing(*args, **kwargs):
            pass

        log_metrics = self.logger.log_metrics
        self.logger.log_metrics = do_nothing

        self.logger.log_hyperparams(
            params=model.hparams,
            metrics={'hp_metric': 0}
        )

        # Restoring TensorBoard.log_metrics
        self.logger.log_metrics = log_metrics

        super().fit(model, *args, **kwargs)

I don't remember exactly why it works but it solves the issues I had:

• the hparams tab is visible and up-to-date in TensorBoard, even during fitting,
• hp_metric doesn't have the initial value of 0 (BTW no initial value at all),
• fitting can be resumed without a zigzag like in the last image of @bartmch .

I've also tried to "hack" the sanity check to run a full validation step at initial epoch and use the resulting metrics in log_hyperparams. The advantage was that I got this way a valid initial value (at step=0) for the validation loss and hp_metric but the code was hard to maintain at each PyTorch-Lightning upgrade.

Hope it helps!