Distillation loss monitoring in LwF

[ghost]

Hi, everyone

Do you know if it is possible to monitor the distillation loss in LwF during training? I have already seen how the LwFPLugin calculates it with the 'penalty' method and sums it accordingly to the 'standard' loss function for the classification metric. However, the loss captured by the evaluation plugin is the total loss but I also want to record them separated. Is there any option to do this?

Thanks in advance

[vlomonaco]

Hi @PabloMese, mmh.. yeah it seems to be a little tricky to monitor that easily at the moment (@AndreaCossu, @AntonioCarta correct me if I'm wrong). The cleanest option would be to create your version of the lwf plugin and save in the strategy object an additional attribute "penalty", then implement a custom metric similar to the loss one to monitor that. This way you'd get for free all the logging as for the other metrics.

A quick hack may be to just add a line here and save the penalty someplace else before it gets summed into the main loss.

In the future we may think of smarter ways to add watchers to user-defined state variables. thanks for opening the issue, hope it helps!

[AntonioCarta]

The plugin doesn't save the metric, but you can easily do it yourself by subclassing lwf and adding the code to send the values to the EvaluationPlugin (not tested):

clas MyLWF(LwFPlugin):
    def before_backward(self, strategy, **kwargs):
        """
        Add distillation loss
        """
        alpha = (
            self.alpha[strategy.clock.train_exp_counter]
            if isinstance(self.alpha, (list, tuple))
            else self.alpha
        )
        penalty = self.penalty(
            strategy.mb_output, strategy.mb_x, alpha, strategy.model
        )
        strategy.loss += penalty
        # NEW CODE HERE
        x = strategy.clock.total_iterations
        mname = 'your_metric_name'
        mval = MetricValue(self, mname, penalty, x)
        strategy.evaluator.publish_metric_value(mval)

This is going to send the value automatically to your loggers.

we may add this in the future.

[ghost]

This seems to work but it falls at publishing de metric because 'publish_metric_value' is not an attribute for the Evaluation Plugin.

[AntonioCarta]

I forgot to tell you, the method was added after the pip release, so you need to use the master branch to use it.

[ghost]

Understood!