Different metrics in Panels and Metrics in the Experiment UI

[kirilllzaitsev]

Describe the Bug

Metrics observed on plots in Panels match the ones logged via stdout, while those in 'Metrics' occasionally do not (some do, but not all of them).

Expected behavior

The metrics are the same everywhere.

Where is the issue?

• Comet Python SDK
• [+] Comet UI
• Third Party Integrations (Huggingface, TensorboardX, Pytorch Lighting etc)

To Reproduce

Steps to reproduce the behavior:
0. Log metrics to an experiment
For example, via Tensorboard: writer.add_scalar(f"{stage}_epoch/{k}", v, epoch_count)

1. Go to 'Panels' -> note the last metric value
2. Go to 'Metrics' -> compare the last metric value with the one in 'Panels'

Stack Trace

If possible please include the full stack trace of your issue here

# Paste stack trace here

Comet Debug Log

If possible, please follow the instructions here to run Comet in debug mode and attach the resulting log file.

Screenshots or GIFs

Correct:

Incorrect:

val_epoch/r_err matches, while train_epoch/r_err is incorrect in 'Metrics'. At the same time, train_epoch/t_err and val_epoch/t_err in the same experiment match.

Link to Comet Project/Experiment

If applicable, please provide a link to your Comet Project or Experiment.

Additional context

Add any other context about the problem here.