Skip to content

wandb session not re-used when resume_from_checkpoint is used #188

@a-r-r-o-w

Description

@a-r-r-o-w

Reproduction:

  • Start a training run for 10 steps with checkpointing set to 5 steps
  • Quit run after 6 steps
  • Start new run with resume_from_checkpoint

Training runs as expected, but all information before step 5 is lost as the same wandb session of the initial run is not resumed.

The same wandb session should be continued to preserve validation videos and other data

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions