VideoVisualizer concistent labeling across frames?

[tamis-laan]

I'm trying to run detectron on a video with the visualizer to create a small demo. However the labeling is not consistent across frames. Each object in each consecutive frame is labeled with a different color causing a disco effect.

I thought by switching to the VideoVisualizer object labeling would be more consistent, but I get the same problem:

        # Create visualizer
        viz = VideoVisualizer(MetadataCatalog.get(self.IS_cfg.DATASETS.TRAIN[0]), ColorMode.IMAGE)
        # Apply visualizer
        output = viz.draw_instance_predictions(image, predictions["instances"].to("cpu"))

I'm finding the documentation on this subject rather sparse. Is it even possible to get consistent labeling between frames?
And if not, what it the purpose of the VideoVisualizer compared to the regular Visualizer?