How to retrain the models in videoseal/cards (PixelSeal / ChunkySeal / VideoSeal)?

[QQiuyp]

Thank you for releasing such an excellent project. I found WAM/VideoSeal and the provided model variants (PixelSeal, ChunkySeal, etc.) very interesting, and I’d like to retrain models for robustness against additional distortions.

I have a question about retraining the models referenced in videoseal/cards/*.yaml:
1. Are the cards intended as inference model cards only, or are they also meant to be full training recipes?
• Some cards (e.g., videoseal_0.0.yaml) look like they contain full training hyperparameters, but others (e.g., pixelseal.yaml, chunkyseal.yaml) seem to only specify architecture + a few key args (nbits, img_size_proc, attenuation, etc.).
• Is the intended way to retrain these variants to start from the docs/training.md commands and substitute the architecture/args from each card?
2. PixelSeal training: In the PixelSeal paper, the training pipeline looks more complex (multi-stage / specific loss).
• Does this repository include the full PixelSeal training pipeline/configs/scripts used to produce the released PixelSeal checkpoint?

[pierrefdz]

Thanks for the kind words!

1. Cards

The cards are mainly here for ease of loading and not intended for the training args.
For training, yes, I recommend going to the docs/training.md. The exact args are saved in the .pth that you can download from the cards (in the key "args" of the state dict).
You can also choose your own args, start by looking at: https://dl.fbaipublicfiles.com/videoseal/train_img_y.json for the training args of videoseal_1.0, update lamda_i to 0 (this one is quite important and improves stability). This should already give something pretty stable. Then choose the embedder/extractor from the cards if you want to train new ones, update your augmentations, etc.

2. Training

Most of what is in the PixelSeal paper is already in the repo. For instance, the scaling schedule corresponds to --scaling_w_schedule, the schedule in the adv loss with "--disc_start". Some elements are not supported (e.g., watermark boosting for the discriminator, temporal interpolation inside the Unet), but are not primordial. We do not plan at the moment of putting them in the repo as our internal training has diverged a little. We might do it in the future, but if you want them you can implement them when we'd be very happy to merge it.

Don't hesitate if you need clarifications or if you have more questions

[QQiuyp]

Thank you for the suggestion! Initializing λᵢ = 0 turned out to be crucial — training is now stable on my end.

One interesting observation is that HiDDeN also benefits from temporarily ignoring the visual/perceptual loss when the message length becomes large, in order to stabilize early training. Your message embedding strategy is also different from MBRS/CIN/FIN: those methods typically reshape the message into a square grid and then use (transpose) convolution to expand it to the same spatial resolution as the image features, whereas your approach (similar to HiDDeN) repeats the message along height/width to match the feature map resolution. When the message length is large, concatenating the image features and the repeated message features yields a very high channel dimension, which may be a key reason for the instability in the initial phase.

A practical trick I used for HiDDeN is a curriculum on message length: first train with a shorter message, then load the weights into a longer-message setting (only updating the few necessary fusion layers). This also tends to improve stability. I haven’t tried this in VideoSeal yet, but I’ll share results if I find anything useful.

Thanks again for your work!

[pierrefdz]

whereas your approach (similar to HiDDeN) repeats the message along height/width to match the feature map resolution

Note that in our case, we first use a message embedding table that converts the $n_{bits}$ into a feature vector of dimension $d$. In practice, $d$ needs to be ≈ or > $n_{bits}$ otherwise the training does not converge to 100% bit acc.
We also concatenate this feature in the downsampled resolution of the UNet, so it's a relatively small compute overhead.

That being said, your idea is very interesting! I think we could do something similar by just increasing the $d$ during the training little by little.

[QQiuyp]

Got it—thanks again for the detailed clarification! It’s really only a small computational overhead.
Wishing you all the best with your work, and happy new year!

[pierrefdz]

Got it—thanks again for the detailed clarification! It’s really only a small computational overhead. Wishing you all the best with your work, and happy new year!

Thanks! Happy new year and good luck with your projects as well, looking forward to seeing them!