Discrepancy in Bit Accuracy During Pretraining (0.8 vs Paper's 0.95) and Calculation Method Clarification

[wangjunjie-1]

When reproducing the model using COCO dataset, I observed a significant gap in bit accuracy during pretraining phase

Performance Discrepancy

After first training phase

My implementation achieves only ~0.8 bit_acc (single watermark case)
Paper reports 0.95 bit_acc (single watermark case)

Calculation Method Conflict

I found conflicting thresholding logic in the codebase when computing bit accuracy:

In msg_predict_inference (IPython notebook):
preds = preds > 0.5 # Explicit 0.5 threshold
In bit_accuracy_inference (training metrics):
preds = preds > threshold # Default threshold=0.0
This discrepancy significantly impacts binary predictions. Which approach is canonical?

[pierrefdz]

Performance Discrepancy

Could you share your command and machine configuration?
I think the issue might come from the fact that you're not training on 8 GPUs which changes the effective batch size.

For information, I've added in the readme the logs of the run on COCO: https://github.com/facebookresearch/watermark-anything/blob/main/README.md#examples

Calculation Method Conflict

You should check the outputs. If they are between 0 and 1, you should use 0.5 as threshold, otherwise (if they are positive and negative), you should use 0 as threshold.

[wangjunjie-1]

Performance Discrepancy

Could you share your command and machine configuration? I think the issue might come from the fact that you're not training on 8 GPUs which changes the effective batch size.

For information, I've added in the readme the logs of the run on COCO: https://github.com/facebookresearch/watermark-anything/blob/main/README.md#examples

Calculation Method Conflict

You should check the outputs. If they are between 0 and 1, you should use 0.5 as threshold, otherwise (if they are positive and negative), you should use 0 as threshold.

Thank you for sharing the detailed training logs! They are extremely helpful for reproducing the model.

I've reviewed the key metrics (e.g., bit_acc, psnr, etc.) and will use them to verify the model's performance.

If I encounter any issues, I might reach out for your advice. Thanks again for your support! 👍

[wangjunjie-1]

First of all, thank you so much for sharing the log files! They have been really helpful in understanding the training process. However, I have a few questions regarding some details in the logs:

1. Data Augmentation File Name:
   I noticed that the data augmentation file used during training is named “all_augs_resize.yaml.” Does this imply that the augmentation strategy used during model training only includes resizing operations? If so, were other augmentation techniques intentionally excluded,?

2. Batch Size Settings:
   I observed that the batch size settings differ between the pretrain and posttrain phases. Was this difference in batch size settings intentional, or was it an oversight? If it was deliberate, could you please explain the rationale behind this design choice?

[pierrefdz]

Hi again,

1. This was the name of the config before, but it corresponds to this one now: https://github.com/facebookresearch/watermark-anything/blob/main/configs/all_augs.yaml

2. When finetuning with multiple messages, the image is forwarded up to 3 times to the embedder, with different messages. Therefore it increases the required GPU memory, which is the reason why we decrease the batchsize. It can be seen here: https://github.com/facebookresearch/watermark-anything/blob/main/watermark_anything/models/wam.py#L112, although I agree that it is not super clear in the code. It's also detailed with a bit more details in the paper: https://arxiv.org/pdf/2411.07231#page=21.84 App D.2.