Offline attack with 4 reference models performs significantly worse than reported in the paper

[sophiaBuss0410]

Dear authors,

Thank you for providing the implementation for RMIA.
After attempts to reproduce the attack on the paper, we observed that the offline attack results with 4 reference models are much lower than those reported in the paper.

Results

Metric	Reported	Reproduced
Avg. Train Accuracy	99.9%	99.92–100%
Avg. Test Accuracy	92.4%	91.23–91.71%
AUC of RMIA (offline, 4 refs, 2 augs)	71.02(*)	64.67
TPR@0.01%FPR (offline, 4 refs, 2 augs)	2.91%(*)	0.73%
TPR@0.0%FPR (offline, 4 refs, 2 augs)	2.13%(*)	0.57%
AUC of RMIA (offline, 4 refs, 18 augs)	71.02(*)	64.88
TPR@0.01%FPR (offline, 4 refs, 18 augs)	2.91%(*)	0.84%
TPR@0.0%FPR (offline, 4 refs, 18 augs)	2.13%(*)	0.45%

(*) We refer to the values in Table 2 in the paper, which does not clearly state the number of augmentations.

---

Environment

OS: Linux 4.18.0-372.9.1.el8.x86_64
Python: 3.10.8 [GCC 12.2.0]
CUDA: 11.7
GPU: Tesla V100-PCIE-32GB
TensorFlow: 2.9.2
JAX: 0.3.17

---

What we did

We employed the code from this tree:
https://github.com/privacytrustlab/ml_privacy_meter/tree/d32734161a3395211fe5f3cd461932290b1fafbe/research/2024_rmia

1. Train 4 reference models.

# # training 4 ref models
prefix="cifar10_4"

if [ ! -d "logs/${prefix}" ]; then
    # If it doesn't exist, create the folder
    mkdir "logs/${prefix}"
    mkdir "exp/${prefix}"
    echo "Folder 'logs/${prefix}' created."
else
    echo "Folder 'logs/${prefix}' already exists."
fi

n_models_end=3 # train 4 reference models

for model in $(seq 0 1 $n_models_end);
do
    train="CUDA_VISIBLE_DEVICES='0' python3 -u train.py --dataset=cifar10 --epochs=100 --save_steps=100 --arch wrn28-2 --num_experiments 4 --expid ${model} --logdir exp/${prefix} &> 'logs/${prefix}/log_${model}'"
    eval ${train}
done

2. Train 1 target model.

CUDA_VISIBLE_DEVICES='0' python3 -u train_one_model.py \
    --dataset=cifar10 \
    --epochs=100 \
    --save_steps=100 \
    --arch wrn28-2 \
    --expid 0 \
    --logdir exp/cifar10_target &> logs/cifar10_target/log_0

train.py was modified to train_one_model.py as below to set num_experiments None and enable one model training:

# elif FLAGS.expid is not None:
elif FLAGS.expid is not None and FLAGS.num_experiments is not None:

ml_privacy_meter/research/2024_rmia/scripts/train.py

Line 336 in d327341

elif FLAGS.expid is not None:

3. python main_one_target_model.py --cf attack_configs/cifar10/rmia_offline_2_ref_models.yaml

main.py was modified to main_one_target_model.py as below to set model None to load the target model logits.

model_logits, _ = load_input_logits(models_path, epoch, 
# model=model_index,
model=None,
num_augmentations=nb_augmentations)

ml_privacy_meter/research/2024_rmia/main.py

Line 189 in d327341

model_logits, _ = load_input_logits(models_path, epoch, model=model_index, num_augmentations=nb_augmentations)

Config was modified as below:

nb_augmentation: 18 # for aug=18
nb_augmentation: 2 # for aug=2

data:
  target_dir: scripts/exp/cifar10_target # directory containing logits of one target model
  reference_dir: scripts/exp/cifar10_4 # directory containing logits of 4 reference models

---

We trained the one target model because specifying the same directory (scripts/exp/cifar10_4) faced a shape error.
I’m happy to share full logs or run additional experiments if that helps.

Thank you for your work!

Best regards,

Haruka Buss

[taojiashu]

Hi, we are not the authors of the RMIA paper, and we are simply hosting RMIA's authors' source code as one of the research foundations behind our privacy meter. Therefore, we are unable to help you debug it fully. However, from my test run (without modifying train.py and main.py), I get the AUCs ranging from 63% to over 70% for different target models (I trained 12). It seems that the models have different extents of memorization, even though they are trained in the same way (with different training data splits). Hence, I guess it might just be the random seed that results in the different numbers. Since the numbers depend on the actual target model(s) being attacked, it is not paramount to reproduce the same numbers. I think the relative attack power between different attacks (e.g., LiRA vs RMIA vs LOSS) is more important. In my test run, RMIA consistently dominates all other attacks.

[sophiaBuss0410]

Hi, thanks for the follow-up!

You pointed out the variability of AUC across different models, which is a valid observation. However, the original RMIA paper reports an AUC of 71.02 ± 0.37, averaged over 10 random models (in Table 9). Your reported range (63%–70%) seems to fall short of that benchmark, which suggests there might be other factors at play beyond randomness.

To better understand your setup, could you share the TPR at fixed FPR values (e.g., 0.01% and 0.0%)? This metric is crucial for assessing attack efficacy and is recommended than AUC.

Looking forward to your results!

[taojiashu]

TPR@0.01%FPR is about 1.5% and TPR@0%FPR is about 0.5%. But there might be differences in how the target models are trained between those evaluated in Table 2 and those created by the training script released. MIAs depend on the target models, so it is not image classification that is only test data dependent. I do not have the information on the exact setup used in the paper, but as long as the rank of attack power among all MIAs holds, I think the code here is not too problematic.

[sophiaBuss0410]

Thank you for your responses. We will investigate further and potentially ask follow-up questions.