Can differential privacy's protective effect be verified？

[MrLinNing]

Your work is excellent, providing a great verification tool for security and privacy researchers. I would like to inquire whether your method can be combined with existing differential privacy defense frameworks, such as the Opacus differential privacy framework. Is it possible to create a tutorial to demonstrate how to verify the effectiveness of differential privacy in defending against your MIA attack method? Thank you!

[MrLinNing]

Additionally, there is a puzzling issue in this tutorial. For the CIFAR-10 dataset, although the training accuracy is relatively high, at over 80%, the testing accuracy is quite poor, at less than 50%. This is an overfitting phenomenon, and the model has no practical value. Suppose we want to increase the test accuracy by changing the training structure or hyperparameters (learning rate, batch size), the resulting MIA ROC is almost the same as random guessing. In this case, it seems that the MIA attack becomes meaningless. How should we understand this situation?

[taojiashu]

Hi @MrLinNing, we have published a new version of the tool that allows verifying the DP parameters of DP training/models. You can refer to this file for instructions. The entire tool has been overhauled and now uses the state-of-the-art MIA, RMIA, as its testing backbone. You can try some of the examples we provided in this version.

[dpkpathak]

Hi @taojiashu, I’m trying to understand how to verify the offline settings in the current implementation. I reviewed the run_mia.py file, but I’m unclear on how the data splits for reference model training are separated from the single target model. Could you provide some clarification? Thanks in advance!

[taojiashu]

The current implementation is based on RMIA. Given a dataset, half of the data will be randomly sampled to train the target model. For each pair of reference models, a fresh round of sampling will be conducted to select half of the dataset for training each reference model. For example, reference_model_0_0 trains on data [1,3,5], which are randomly selected, while reference_model_0_1 trains on the other half, which is [2,4,6]. In this way, each data point would have an equal number of reference models trained/not trained on it. In the offline attack, we only use reference models that are not trained on the data point.
The particular functional call is here

ml_privacy_meter/run_mia.py

Line 87 in 71fb4d0

data_splits, memberships = split_dataset_for_training(

The implementation is here

ml_privacy_meter/models/utils.py

Line 298 in 71fb4d0

def split_dataset_for_training(dataset_size, num_model_pairs):

Basically we create a Boolean split matrix where each row represents the model's usage of a particular point in training. Hope it explains.

[dpkpathak]

Thank you for your detailed response. However, I still find some discrepancies compared to your explanation.

The target model is defined as the model trained on the first dataset returned from:

ml_privacy_meter/run_mia.py

Line 87 in 71fb4d0

data_splits, memberships = split_dataset_for_training(

Later, during the calculation of the out signal, a reference model trained on the out world dataset is required, as seen here:

ml_privacy_meter/attacks.py

Line 34 in 71fb4d0

selected_signals = all_signals[:, columns]

I have a concern regarding whether the reference model’s signal truly comes only from the out world dataset. While debugging with a single reference model (using the default CIFAR-10 configuration), I observed that the system trains four models:

The first model is treated as the target model.

However, models 2 and 3 are still used as reference models.

According to the split_dataset_for_training method, the third model and the target model share some overlap in training data, which seems incorrect in a practical attack setting. Could you confirm if this behavior is intended?

Additionally, I noticed that in

ml_privacy_meter/attacks.py

Line 132 in 71fb4d0

mean_z = (1 + offline_a) / 2 * mean_out_z + (1 - offline_a) / 2

,
the probability Pr(z) is being approximated differently compared to the algorithm in the RMIA paper. Could you clarify the reasoning behind this approach?

[taojiashu]

Yes, the overlap in training data between models is correct, and identical to the setup of LiRA and RMIA. To understand why a completely disjoint dataset is not used to train the reference models, we need to understand the underlying privacy game that defines the threat model and definition of privacy. The current examples in this tool is based on the most prevalent privacy game, Membership Inference Game for Average Model and Record (Definition 3.1 in this paper). Under this framework, we audit the privacy risk of a training algorithm by analyzing "sample" models under with the algorithm. Both target and reference models are samples as they are trained with the target training algorithm and same training data distribution. In each trial of attack, we set one of the sample model as a target model, and use the rest as reference models. (Technically, in Privacy Meter, we give users the freedom to specify how many target models and reference models to use in the config file.) For an offline attack under this privacy game, we only use models that are not trained on the target point. Since there is no FIXED target model, it is infeasible to train a set of reference models for each target model. Hence, this setup can guarantee both correctness and efficiency.

If you want to train OUT models with a disjoint set of data, the question is how do you realistically get a disjoint set of data that are also from the same data distribution. This privacy game will be similar to Def 3.2 in this paper, where there is only one target model and the members are thus already fixed. Privacy Meter does not audit privacy defined in this privacy game right now out of the box, as an additional dataset needs to be passed in.

Lastly, the probability Pr(z) is computed in the exactly the same way as the RMIA paper. We just rearranged the terms for efficiency and brevity of coding.