Running MIA Attacks

Running MIA Attacks

This page explains how to configure and run Membership Inference Attacks (MIA) in LeakPro by editing the audit.yaml file.

---

How it works

LeakPro runs attacks based on what you declare in audit.yaml. You pick which attacks to run, set their parameters, and point them at your target model and data. No code changes needed.

---

audit.yaml structure

audit:
  random_seed: 1234
  attack_type: "mia"
  data_modality: "image"     # "image" or "tabular"
  output_dir: "./leakpro_output"
  attack_list:
    - attack: lira            # attack key (see table below)
      num_shadow_models: 4    # attack-specific params (optional, overrides defaults)

target:
  module_path: "./target_model_class.py"
  model_class: "MyModel"
  target_folder: "./target"
  data_path: "./data/dataset.pkl"

shadow_model:

distillation_model:

• attack_list accepts multiple entries — you can run several attacks in one go.
• Any parameter you omit falls back to its default value.
• shadow_model and distillation_model sections are needed only for attacks that train shadow/distillation models (see per-attack notes below).

---

Available attacks

attack key	Attack name	Type	Compute cost
population	Population Attack	Label-only	Low
HSJ	HopSkipJump	Label-only	Medium
lira	LiRA	Shadow model	Medium
base	BASE	Shadow model	Medium
rmia	RMIA	Shadow model	Medium
qmia	QMIA	Shadow model	Medium
yoqo	YOQO	Shadow model	Medium
multi_signal_lira	Multi-Signal LiRA	Shadow model	Medium
ramia	RaMIA	Shadow model	Medium–High
loss_traj	Loss Trajectory	Distillation	High
dts	DTS	Shadow model	High
oslo	OSLO	Shadow model	High

---

Common parameters

Most shadow-model attacks share these parameters. They are explained here once.

Parameter	Type	Default	Description
num_shadow_models	int	varies	Number of shadow models to train. More → better signal, more compute.
training_data_fraction	float	0.5	Fraction of auxiliary data used to train each shadow model.
online	bool	False	Online mode: shadow models are trained with the audit samples included — stronger attack, requires num_shadow_models >= 2. Offline mode: audit samples are excluded — weaker but cheaper.

---

Per-attack reference

---

population

Simple baseline attack. Compares the target model's loss on a sample against the loss distribution over a reference population. No shadow models needed.

Parameter	Type	Default	Notes
attack_data_fraction	float	0.5	Fraction of population data to use as reference.

- attack: population
  attack_data_fraction: 0.5

---

HSJ

HopSkipJump — a label-only attack based on decision boundary distance. Works without confidence scores, only needs predicted labels.

Note: The attack key is case-sensitive: use HSJ, not hsj.

Parameter	Type	Default	Notes
attack_data_fraction	float	0.1	Fraction of data to audit.
norm	int or float	2	Distance norm: 1, 2, or inf.
initial_num_evals	int	100	Boundary search evaluations at start.
max_num_evals	int	10000	Maximum boundary search evaluations.
num_iterations	int	2	Number of attack iterations per sample.
gamma	float	1.0	Step scaling factor.
constraint	int	2	Constraint type: 1 or 2.
batch_size	int	64	Batch size during evaluation.
epsilon_threshold	float	1e-6	Early stopping threshold.

- attack: HSJ
  attack_data_fraction: 0.1
  num_iterations: 5

---

lira

Likelihood Ratio Attack. Trains shadow models to estimate the distribution of logits for members vs. non-members using a Gaussian likelihood ratio test.

Parameter	Type	Default	Notes
num_shadow_models	int	1	Online mode requires >= 2.
training_data_fraction	float	0.5	See common params.
online	bool	False	See common params.
var_calculation	"carlini" / "individual_carlini" / "fixed"	"carlini"	How variance is estimated for the Gaussian model.

- attack: lira
  num_shadow_models: 4
  online: False
  var_calculation: "carlini"

---

base

BASE attack — a simpler shadow-model baseline using LogSumExp thresholding.

Parameter	Type	Default	Notes
num_shadow_models	int	1	Online mode requires >= 2.
training_data_fraction	float	0.5	See common params.
online	bool	False	See common params.
temperature	float	2.0	Softmax temperature applied to logits.
offline_scale_factor	float	0.33	Rescales the LogSumExp threshold to compensate for the absence of in-models (offline only).

- attack: base
  num_shadow_models: 2
  online: False
  temperature: 2.0

---

rmia

Relative MIA. Improves on LiRA by normalising the likelihood ratio against a reference distribution p(z) estimated from auxiliary data.

Parameter	Type	Default	Notes
num_shadow_models	int	1	Online mode requires >= 2.
training_data_fraction	float	0.5	See common params.
online	bool	False	See common params.
temperature	float	2.0	Softmax temperature.
z_data_sample_fraction	float	0.5	Fraction of auxiliary data used to estimate p(z).
gamma	float	2.0	Threshold on the likelihood ratio. Optuna-tunable (0.1–10, log scale).
offline_a	float	0.33	Approximation of the marginal p(x) (offline only). Optuna-tunable (0.0–1.0).

- attack: rmia
  num_shadow_models: 2
  online: False
  gamma: 2.0

---

qmia

Quantile MIA. Trains a quantile regressor on shadow model outputs to build a membership signal without assuming a parametric distribution.

Parameter	Type	Default	Notes
training_data_fraction	float	0.5	See common params.
quantiles	list[float]	[0.05, 0.25, 0.5, 0.75, 0.95]	Quantiles to estimate. All values must be in (0, 1).
epochs	int	100	Training epochs for the quantile regressor.

- attack: qmia
  training_data_fraction: 0.5
  epochs: 100
  quantiles: [0.05, 0.25, 0.5, 0.75, 0.95]

---

yoqo

YOQO (You Only Query Once). Optimises a synthetic proxy sample x' close to the target x so that shadow models trained with x but not x' can be distinguished using only one query per sample.

Parameter	Type	Default	Notes
num_shadow_models	int	1	Online mode requires >= 2.
training_data_fraction	float	0.01	Small fraction is typical — each shadow model is cheap.
online	bool	False	See common params.
lr_xprime_optimization	float	0.001	Learning rate for optimising x'.
max_iterations	int	1	Iterations of the x' optimisation loop.

- attack: yoqo
  num_shadow_models: 2
  training_data_fraction: 0.01
  lr_xprime_optimization: 0.001

---

multi_signal_lira

Extension of LiRA that combines multiple model signals (e.g. logits, loss, gradients) into the likelihood ratio test.

Parameter	Type	Default	Notes
num_shadow_models	int	1	Online mode requires >= 2.
training_data_fraction	float	0.5	See common params.
online	bool	False	See common params.
signal_names	list[str]	["ModelRescaledLogits"]	Signals to use.
var_calculation	"carlini" / "individual_carlini" / "fixed"	"carlini"	Variance estimation method.
std_eps	float	1e-30	Small constant added to std for numerical stability.

- attack: multi_signal_lira
  num_shadow_models: 4
  signal_names: ["ModelRescaledLogits"]
  online: False

---

ramia

Range MIA. Augments each sample into a range of nearby points and uses the spread of membership signals across that range as the attack feature. Optionally supports stealth mode via group testing to reduce per-sample query cost.

Parameter	Type	Default	Notes
num_shadow_models	int	2	Online mode requires >= 2.
training_data_fraction	float	0.5	See common params.
online	bool	False	See common params.
num_transforms	int	10	Augmentations per sample.
n_ops	int	1	Augmentation operations per transform.
augment_strength	"none" / "easy" / "medium" / "strong"	"easy"	Strength of augmentations applied to generate the range.
num_audit	int	1000	Number of member samples to audit (non-members are matched).
qs	float	0.4	Start quantile for trimmed averaging of transformed scores.
qe	float	1.0	End quantile for trimmed averaging of transformed scores.
stealth_method	"none" / "bcjr" / "agglomerative"	"none"	Group testing strategy to hide which individual was queried.
groups	int	8	Number of groups (stealth mode only).
groups_per_sample	int	3	Groups each sample is assigned to (stealth mode only).
n_comp	int	2	PCA/UMAP components for latent features in group testing.
group_score_threshold	float	0.5	Threshold for deciding if a group contains a member.

- attack: ramia
  num_shadow_models: 4
  num_transforms: 8
  augment_strength: "strong"
  online: False

---

loss_traj

Loss Trajectory Attack. Trains a distillation model and records the loss trajectory across training epochs as the membership signal, then classifies with a small MIA classifier.

Note: requires a distillation_model section in audit.yaml.

Parameter	Type	Default	Notes
distillation_data_fraction	float	0.5	Fraction of auxiliary data used for distillation.
train_mia_batch_size	int	64	Batch size for training the MIA classifier.
number_of_traj	int	1	Number of loss trajectories to record.
mia_classifier_epochs	int	100	Training epochs for the MIA classifier.
label_only	bool	False	Use only labels (not logits) as the signal.
temperature	float	2.0	Softmax temperature applied to logits.

- attack: loss_traj
  distillation_data_fraction: 0.5
  number_of_traj: 3
  mia_classifier_epochs: 100

---

seqmia

Sequential-Metric Based Membership Inference Attack. Trains a student and target distillation model and records the loss trajectory across training epochs as the membership signal, then classifies with a LSTM MIA classifier.

Note: requires the sum of f_train and f_test to be <1.0 train_config.yaml.

Parameter	Type	Default	Notes
distillation_data_fraction	float	0.5	Fraction of auxiliary data used for distillation.
train_mia_batch_size	int	64	Batch size for training the MIA classifier.
number_of_traj	int	1	Number of loss trajectories to record.
mia_classifier_epochs	int	100	Training epochs for the MIA classifier.
mia_classifier_lr	float	0.0001	Learning rate for training the MIA classifier.
mia_classifier_momentum	float	0.9	Momentum for training the MIA classifier.
mia_classifier_weight_decay	float	0.0	Weight decay for training the MIA classifier.
attention_model	bool	True	Whether to use an LSTM with or without attention.
label_only	bool	False	Use only labels (not logits) as the signal.
temperature	float	2.0	Softmax temperature applied to logits.

- attack: seqmia
  number_of_traj: 50
  mia_classifier_epochs: 1000

---

dts

Decision Tree Shadow (DTS). Trains many shadow models and feeds their per-sample loss trajectories into a learned classifier (LSTM or InceptionTime) to predict membership. High shadow model count is intentional.

Note: online: True by default — requires num_shadow_models >= 2.

Parameter	Type	Default	Notes
num_shadow_models	int	16	High count is expected; more models = better classifier training data.
training_data_fraction	float	0.5	See common params.
online	bool	True	Online by default. Requires num_shadow_models >= 2.
clf_model	"LSTM" / "InceptionTime"	"LSTM"	Architecture for the MIC (membership inference classifier).
clf_model_kwargs	dict	None	Extra keyword arguments passed to the MIC constructor.
clf_data_fraction	float	0.1	Fraction of shadow population appended to MIC training data per shadow model.
clf_batch_size	int	128	Batch size for MIC training.
clf_max_epochs	int	32	Max training epochs for MIC.
clf_val_fraction	float	0.2	Fraction of MIC data used for validation / early stopping.
clf_early_stopping_patience	int	2	Epochs without improvement before early stopping triggers.
clf_fit_verbose	0 / 1	0	1 prints loss and accuracy per epoch.

- attack: dts
  num_shadow_models: 16
  online: True
  clf_model: "LSTM"
  clf_max_epochs: 32

---

oslo

OSLO. Optimises a perturbed version x' of each sample to maximally separate member and non-member signals, then uses shadow models to threshold the result. Image-specific — not suited for tabular data.

Parameter	Type	Default	Notes
num_source_models	int	9	Shadow models used to optimise x'.
num_validation_models	int	3	Shadow models used to validate the threshold.
training_data_fraction	float	0.5	See common params.
online	bool	False	See common params.
n_audits	int	500	Number of data points to audit.
num_sub_procedures	int	80	Outer optimisation iterations.
num_iterations	int	5	Inner iterations per sub-procedure.
step_size	float	0.01	Step size for x' optimisation.
max_perturbation_size	float	0.314	Maximum L-inf distance between x and x' (~80/255 for images in [0,1]).
min_threshold	float	0.0001	Lower bound of threshold search range.
max_threshold	float	1.0	Upper bound of threshold search range.
n_thresholds	int	5	Number of thresholds to evaluate.

- attack: oslo
  num_source_models: 6
  num_validation_models: 2
  n_audits: 500
  online: True
  step_size: 0.01
  max_perturbation_size: 0.314

---

Running multiple attacks at once

You can stack as many attacks as you like in attack_list:

audit:
  random_seed: 1234
  attack_type: "mia"
  data_modality: "image"
  output_dir: "./leakpro_output"
  attack_list:
    - attack: population
    - attack: lira
      num_shadow_models: 4
    - attack: rmia
      num_shadow_models: 4
      online: False

Shadow models are shared across attacks where possible, so running multiple attacks together is more efficient than running them separately.