Conformal Prediction as Bayesian Quadrature

Code to accompany "Conformal Prediction as Bayesian Quadrature" by Jake Snell & Tom Griffiths (ICML 2025 Outstanding Paper).

Dependencies

• uv for managing python packages and dependencies
• just for running commands
• gdown for downloading MS-COCO data

Running Tests

Run just test.

Running Synthetic Binomial Experiments

1. Be sure that the output directory exists (e.g. by running mkdir output).
2. Run just synth-run {method}, where {method} is crc for Conformal Risk Control, rcps for Risk-controlling Prediction Sets, or hpd for our highest posterior density method. This will create a CSV file in output that contains the results of the experiment.
3. To summarize the results, run just synth-analyze {method}.

Running Synthetic Heteroskedastic Experiments

Follow the same steps as the synthetic binomial experiments, but replace synth with heteroskedastic.

1. Be sure that the output directory exists (e.g. by running mkdir output).
2. Run just heteroskedastic-run {method}, where {method} is crc for Conformal Risk Control, rcps for Risk-controlling Prediction Sets, or hpd for our highest posterior density method. This will create a CSV file in output that contains the results of the experiment.
3. To summarize the results, run just heteroskedastic-analyze {method}.

Running MS-COCO Experiments

First, run just fetch to download the necessary data¹. Then, follow the same steps as the heteroskedastic experiments above but replace heteroskedastic with coco.

1. Be sure that the output directory exists (e.g. by running mkdir output).
2. Run just coco-run {method}, where {method} is crc for Conformal Risk Control, rcps for Risk-controlling Prediction Sets, or hpd for our highest posterior density method. This will create a CSV file in output that contains the results of the experiment.
3. To summarize the results, run just coco-analyze {method}.

Footnotes

1. Data credit: conformal-prediction. ↩