Circom benchmarks

This repo contains circom files used for benchmarking different tools.

There are two types of tests:

• Frontend coverage tests: This tests are used for frontend circom parsers to study the feature coverage of the tool.
• Detector ground truth: Collection of circom circuits with defects used for testing the different analysis tools in Veridise.

Testing the circom frontend with this repo

You will need to clone circom and this repo.

Go to circom's repo and build it with cargo build --release. Make sure the binary is in target/release/circom. The script will look for the circom compiler in that path, if your compiler is in a different path you can specify the path in scripts/benchmarks.py with the --circom PATH flag.

Run the following commands to compile each test case with the frontend:

cd path/to/circom
cargo build --release
python scripts/benchmark.py --src path/to/this/repo --out report.csv --timeout 300

The results of the compilation will be in report.csv. Tests can either succeed, fail or timeout. Failures could also be either circom failures (the compiler refuses the output), panics or unimplemented errors.

Structure

Each test should be in a directory following this path:

tests/<test_category>/<type>/<test_name>

For example, test_category could be vulnerabilities, type could be uo, and test_name could be ArrayXOR, resulting in the directory tests/vulnerabilities/uo/ArrayXOR/.

Each test case should contain the following files:

• test.json: Description of the test case.
• src: Code of the test case.
• src/template_file.circom: The file that contains the targeted template.
• src/main.circom: An instantiation of the main template.
• src/template_file_dummy.circom: A modified file that contains the content of template_file.circom, along with additional code to bypass this issue: GitHub Issue #103.
• README.md: This file is automatically generated by test.json.

Each test.json should contain the following:

{
    "id": "LessThanPower_UO",
    "source": "ZKSolid",
    "project_url": "https://github.com/Electron-Labs/ed25519-circom/",
    "commit": "3fef353",
    "synthetic": false,
    "internal": false,
    "link_libraries": true,
	"setup": [],
    "tests": [
        {
            "id": "LessThanPower",
            "file": "src/lt.circom",
            "template": "LessThanPower",
            "main": "src/main.circom",
            "description": "Output signals 'out' are not constrained."
        }
    ]
}

• id: the id of the test case (should be any string)
• internal: boolean value to identify if a test case is from a private audit
• project_url: a pointer to the URL of the project where the vulnerability was found
• commit: commit hash of the project for the targeted test case(s)
• synthetic: is the test case a synthetic example or ported from a real-world project
• source: where did we get that benchmark from
• link_libraries: Instructs to the circom compiler if they should link the libraries in tests/libs while compiling the circuit. Optional and true by default
• setup: Defines a sequence of shell commands that need to be executed before compiling the circuits. Optional and an empty array by default.
• tests: a directory containing details of each sub-test case.
  • id: the id of the sub-test case (string)
  • file: the relative path to the circom file containing the targeted template
  • template: the name of the targeted template
  • main: the relative path to a circom file that instantiates the targeted template with arguments that will trigger the vulnerability (if there is a vulnerability test case)
  • description: the description of the test case