feat: merge-train/barretenberg

barretenberg/cpp/scripts/audit/README.txt

@@ -10,19 +10,19 @@ HOW IT WORKS
    Each file being audited includes a structured comment block with audit status for multiple roles, e.g.:
 
    // === AUDIT STATUS ===
-   // internal:    { status: not started, auditors: [], date: YYYY-MM-DD }
-   // external_1:  { status: not started, auditors: [], date: YYYY-MM-DD }
-   // external_2:  { status: not started, auditors: [], date: YYYY-MM-DD }
+   // internal:    { status: not started, auditors: [], commit: }
+   // external_1:  { status: not started, auditors: [], commit: }
+   // external_2:  { status: not started, auditors: [], commit: }
    // =====================
 
    The script generate_audit_status_headers.sh populates all source files (in included dirs) with this header, unless one is already present.
 
 2. The header in each file should be manually updated when the audit status changes, e.g.:
 
    // === AUDIT STATUS ===
-   // internal:    { status: complete, auditors: [luke], date: 2025-04-17 }
-   // external_1:  { status: not started, auditors: [], date: YYYY-MM-DD }
-   // external_2:  { status: not started, auditors: [], date: YYYY-MM-DD }
+   // internal:    { status: Complete, auditors: [Luke], commit: abc123def }
+   // external_1:  { status: not started, auditors: [], commit: }
+   // external_2:  { status: not started, auditors: [], commit: }
    // =====================
 
 3. Summary generation
@@ -31,8 +31,37 @@ HOW IT WORKS
 4. Dashboard visualization
    A static HTML file (audit_dashboard.html) visualizes the JSON using pie charts, with one chart per audit role per module. A summary section at the top gives a full-repo overview.
 
-USAGE
------
+
+CLI USAGE
+---------
+
+The Python script supports multiple modes:
+
+# Generate audit_summary.json (default)
+python3 generate_audit_summary.py
+
+# List files with incomplete internal audit
+python3 generate_audit_summary.py --list-unaudited
+
+# List files with incomplete audit, filtered by directory
+python3 generate_audit_summary.py --list-unaudited --dir chonk
+python3 generate_audit_summary.py --list-unaudited --dir stdlib/primitives
+
+# List files without any audit headers
+python3 generate_audit_summary.py --list-missing
+
+# List files with complete internal audit
+python3 generate_audit_summary.py --list-complete
+
+# Check a different audit role (default: internal)
+python3 generate_audit_summary.py --list-unaudited --role external_1
+
+# Output as JSON instead of human-readable
+python3 generate_audit_summary.py --list-unaudited --json
+
+
+DASHBOARD USAGE
+---------------
 
 1. Run the server on the remote machine:
 
@@ -45,3 +74,21 @@ USAGE
 3. View the dashboard in your browser:
 
    http://localhost:8080/audit_dashboard.html
+
+
+QUICK EXAMPLES
+--------------
+
+# See what needs auditing in stdlib
+python3 generate_audit_summary.py --list-unaudited --dir stdlib
+
+# See which files are missing headers entirely
+python3 generate_audit_summary.py --list-missing
+
+# Quick stats
+python3 generate_audit_summary.py
+# Output:
+#   Total files with headers: 551
+#   Internal audit complete:  172
+#   Internal audit pending:   379
+#   Files missing headers:    1103

barretenberg/cpp/scripts/audit/audit_scopes/aes128_audit_scope.md

@@ -0,0 +1,35 @@
+# External Audit Scope: aes128
+
+Repository: https://github.com/AztecProtocol/aztec-packages
+Commit hash: TBD (link)
+
+## Files to Audit
+Note: Paths relative to `aztec-packages/barretenberg/cpp/src/barretenberg`
+
+### Native AES-128 Implementation
+1. `crypto/aes128/aes128.cpp`
+2. `crypto/aes128/aes128.hpp`
+3. `crypto/aes128/c_bind.cpp`
+4. `crypto/aes128/c_bind.hpp`
+
+### Circuit-Friendly AES-128 Implementation
+5. `stdlib/encryption/aes128/aes128.cpp`
+6. `stdlib/encryption/aes128/aes128.hpp`
+
+### ACIR Integration
+7. `dsl/acir_format/aes128_constraint.cpp`
+8. `dsl/acir_format/aes128_constraint.hpp`
+
+### Lookup Tables
+9. `stdlib_circuit_builders/plookup_tables/aes128.hpp`
+
+## Summary of Module
+
+The `aes128` module provides both native and circuit-friendly implementations of the AES-128 block cipher with CBC (Cipher Block Chaining) mode. The native implementation in `crypto/aes128` provides standard AES-128 encryption/decryption for general use, including C bindings for external interfaces. The circuit-friendly implementation in `stdlib/encryption/aes128` enables AES-128 encryption to be proven inside arithmetic circuits using lookup tables to represent the S-box operations efficiently. The ACIR format integration allows AES-128 constraints to be specified in Noir programs, and the plookup tables module provides the lookup table definitions needed for efficient in-circuit S-box lookups.
+
+## Test Files
+1. `crypto/aes128/aes128.test.cpp`
+2. `stdlib/encryption/aes128/aes128.test.cpp`
+
+## Security Mechanisms
+None identified.

barretenberg/cpp/scripts/audit/audit_scopes/avm_recursive_verifier_audit_scope.md

@@ -0,0 +1,33 @@
+# External Audit Scope: avm_recursive_verifier
+
+Repository: https://github.com/AztecProtocol/aztec-packages
+Commit hash: TBD (link)
+
+## Files to Audit
+Note: Paths relative to `aztec-packages/barretenberg/cpp/src/barretenberg`
+
+### AVM Recursive Verifier Core
+1. `vm2/constraining/recursion/recursive_verifier.hpp`
+2. `vm2/constraining/recursion/recursive_verifier.cpp`
+3. `vm2/constraining/recursion/recursive_flavor.hpp`
+4. `vm2/constraining/recursion/recursive_flavor_settings.hpp`
+5. `vm2/constraining/recursion/goblin_avm_recursive_verifier.hpp`
+
+### Flavor Definitions
+6. `flavor/mega_avm_flavor.hpp`
+7. `flavor/mega_avm_recursive_flavor.hpp`
+
+### ACIR Integration
+8. `vm2/dsl/avm2_recursion_constraint.hpp`
+9. `vm2/dsl/avm2_recursion_constraint.cpp`
+10. `dsl/acir_format/avm2_recursion_constraint.hpp`
+
+## Summary of Module
+
+The AVM recursive verifier implements in-circuit verification of AVM2 proofs using two-layer recursive composition for efficiency. The core recursive verifier (AvmRecursiveVerifier) performs Mega-arithmetized verification of AVM2 proofs with a fixed verification key, implementing sumcheck verification, public input validation through multilinear evaluation, and Shplemini batch opening. The Goblin-based recursive verifier (AvmGoblinRecursiveVerifier) provides a two-phase approach: an inner Mega circuit recursively verifies the AVM2 proof and produces a MegaHonk proof plus Goblin proof (ECCVM, Translator, Merge), then an outer Ultra circuit verifies both proofs with hash-based consistency checks to ensure proper transfer of verifier inputs between layers. The recursive flavor definitions configure the circuit builder, curve operations (bn254), and commitment scheme (KZG) for the Mega-arithmetized recursive verifier. The ACIR integration layer (avm2_recursion_constraint) provides the interface for Noir programs to trigger AVM2 recursive verification, handling proof deserialization and witness generation for integration with the Ultra circuit builder.
+
+## Test Files
+1. `vm2/constraining/recursion/recursive_verifier.test.cpp`
+2. `vm2/dsl/avm2_recursion_constraint.test.cpp`
+
+## Security Mechanisms

barretenberg/cpp/scripts/audit/audit_scopes/avm_utils_audit_scope.md

@@ -0,0 +1,22 @@
+# External Audit Scope: AVM Utils
+
+Repository: https://github.com/AztecProtocol/aztec-packages
+Commit hash: TBD
+
+## Files to Audit
+
+### Generic Relations
+1. `barretenberg/cpp/src/barretenberg/relations/generic_lookup/generic_lookup_relation.hpp`
+2. `barretenberg/cpp/src/barretenberg/relations/generic_permutation/generic_permutation_relation.hpp`
+
+## Brief Summary of Module
+
+Utility relations used by the AVM (Aztec Virtual Machine) for generic lookup and permutation operations.
+
+## Test Files
+
+TBD
+
+## Security Mechanisms
+
+TBD

barretenberg/cpp/scripts/audit/audit_scopes/biggroup_audit_scope.md

@@ -0,0 +1,36 @@
+# External Audit Scope: Biggroup
+
+Repository: https://github.com/AztecProtocol/aztec-packages
+Commit hash: [553c5eb82901955c638b943065acd3e47fc918c0](https://github.com/AztecProtocol/aztec-packages/tree/553c5eb82901955c638b943065acd3e47fc918c0)
+
+## Files to Audit
+
+The following files are to be audited, located in the `stdlib/primitives/biggroup` module (in no particular order):
+
+1. `stdlib/primitives/biggroup/biggroup.hpp`
+2. `stdlib/primitives/biggroup/biggroup_impl.hpp`
+3. `stdlib/primitives/biggroup/biggroup_nafs.hpp`
+4. `stdlib/primitives/biggroup/biggroup_tables.hpp`
+5. `stdlib/primitives/biggroup/biggroup_secp256k1.hpp`
+6. `stdlib/primitives/biggroup/biggroup_edgecase_handling.hpp`
+
+Update: Fixed lookup tables are implemented in `stdlib_circuit_builders/plookup_tables/non_native_group_generator.cpp` which must be added to the scope.
+
+7. `stdlib_circuit_builders/plookup_tables/non_native_group_generator.cpp`
+8. `stdlib_circuit_builders/plookup_tables/non_native_group_generator.hpp`
+
+## Brief Summary of Module
+
+The biggroup module implements elliptic-curve operations using UltraHonk arithmetisation in barretenberg. This is specifically implemented to work for three curves[^1]: bn254, secp256k1 and secp256r1.
+
+Please refer to the [biggroup README](https://github.com/AztecProtocol/aztec-packages/blob/553c5eb82901955c638b943065acd3e47fc918c0/barretenberg/cpp/src/barretenberg/stdlib/primitives/biggroup/README.md) for details on the specification and implementation details.[^2]
+
+## Test Files
+1. `stdlib/primitives/biggroup/biggroup.test.cpp`
+2. `stdlib/primitives/biggroup/biggroup_secp256k1.test.cpp`
+
+
+
+[^1]: Our implementation can _technically_ work for other curves as well (so long as the base and scalar fields of the curve can be represented with our bigfield module) but we have not tested it for other curves.
+
+[^2]: The README uses Latex notation which doesn't render well on Github, you might need to use Markdown preview in VS code to render the file.

barretenberg/cpp/scripts/audit/audit_scopes/blake2s_blake3_audit_scope.md

@@ -0,0 +1,77 @@
+# BLAKE2s + BLAKE3 Audit Scope
+
+Repository: https://github.com/AztecProtocol/aztec-packages
+Commit hash: 4a956ceb179c2fe855e4f1fd78f2594e7fc3f5ea
+
+### Files to audit
+
+#### Native implementation
+1. ```barretenberg/cpp/src/barretenberg/crypto/blake2s/blake2s.hpp```
+2. ```barretenberg/cpp/src/barretenberg/crypto/blake2s/blake2s.cpp```
+3. ```barretenberg/cpp/src/barretenberg/crypto/blake2s/blake2-impl.hpp```
+4. ```barretenberg/cpp/src/barretenberg/crypto/blake3s/blake3s.hpp```
+5. ```barretenberg/cpp/src/barretenberg/crypto/blake3s/blake3s.tcc```
+6. ```barretenberg/cpp/src/barretenberg/crypto/blake3s/blake3-impl.hpp```
+
+#### Stdlib circuit implementation
+
+7. ```barretenberg/cpp/src/barretenberg/stdlib/hash/blake2s/blake2s.hpp```
+8. ```barretenberg/cpp/src/barretenberg/stdlib/hash/blake2s/blake2s.cpp```
+9. ```barretenberg/cpp/src/barretenberg/stdlib/hash/blake3s/blake3s.hpp```
+10. ```barretenberg/cpp/src/barretenberg/stdlib/hash/blake3s/blake3s.cpp```
+11. ```barretenberg/cpp/src/barretenberg/stdlib/hash/blake2s/blake_util.hpp```
+
+#### Lookup related
+12. ```barretenberg/cpp/src/barretenberg/stdlib/primitives/plookup/plookup.cpp```
+    - for `lookup_read<Builder>::get_lookup_accumulators` and its path from thereon
+13. ```barretenberg/cpp/src/barretenberg/stdlib_circuit_builders/plookup_tables/plookup_tables.hpp```
+```barretenberg/cpp/src/barretenberg/stdlib_circuit_builders/plookup_tables/plookup_tables.cpp```
+    - for `ReadData<bb::fr> get_lookup_accumulators`
+14. ```barretenberg/cpp/src/barretenberg/stdlib_circuit_builders/plookup_tables/blake2s.hpp```
+    - for BLAKE tables such as `BLAKE_XOR`, `BLAKE_XOR_ROTATE_16`, `BLAKE_XOR_ROTATE_8`, etc.
+
+#### DSL/ACIR format
+15. ```barretenberg/cpp/src/barretenberg/dsl/acir_format/blake2s_constraint.hpp```
+16. ```barretenberg/cpp/src/barretenberg/dsl/acir_format/blake2s_constraint.cpp```
+17. ```barretenberg/cpp/src/barretenberg/dsl/acir_format/blake3_constraint.hpp```
+18. ```barretenberg/cpp/src/barretenberg/dsl/acir_format/blake3_constraint.cpp```
+
+#### Tests
+19. ```barretenberg/cpp/src/barretenberg/crypto/blake2s/blake2s.test.cpp```
+    - a set of test vectors that cover all message lengths from 0 to 72 bytes
+20.  ```barretenberg/cpp/src/barretenberg/stdlib/hash/blake2s/blake2s.test.cpp```
+      - a set of test vectors that cover varying message lengths up to 72 bytes including (powers of 2) boundaries, tested against the native implementation hash
+      -  single and double block tests
+      -  all witness, all constant and witness+constant message as input
+21. ```barretenberg/cpp/src/barretenberg/dsl/acir_format/blake2s_constraint.test.cpp```
+    - ACIR integration test
+22. ```barretenberg/cpp/src/barretenberg/crypto/blake3s/blake3s.test.cpp```
+    - a set of test vectors that cover all message lengths from 0 to 72 bytes
+    - subset (unkeyed hash mode, with varying message length $\leq$ 1024 bytes) of official BLAKE3 test vectors
+    - a test to check that inputs greater than 1024 bytes trigger the intended assertion
+23. ```barretenberg/cpp/src/barretenberg/stdlib/hash/blake3s/blake3s.test.cpp```
+    - a set of test vectors that cover varying message lengths up to 72 bytes including (powers of 2) boundaries.
+    - single and double block tests
+    - all witness, all constant and witness+constant message as input
+24. ```barretenberg/cpp/src/barretenberg/dsl/acir_format/blake3_constraint.test.cpp```
+    - ACIR integration test
+
+
+
+### Summary of the module
+#### BLAKE2s
+Implements the unkeyed, sequential BLAKE2s and outputs a 32 byte hash. XOR+Rotate operations are implemented via lookups. The relevant files related to lookups used in BLAKE are mentioned above.
+
+#### BLAKE3
+A restricted variant of BLAKE3 with inputs limited to $\leq$ 1024 bytes that generates a 32 byte hash. This kind of constraint on the input size simplifies the code and helps get rid of the recursive merkle-tree like operations on chunks (data of size 1024 bytes). This is because in Barretenberg, BLAKE3 is only used to hash inputs of size 32 bytes (or lesser).
+
+### Documentation
+
+BLAKE2s: https://www.blake2.net/blake2.pdf
+
+BLAKE3: https://github.com/BLAKE3-team/BLAKE3-specs/blob/master/blake3.pdf
+
+Lookup tables: https://github.com/AztecProtocol/aztec-packages/blob/next/barretenberg/cpp/src/barretenberg/stdlib_circuit_builders/plookup_tables/README.md
+
+### Note
+Previously, certain inputs were pushing the addition overflows in `g` to beyond 3 bits (where `add_normalize` can tolerate up to 3 bits of overflow), causing failures. This has been addressed by calling `add_normalize` in the second half of every call to `g` to ensure that the overflow doesn't go beyond 3 bits. The input that was causing failures earlier has been added as a test case now. A detailed description of the issue can be found here: https://hackmd.io/@aztec-network/SyTHLkAWZx.