Merge branch 'feat/new-execution' into feat/sha-512-new-execution

Author: Avaneesh-axiom

.github/workflows/cli.yml

@@ -3,6 +3,7 @@ name: OpenVM CLI Tests
 on:
   push:
     branches: ["main"]
+    tags: ["v*"]
   pull_request:
     branches: ["**"]
     paths:
@@ -14,6 +15,13 @@ on:
       - "examples/**"
       - "Cargo.toml"
       - ".github/workflows/cli.yml"
+  workflow_dispatch:
+    inputs:
+      use_local_openvm:
+        description: "Test cargo openvm init using local patch"
+        required: true
+        type: boolean
+        default: false
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
@@ -76,6 +84,14 @@ jobs:
           export RUST_BACKTRACE=1
           cargo openvm keygen --config ./example/app_config.toml --output-dir .
 
+      - name: Set USE_LOCAL_OPENVM environment variable
+        run: |
+          if [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == refs/tags/* ]] || [[ "${{ github.event_name }}" == "workflow_dispatch" && "${{ github.event.inputs.use_local_openvm }}" == "false" ]]; then
+            echo "USE_LOCAL_OPENVM=0" >> $GITHUB_ENV
+          else
+            echo "USE_LOCAL_OPENVM=1" >> $GITHUB_ENV
+          fi
+
       - name: Run CLI tests
         working-directory: crates/cli
         run: |

book/src/SUMMARY.md

@@ -17,7 +17,7 @@
 - [Verifying Proofs](./writing-apps/verify.md)
 - [Solidity SDK](./writing-apps/solidity.md)
 
-# Using Extensions
+# Acceleration Using Extensions
 
 - [Overview](./custom-extensions/overview.md)
 - [Keccak](./custom-extensions/keccak.md)
@@ -27,12 +27,20 @@
 - [Elliptic Curve Cryptography](./custom-extensions/ecc.md)
 - [Elliptic Curve Pairing](./custom-extensions/pairing.md)
 
+# Guest Libraries
+
+- [Keccak256](./guest-libs/keccak256.md)
+- [SHA2](./guest-libs/sha2.md)
+- [Ruint](./guest-libs/ruint.md)
+- [K256](./guest-libs/k256.md)
+- [P256](./guest-libs/p256.md)
+- [Pairing](./guest-libs/pairing.md)
+- [Verify STARK](./guest-libs/verify-stark.md)
+
 # Advanced Usage
 
 - [SDK](./advanced-usage/sdk.md)
 - [Creating a New Extension](./advanced-usage/new-extension.md)
 - [Recursive Verification](./advanced-usage/recursion.md)
 
-# Guest Libraries
 
-- [Verify STARK](./guest-libs/verify-stark.md)

book/src/custom-extensions/algebra.md

@@ -22,7 +22,7 @@ The functional part is provided by the `openvm-algebra-guest` crate, which is a
 
 ## Modular arithmetic
 
-To [leverage](./overview.md) compile-time known moduli for performance, you declare, initialize, and then set up the arithmetic structures:
+To [leverage](./overview.md) compile-time known moduli for performance, you declare and initialize the arithmetic structures:
 
 1. **Declare**: Use the `moduli_declare!` macro to define a modular arithmetic struct. This can be done multiple times in various crates or modules:
 
@@ -37,10 +37,10 @@ This creates `Bls12_381Fp` and `Bn254Fp` structs, each implementing the `IntMod`
 Since both moduli are prime, both structs also implement the `Field` and `Sqrt` traits.
 The modulus parameter must be a string literal in decimal or hexadecimal format.
 
-2. **Init**: Use the `init!` macro exactly once in the final binary:
+2. **Init**: Use the [`openvm::init!` macro](./overview.md#automating-the-init-step) exactly once in the final binary:
 
 ```rust
-init!();
+openvm::init!();
 /* This expands to
 moduli_init! {
     "0x1a0111ea397fe69a4b1ba7b6434bacd764774b84f38512bf6730d2a0f6b0f6241eabfffeb153ffffb9feffffffffaaab",
@@ -70,10 +70,10 @@ complex_declare! {
 
 This creates a `Bn254Fp2` struct, representing a complex extension field. The `mod_type` must implement `IntMod`.
 
-2. **Init**: After calling `complex_declare!`, the `init!` macro will now expand to the appropriate call to `complex_init!`.
+2. **Init**: After calling `complex_declare!`, the [`openvm::init!` macro](./overview.md#automating-the-init-step) will now expand to the appropriate call to `complex_init!`.
 
 ```rust
-init!();
+openvm::init!();
 /* This expands to:
 moduli_init! {
     "0x1a0111ea397fe69a4b1ba7b6434bacd764774b84f38512bf6730d2a0f6b0f6241eabfffeb153ffffb9feffffffffaaab",
@@ -115,9 +115,7 @@ To have the correct imports for the above example, add the following to the `Car
 ```toml
 [dependencies]
 openvm = { git = "https://github.com/openvm-org/openvm.git" }
-openvm-platform = { git = "https://github.com/openvm-org/openvm.git" }
 openvm-algebra-guest = { git = "https://github.com/openvm-org/openvm.git" }
-openvm-algebra-complex-macros = { git = "https://github.com/openvm-org/openvm.git" }
 serde = { version = "1.0.216", default-features = false }
 ```
 

book/src/custom-extensions/bigint.md

@@ -29,20 +29,6 @@ All of the operations can be used in 6 different ways:
 
 When using the `U256` struct with `target_os = "zkvm"`, the struct utilizes efficient implementations of comparison operators as well as the `clone` method.
 
-### Example matrix multiplication using `U256`
-
-See the full example [here](https://github.com/openvm-org/openvm/blob/main/examples/u256/src/main.rs).
-
-```rust,no_run,noplayground
-{{ #include ../../../examples/u256/src/main.rs }}
-```
-
-To be able to import the `U256` struct, add the following to your `Cargo.toml` file:
-
-```toml
-openvm-bigint-guest = { git = "https://github.com/openvm-org/openvm.git" }
-```
-
 ## `I256`
 
 The `I256` struct is a 256-bit signed integer type. The `I256` struct is very similar to the `U256` struct.
@@ -70,19 +56,7 @@ The `I256` struct implements the following constructors: `from_i8`, `from_i32`,
 
 When using the `I256` struct with `target_os = "zkvm"`, the struct utilizes efficient implementations of comparison operators as well as the `clone` method.
 
-### Example matrix multiplication using `I256`
-
-See the full example [here](https://github.com/openvm-org/openvm/blob/main/examples/i256/src/main.rs).
-
-```rust,no_run,noplayground
-{{ #include ../../../examples/i256/src/main.rs }}
-```
 
-To be able to import the `I256` struct, add the following to your `Cargo.toml` file:
-
-```toml
-openvm-bigint-guest = { git = "https://github.com/openvm-org/openvm.git" }
-```
 
 ## External Linking
 

book/src/custom-extensions/ecc.md

@@ -2,7 +2,7 @@
 
 The OpenVM Elliptic Curve Cryptography Extension provides support for elliptic curve operations through the `openvm-ecc-guest` crate.
 
-The secp256k1 and secp256r1 curves are supported out of the box, and developers can enable arbitrary Weierstrass curves by configuring this extension with the modulus for the coordinate field and the coefficients in the curve equation.
+Developers can enable arbitrary Weierstrass curves by configuring this extension with the modulus for the coordinate field and the coefficients in the curve equation. Preset configurations for the secp256k1 and secp256r1 curves are provided through the [K256](../guest-libs/k256.md) and [P256](../guest-libs/p256.md) guest libraries.
 
 ## Available traits and methods
 
@@ -43,10 +43,10 @@ sw_declare! {
 Each declared curve must specify the `mod_type` (implementing `IntMod`) and a constant `b` for the Weierstrass curve equation \\(y^2 = x^3 + ax + b\\). `a` is optional and defaults to 0 for short Weierstrass curves.
 This creates `Bls12_381G1Affine` and `P256Affine` structs which implement the `Group` and `WeierstrassPoint` traits. The underlying memory layout of the structs uses the memory layout of the `Bls12_381Fp` and `P256Coord` structs, respectively.
 
-2. **Init**: Called once, the `init!` macro produces a call to `sw_init!` that enumerates these curves and allows the compiler to produce optimized instructions:
+2. **Init**: Called once, the [`openvm::init!` macro](./overview.md#automating-the-init-step) produces a call to `sw_init!` that enumerates these curves and allows the compiler to produce optimized instructions:
 
 ```rust
-init!();
+openvm::init!();
 /* This expands to
 sw_init! {
     Bls12_381G1Affine, P256Affine,
@@ -67,51 +67,3 @@ For the basic operations provided by the `WeierstrassPoint` trait, the scalar fi
 
 The ECC extension supports ECDSA signature verification on any elliptic curve, and pre-defined implementations are provided for the secp256k1 and secp256r1 curves.
 To verify an ECDSA signature, first call the `VerifyingKey::recover_from_prehash_noverify` associated function to recover the verifying key, then call the `VerifyingKey::verify_prehashed` method on the recovered verifying key.
-
-## Example program
-
-See a working example [here](https://github.com/openvm-org/openvm/blob/main/examples/ecc/src/main.rs).
-
-To use the ECC extension, add the following dependencies to `Cargo.toml`:
-
-```toml
-openvm-algebra-guest = { git = "https://github.com/openvm-org/openvm.git" }
-openvm-ecc-guest = { git = "https://github.com/openvm-org/openvm.git", features = ["k256"] }
-```
-
-One can define their own ECC structs but we will use the Secp256k1 struct from `openvm-ecc-guest` and thus the `k256` feature should be enabled.
-
-```rust,no_run,noplayground
-{{ #include ../../../examples/ecc/src/main.rs:imports }}
-{{ #include ../../../examples/ecc/src/main.rs:init }}
-```
-
-`moduli_init!` is called for both the coordinate and scalar field because they were declared in the `k256` module, although we will not be using the scalar field below.
-
-With the above we can start doing elliptic curve operations like adding points:
-
-```rust,no_run,noplayground
-{{ #include ../../../examples/ecc/src/main.rs:main }}
-```
-
-### Config parameters
-
-For the guest program to build successfully, all used moduli and curves must be declared in the `.toml` config file in the following format:
-
-```toml
-[app_vm_config.modular]
-supported_moduli = ["115792089237316195423570985008687907853269984665640564039457584007908834671663", "115792089237316195423570985008687907852837564279074904382605163141518161494337"]
-
-[[app_vm_config.ecc.supported_curves]]
-struct_name = "Secp256k1Point"
-modulus = "115792089237316195423570985008687907853269984665640564039457584007908834671663"
-scalar = "115792089237316195423570985008687907852837564279074904382605163141518161494337"
-a = "0"
-b = "7"
-```
-
-The `supported_moduli` parameter is a list of moduli that the guest program will use. As mentioned in the [algebra extension](./algebra.md) chapter, the order of moduli in `[app_vm_config.modular]` must match the order in the `moduli_init!` macro.
-
-The `ecc.supported_curves` parameter is a list of supported curves that the guest program will use. They must be provided in decimal format in the `.toml` file. For multiple curves create multiple `[[app_vm_config.ecc.supported_curves]]` sections. The order of curves in `[[app_vm_config.ecc.supported_curves]]` must match the order in the `sw_init!` macro.
-Also, the `struct_name` field must be the name of the elliptic curve struct created by `sw_declare!`.
-In this example, the `Secp256k1Point` struct is created in `openvm_ecc_guest::k256`.

book/src/custom-extensions/keccak.md

@@ -1,34 +1,6 @@
 # Keccak256
 
-The OpenVM Keccak256 extension provides tools for using the Keccak-256 hash function.
-The functional part is provided by the `openvm-keccak-guest` crate, which is a guest library that can be used in any OpenVM program.
-
-## Functions for guest code
-
-The OpenVM Keccak256 Guest extension provides two functions for use in your guest code:
-
-- `keccak256(input: &[u8]) -> [u8; 32]`: Computes the Keccak-256 hash of the input data and returns it as an array of 32 bytes.
-- `set_keccak256(input: &[u8], output: &mut [u8; 32])`: Sets the output to the Keccak-256 hash of the input data into the provided output buffer.
-
-See the full example [here](https://github.com/openvm-org/openvm/blob/main/examples/keccak/src/main.rs).
-
-### Example
-
-```rust,no_run,noplayground
-{{ #include ../../../examples/keccak/src/main.rs:imports }}
-{{ #include ../../../examples/keccak/src/main.rs:main }}
-```
-
-To be able to import the `keccak256` function, add the following to your `Cargo.toml` file:
-
-```toml
-openvm-keccak256-guest = { git = "https://github.com/openvm-org/openvm.git" }
-hex = { version = "0.4.3", default-features = false, features = ["alloc"] }
-```
-
-## External Linking
-
-The keccak guest extension also provides another way to use the keccak-256 intrinsic implementation. It provides a function that is meant to be linked to other external libraries. The external libraries can use this function as a hook for the keccak-256 intrinsic. This is enabled only when the target is `zkvm`.
+The Keccak256 extension guest provides a function that is meant to be linked to other external libraries. The external libraries can use this function as a hook for the keccak-256 intrinsic. This is enabled only when the target is `zkvm`.
 
 - `native_keccak256(input: *const u8, len: usize, output: *mut u8)`: This function has `C` ABI. It takes in a pointer to the input, the length of the input, and a pointer to the output buffer.
 

book/src/custom-extensions/overview.md

@@ -1,28 +1,32 @@
-# Using Existing Extensions
+# Acceleration Using Pre-Built Extensions
 
-You can seamlessly integrate certain performance-optimized extensions maintained by the OpenVM team to enhance your arithmetic operations and cryptographic computations.
+OpenVM ships with a set of pre-built extensions maintained by the OpenVM team. Below, we highlight six of these extensions designed to accelerate common arithmetic and cryptographic operations that are notoriously expensive to execute. Some of these extensions have corresponding guest libraries which provide convenient, high-level interfaces for your guest program to interact with the extension.
 
-In this chapter, we will explain how to use the following existing extensions:
-
-- [`openvm-keccak-guest`](./keccak.md) - Keccak256 hash function.
-- [`openvm-sha2-guest`](./sha2.md) - SHA-2 hash functions.
-- [`openvm-bigint-guest`](./bigint.md) - Big integer arithmetic for 256-bit signed and unsigned integers.
+- [`openvm-keccak-guest`](./keccak.md) - Keccak256 hash function. See the [Keccak256 guest library](../guest-libs/keccak256.md) for usage details.
+- [`openvm-sha2-guest`](./sha2.md) - SHA-2 family of hash functions. See the [SHA-2 guest library](../guest-libs/sha2.md) for usage details.
+- [`openvm-bigint-guest`](./bigint.md) - Big integer arithmetic for 256-bit signed and unsigned integers. See the [ruint guest library](../guest-libs/ruint.md) for using accelerated 256-bit integer ops in rust.
 - [`openvm-algebra-guest`](./algebra.md) - Modular arithmetic and complex field extensions.
-- [`openvm-ecc-guest`](./ecc.md) - Elliptic curve cryptography.
-- [`openvm-pairing-guest`](./pairing.md) - Elliptic curve optimal Ate pairings.
+- [`openvm-ecc-guest`](./ecc.md) - Elliptic curve cryptography. See the [k256](../guest-libs/k256.md) and [p256](../guest-libs/p256.md) guest libraries for using this extension over the respective curves.
+- [`openvm-pairing-guest`](./pairing.md) - Elliptic curve optimal Ate pairings. See the [pairing guest library](../guest-libs/pairing.md) for usage details.
 
-Some extensions such as `openvm-keccak-guest`, `openvm-sha2-guest`, and `openvm-bigint-guest` can be enabled without specifying any additional configuration.
+## Optimizing Modular Arithmetic
 
-On the other hand certain arithmetic operations, particularly modular arithmetic, can be optimized significantly when the modulus is known at compile time. This approach requires a framework to inform the compiler about all the moduli and associated arithmetic structures we intend to use. To achieve this, three steps are involved:
+Some of these extensions—specifically `algebra`, `ecc`, and `pairing`—perform modular arithmetic, which can be significantly optimized when the modulus is known at compile time.  Therefore, these extensions provide a framework to inform the compiler about all the moduli and associated arithmetic structures we intend to use. To achieve this, two steps are involved:
 
 1. **Declare**: Introduce a modular arithmetic or related structure, along with its modulus and functionality. This can be done in any library or binary file.
 2. **Init**: Performed exactly once in the final binary. It aggregates all previously declared structures, assigns them stable indices, and sets up linkage so that they can be referenced in generated code.
-3. **Setup**: A one-time runtime procedure for security. This ensures that the compiled code matches the virtual machine’s expectations and that each instruction set is tied to the correct modulus or extension.
 
 These steps ensure both performance and security: performance because the modulus is known at compile time, and security because runtime checks confirm that the correct structures have been initialized.
 
 Our design for the configuration procedure above was inspired by the [EVMMAX proposal](https://github.com/jwasinger/EIPs/blob/evmmax-2/EIPS/eip-6601.md).
 
+### Automating the `init!` step
+
+The `openvm` crate provides an `init!` macro to automate the **init** step:
+1. Call `openvm::init!()` exactly once in the code of the final program binary.
+2. When [compiling the program](../writing-apps/build.md), `cargo openvm build` will read the [configuration file](#configuration) to automatically generate the correct init code and write it to `<INIT_FILE_NAME>`, which defaults to `openvm_init.rs` in the manifest directory.
+3. The `openvm::init!()` macro will include the `openvm_init.rs` file into the final binary to complete the init process. You can call `openvm::init!(INIT_FILE_NAME)` to include init code from a different file if needed.
+
 ## Configuration
 
 To use these extensions, you must populate an `openvm.toml` in your package root directory (where the `Cargo.toml` file is located).
@@ -55,12 +59,14 @@ supported_moduli = ["<modulus_1>", "<modulus_2>", ...]
 supported_curves = ["Bls12_381", "Bn254"]
 
 [[app_vm_config.ecc.supported_curves]]
+struct_name = "<curve_name_1>"
 modulus = "<modulus_1>"
 scalar = "<scalar_1>"
 a = "<a_1>"
 b = "<b_1>"
 
 [[app_vm_config.ecc.supported_curves]]
+struct_name = "<curve_name_2>"
 modulus = "<modulus_2>"
 scalar = "<scalar_2>"
 a = "<a_2>"