diff --git a/.config/dictionaries/project.dic b/.config/dictionaries/project.dic index 9ffd8cfc4..4a2b0d577 100644 --- a/.config/dictionaries/project.dic +++ b/.config/dictionaries/project.dic @@ -26,6 +26,7 @@ cardano catid cbor cdylib +CFLAGS chaincode childs chksum @@ -100,10 +101,7 @@ jsonschema justfile keepalives keypair -Keypair -KEYPAIR keypairs -Keypairs libp libsqlite libtest @@ -125,7 +123,6 @@ mkcron mkdelay mkdirat multiaddr -Multiaddr multiaddrs multicodec multihash @@ -164,7 +161,6 @@ pubkey pwrite qmark quic -QUIC rafal rankdir ranksep @@ -200,6 +196,7 @@ subpaths subsec symlinkat syscall +sysroot tailscale Tdiff testcov @@ -243,4 +240,4 @@ xprivate xprv xpub zilla -zillable +zillable \ No newline at end of file diff --git a/README.md b/README.md index 35a9fe18b..8de62d934 100644 --- a/README.md +++ b/README.md @@ -5,83 +5,128 @@ Hermes is a high-performance WebAssembly (WASM) application engine that provides secure, sandboxed execution of modular applications. -## Quick Start +* [Hermes](#hermes) + * [Prerequisites](#prerequisites) + * [GitHub Token Setup](#github-token-setup) + * [Rust Toolchain Setup](#rust-toolchain-setup) + * [WASI SDK Setup (Local Builds Only)](#wasi-sdk-setup-local-builds-only) + * [Quick Start](#quick-start) + * [Build System](#build-system) + * [Key Commands](#key-commands) + * [Contributing](#contributing) + * [License](#license) + +## Prerequisites + +Before building Hermes, ensure you have the following installed: + +| Tool | Purpose | Installation | +| ------------ | ------------------------------ | ----------------------------------------------------------- | +| **Rust** | Build toolchain | [rustup.rs](https://rustup.rs) | +| **Just** | Command runner | `cargo install just` or `brew install just` | +| **Docker** | Container runtime | [docker.com](https://www.docker.com/get-started) | +| **Earthly** | Containerized builds | [earthly.dev](https://earthly.dev/get-earthly) | +| **LLVM** | WASM target support | `brew install llvm` (macOS) or `apt install llvm` (Linux) | +| **cmake** | Native deps (local builds) | `brew install cmake` (macOS) or `apt install cmake` (Linux) | +| **WASI SDK** | WASM C compiler (local builds) | See [WASI SDK Setup](#wasi-sdk-setup-local-builds-only) | + +### GitHub Token Setup + +A GitHub token is required for accessing private dependencies: + +1. Go to [github.com/settings/tokens](https://github.com/settings/tokens) +2. Generate a new **classic token** with these permissions: + * `public_repo` - Access public repositories + * `read:packages` - Read packages from GitHub Package Registry +3. Create a `.secret` file in the project root (use `.secret.template` as reference): -0. **Github token**: + ```bash + cp .secret.template .secret + # Edit .secret and add your token + ``` - ## GitHub Token Setup +### Rust Toolchain Setup - * Go to [github.com/settings/tokens](https://github.com/settings/tokens) - * Generate new classic token with + permissions - * Add to .secret file. +The project uses Rust 1.89 (specified in `hermes/rust-toolchain.toml`). +For local WASM builds, install the required target: -1. **Install Just command runner**: +```bash +rustup install 1.89 +rustup target add wasm32-wasip2 --toolchain 1.89 +``` - ```bash - cargo install just - # Or: sudo apt install just - # Or: brew install just - ``` +### WASI SDK Setup (Local Builds Only) -2. **See all available commands and documentation**: +For local builds (`just build-run-dev-fastest`), the [WASI SDK](https://github.com/WebAssembly/wasi-sdk) +is required to compile C dependencies to WebAssembly. +Skip this if using containerized builds. - ```bash - just --list - ``` +**Download and install** (version 29+): -3. **Build and run:** + +```bash +WASI_VERSION=29 +WASI_VERSION_FULL=${WASI_VERSION}.0 - ```bash - # First time setup (run once) - just check-local-build - - # Choose your build approach: - just build-run-dev-fastest # 🚀 Daily dev (local, fastest) - just build-run-dev # 🐳 Team consistency (containerized) - just build-run-all # 📦 Production (full assets) - ``` +# macOS (Apple Silicon) +curl -LO https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-arm64-macos.tar.gz +tar xzf wasi-sdk-${WASI_VERSION_FULL}-arm64-macos.tar.gz +sudo mv wasi-sdk-${WASI_VERSION_FULL}-arm64-macos /opt/wasi-sdk -## Build System - -This project uses [Just](https://github.com/casey/just) with two build approaches: +# macOS (Intel) +# Use: wasi-sdk-${WASI_VERSION_FULL}-x86_64-macos.tar.gz -| Approach | When to Use | Requirements | -|----------|-------------|-------------| -| 🚀 **Local** | Daily development, rapid iteration | Local Rust + `wasm32-wasip2` | -| 🐳 **Containerized** | Team consistency, CI/CD, final testing | Docker/Podman + Earthly | +# Linux (x86_64) +# Use: wasi-sdk-${WASI_VERSION_FULL}-x86_64-linux.tar.gz -**All detailed documentation is in the `justfile`.** Run `just --list` to see all commands. +# Linux (arm64) +# Use: wasi-sdk-${WASI_VERSION_FULL}-arm64-linux.tar.gz +``` -## Key Commands +See [WASI SDK Releases](https://github.com/WebAssembly/wasi-sdk/releases) for all available platforms. -### Build Commands +> **Note:** make sure wasi-sdk is installed to `/opt/wasi-sdk`. -**Main workflows:** +**Set environment variables** (add to `~/.zshrc`, `~/.bashrc`, or `~/.profile`): -* `just build-run-dev-fastest` - 🚀 **Daily development** (local builds, fastest) -* `just build-run-dev` - 🐳 **Team consistency** (containerized, matches CI) -* `just build-run-all` - 📦 **Production** (full assets, deployments) +```bash +export WASI_SDK_PATH="/opt/wasi-sdk" +export CC_wasm32_wasip2="${WASI_SDK_PATH}/bin/clang" +``` -**Quick rebuilds:** +Then reload your shell: `source ~/.zshrc` (or restart your terminal) -* `just dev-athena-fast` - WASM only (development) -* `just dev-athena` - WASM only (production) +## Quick Start -**Setup:** +```bash +just check-local-build # Verify your setup (run once) +just build-run-dev # 🐳 Recommended: containerized build (first-time setup) +just build-run-dev-fastest # 🚀 Fast iteration (requires local WASM toolchain) +``` -* `just check-local-build` - Verify local Rust (run once) +> **Tip:** Use `just --list` to see all available commands, `just --show ` for details. -### Other Commands +## Build System -* `just status` - Show build status -* `just clean-hfs` - Clean application state -* `just --list` - See all available commands +This project uses [Just](https://github.com/casey/just) with two build approaches: -For detailed help: `just --show ` +| Approach | When to Use | Requirements | +| ------------------- | -------------------------------------- | ---------------------------- | +| 🐳 **Containerized** | First-time setup, CI/CD, final testing | Docker/Podman + Earthly | +| 🚀 **Local** | Daily development, rapid iteration | Local Rust + `wasm32-wasip2` | -## Development +## Key Commands -For development guidelines, tooling information, and best practices, see DEVELOPMENT.md. +| Command | Description | +| ---------------------------- | ------------------------------------ | +| `just build-run-dev` | 🐳 Containerized build (recommended) | +| `just build-run-dev-fastest` | 🚀 Local build (fast iteration) | +| `just build-run-all` | 📦 Production build (full assets) | +| `just dev-athena-fast` | WASM modules only (dev) | +| `just dev-athena` | WASM modules only (prod) | +| `just check-local-build` | Verify local Rust and WASM toolchain | +| `just status` | Show build status | +| `just clean-hfs` | Clean application state | ## Contributing @@ -90,7 +135,8 @@ Please read our [CONTRIBUTING](CONTRIBUTING.md) for guidelines on how to contrib ## License -Licensed under either of [Apache License, Version 2.0](LICENSE-APACHE) or [MIT license](LICENSE-MIT) at your option. +Licensed under either of [Apache License, Version 2.0](LICENSE-APACHE) or +[MIT license](LICENSE-MIT) at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall diff --git a/hermes/Earthfile b/hermes/Earthfile index 613f5a0c7..432067f8e 100644 --- a/hermes/Earthfile +++ b/hermes/Earthfile @@ -11,6 +11,24 @@ IMPORT ../wasm/stub-module AS stub-module # builder : Set up our target toolchains, and copy our files. builder: DO rust-ci+SETUP + + # Install WASI SDK for compiling C code to wasm32-wasip2 + ARG WASI_SDK_VERSION=25 + ARG WASI_SDK_VERSION_FULL=${WASI_SDK_VERSION}.0 + RUN apt-get update && apt-get install -y wget && \ + ARCH=$(dpkg --print-architecture) && \ + if [ "$ARCH" = "amd64" ]; then WASI_ARCH="x86_64"; else WASI_ARCH="arm64"; fi && \ + wget -q https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + tar xzf wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + mv wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux /opt/wasi-sdk && \ + rm wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + apt-get clean && rm -rf /var/lib/apt/lists/* + + # Configure environment for wasm32-wasip2 C compilation + ENV WASI_SDK_PATH="/opt/wasi-sdk" + ENV CC_wasm32_wasip2="${WASI_SDK_PATH}/bin/clang" + ENV CFLAGS_wasm32_wasip2="--sysroot=${WASI_SDK_PATH}/share/wasi-sysroot" + COPY --dir .cargo .config bin apps schemas . COPY Cargo.toml . COPY clippy.toml deny.toml rustfmt.toml . diff --git a/hermes/apps/athena/Earthfile b/hermes/apps/athena/Earthfile index b63f38ace..61d94510b 100644 --- a/hermes/apps/athena/Earthfile +++ b/hermes/apps/athena/Earthfile @@ -19,6 +19,23 @@ builder: DO rust-ci+SETUP + # Install WASI SDK for compiling C code to wasm32-wasip2 + ARG WASI_SDK_VERSION=25 + ARG WASI_SDK_VERSION_FULL=${WASI_SDK_VERSION}.0 + RUN apt-get update && apt-get install -y wget && \ + ARCH=$(dpkg --print-architecture) && \ + if [ "$ARCH" = "amd64" ]; then WASI_ARCH="x86_64"; else WASI_ARCH="arm64"; fi && \ + wget -q https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + tar xzf wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + mv wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux /opt/wasi-sdk && \ + rm wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + apt-get clean && rm -rf /var/lib/apt/lists/* + + # Configure environment for wasm32-wasip2 C compilation + ENV WASI_SDK_PATH="/opt/wasi-sdk" + ENV CC_wasm32_wasip2="${WASI_SDK_PATH}/bin/clang" + ENV CFLAGS_wasm32_wasip2="--sysroot=${WASI_SDK_PATH}/share/wasi-sysroot" + COPY wasm+wasi-src/wasi $wasi_src_dir # check-shared : runs native unit tests of the shared crate. diff --git a/hermes/apps/athena/modules/doc-sync/README.md b/hermes/apps/athena/modules/doc-sync/README.md index 322604b3e..97ab648ee 100644 --- a/hermes/apps/athena/modules/doc-sync/README.md +++ b/hermes/apps/athena/modules/doc-sync/README.md @@ -26,7 +26,7 @@ All heavy operations are delegated to the host-side implementation for performan The module exposes an HTTP endpoint for testing: ```bash -curl -X POST http://localhost:5000/api/doc-sync/post \ +curl -X POST http://localhost:7878/api/doc-sync/post \ -H "Host: athena.hermes.local" \ -H "Content-Type: text/plain" \ -d "Hello, IPFS!" diff --git a/hermes/apps/athena/modules/doc-sync/src/lib.rs b/hermes/apps/athena/modules/doc-sync/src/lib.rs index 860d0e4ac..ece48f545 100644 --- a/hermes/apps/athena/modules/doc-sync/src/lib.rs +++ b/hermes/apps/athena/modules/doc-sync/src/lib.rs @@ -175,7 +175,7 @@ fn get_document_by_cid(cid: &[u8]) -> anyhow::Result>> { /// /// Example: /// ```bash -/// curl -X POST http://localhost:5000/api/doc-sync/post \ +/// curl -X POST http://localhost:7878/api/doc-sync/post \ /// -H "Host: athena.hermes.local" \ /// -H "Content-Type: text/plain" \ /// -d "Hello, IPFS!" diff --git a/hermes/bin/src/runtime_extensions/hermes/http_gateway/gateway_task.rs b/hermes/bin/src/runtime_extensions/hermes/http_gateway/gateway_task.rs index 72d8eed4f..6315f3709 100644 --- a/hermes/bin/src/runtime_extensions/hermes/http_gateway/gateway_task.rs +++ b/hermes/bin/src/runtime_extensions/hermes/http_gateway/gateway_task.rs @@ -14,8 +14,8 @@ use tracing::{debug, error, info}; use super::routing::router; -/// HTTP Gateway default port -const DEFAULT_GATEWAY_PORT: u16 = 5000; +/// HTTP Gateway default port (`7878` avoids conflicts with macOS `AirPlay` on `5000`) +const DEFAULT_GATEWAY_PORT: u16 = 7878; /// hostname (node name) #[cfg_attr(debug_assertions, derive(Debug))] diff --git a/hermes/bin/src/runtime_extensions/hermes/http_gateway/routing.rs b/hermes/bin/src/runtime_extensions/hermes/http_gateway/routing.rs index 17ddfc7bc..c635de30f 100644 --- a/hermes/bin/src/runtime_extensions/hermes/http_gateway/routing.rs +++ b/hermes/bin/src/runtime_extensions/hermes/http_gateway/routing.rs @@ -142,7 +142,7 @@ pub(crate) fn host_resolver(headers: &HeaderMap) -> anyhow::Result<(ApplicationN .and_then(|h| h.to_str().ok()) .ok_or_else(|| anyhow!("Missing or invalid Host header"))?; - // Strip port if present (e.g., "app.hermes.local:5000" -> "app.hermes.local") + // Strip port if present (e.g., "app.hermes.local:7878" -> "app.hermes.local") let host_without_port = host.split(':').next().unwrap_or(host); // Parse .hermes.local format diff --git a/hermes/bin/tests/integration/tests/serial/staked_ada.rs b/hermes/bin/tests/integration/tests/serial/staked_ada.rs index 49c37b11a..270ec5f6e 100644 --- a/hermes/bin/tests/integration/tests/serial/staked_ada.rs +++ b/hermes/bin/tests/integration/tests/serial/staked_ada.rs @@ -115,7 +115,7 @@ fn staked_ada_requests() -> anyhow::Result<()> { std::thread::sleep(Duration::from_secs(10)); let url_builder = |stake_address: &str| { - format!("http://localhost:5000/api/gateway/v1/cardano/assets/{stake_address}") + format!("http://localhost:7878/api/gateway/v1/cardano/assets/{stake_address}") }; // cspell:disable let url_with_mocked_address = diff --git a/justfile b/justfile index ff7c436e0..80f5f4e2c 100644 --- a/justfile +++ b/justfile @@ -260,7 +260,7 @@ clean-hfs: fi # Run the Athena application -# Test with: curl -H "Host: app.hermes.local" http://localhost:5000/api/gateway/v1/rbac/registration +# Test with: curl -H "Host: app.hermes.local" http://localhost:7878/api/gateway/v1/rbac/registration run-athena: #!/usr/bin/env bash set -euo pipefail @@ -289,7 +289,7 @@ run-athena: echo " Allowed Path Prefixes: $REDIRECT_ALLOWED_PATH_PREFIXES" echo "" echo "🌐 Starting HTTP server..." - echo "💡 Test with: curl -H 'Host: app.hermes.local' http://localhost:5000/api/gateway/v1/rbac/registration" + echo "💡 Test with: curl -H 'Host: app.hermes.local' http://localhost:7878/api/gateway/v1/rbac/registration" echo "🛑 Press Ctrl+C to stop" echo "" diff --git a/wasm/wasi/Earthfile b/wasm/wasi/Earthfile index f68ebba7c..62b1b64cb 100644 --- a/wasm/wasi/Earthfile +++ b/wasm/wasi/Earthfile @@ -15,6 +15,23 @@ builder: DO rust-ci+SETUP + # Install WASI SDK for compiling C code to wasm32-wasip2 + ARG WASI_SDK_VERSION=25 + ARG WASI_SDK_VERSION_FULL=${WASI_SDK_VERSION}.0 + RUN apt-get update && apt-get install -y wget && \ + ARCH=$(dpkg --print-architecture) && \ + if [ "$ARCH" = "amd64" ]; then WASI_ARCH="x86_64"; else WASI_ARCH="arm64"; fi && \ + wget -q https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + tar xzf wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + mv wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux /opt/wasi-sdk && \ + rm wasi-sdk-${WASI_SDK_VERSION_FULL}-${WASI_ARCH}-linux.tar.gz && \ + apt-get clean && rm -rf /var/lib/apt/lists/* + + # Configure environment for wasm32-wasip2 C compilation + ENV WASI_SDK_PATH="/opt/wasi-sdk" + ENV CC_wasm32_wasip2="${WASI_SDK_PATH}/bin/clang" + ENV CFLAGS_wasm32_wasip2="--sysroot=${WASI_SDK_PATH}/share/wasi-sysroot" + COPY ..+wasi-src/wasi $wasi_src_dir # Build the WASI Docs: @@ -103,9 +120,6 @@ BUILD_RUST_COMPONENT: COPY --keep-ts --dir src . COPY Cargo.toml . - # Fix missing header error when compiling zstd while building rbac-registration - ENV CFLAGS_wasm32_wasip2="--sysroot=/usr/aarch64-linux-gnu" - DO rust-ci+CARGO \ --args "build --target wasm32-wasip2 --release" \ --output="wasm32-wasip2/release/$out"