Skip to content

docs: ARM64 architecture support in orchestrator README #2254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tomassrnka wants to merge 5 commits into
base: main
Choose a base branch
from
Open

docs: ARM64 architecture support in orchestrator README #2254

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4f82a51
docs: add ARM64 architecture support section to orchestrator README
Mar 29, 2026
b5a86e6
fix: correct Firecracker version in ARM64 docs examples
Mar 29, 2026
483f28e
fix: update FC version, busybox docs, and BUILD_ARCH reference
Mar 30, 2026
dbcb02d
fix: update kernel version to vmlinux-6.1.158 in docs
Mar 31, 2026
3592952
docs: add BUILD_ARCH=arm64 instructions for ARM64 builds
tomassrnka Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 73 additions & 2 deletions packages/orchestrator/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ Flags:
- `-template <id>` - Template ID (default: `local-template`)
- `-storage <path>` - Local path or `gs://bucket` (enables local mode with auto-download of kernel/FC)
- `-sandbox-dir <path>` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)
- `-kernel <version>` - Kernel version (default: `vmlinux-6.1.102`)
- `-firecracker <version>` - Firecracker version (default: `v1.12.1_a41d3fb`)
- `-kernel <version>` - Kernel version (default: `vmlinux-6.1.158`)
- `-firecracker <version>` - Firecracker version (default: `v1.12.1_210cbac`)
- `-vcpu <n>` - vCPUs (default: `1`)
- `-memory <mb>` - Memory in MB (default: `512`)
- `-disk <mb>` - Disk in MB (default: `1000`)
Expand Down Expand Up @@ -219,10 +219,81 @@ Flags:

---

## Architecture (ARM64) Support

The orchestrator supports both `amd64` (x86_64) and `arm64` (aarch64) architectures. Architecture is detected automatically via `runtime.GOARCH` at compile time.

### Architecture naming convention

This project uses **Go/Docker/Debian naming** (`amd64`/`arm64`) for architecture directories in binary paths and GCS buckets:

| Convention | x86_64 name | ARM64 name | Used by |
|------------|-------------|------------|---------|
| **Go/Docker/Debian** | `amd64` | `arm64` | This repo, Docker, `dpkg --print-architecture` |
| Linux/GNU | `x86_64` | `aarch64` | `uname -m`, kernel Makefiles |

Binary paths follow the `{version}/{arch}/` layout:

```
# Firecracker (GCS bucket or FIRECRACKER_VERSIONS_DIR)
fc-versions/v1.12.1_210cbac/amd64/firecracker
fc-versions/v1.12.1_210cbac/arm64/firecracker

# Kernels (GCS bucket or HOST_KERNELS_DIR)
kernels/vmlinux-6.1.158/amd64/vmlinux.bin
kernels/vmlinux-6.1.158/arm64/vmlinux.bin
```

> **Note:** The [fc-kernels](https://github.com/e2b-dev/fc-kernels) repo currently uses `x86_64` instead of `amd64` for its directory names. This will be aligned in a follow-up change.

### ARM64-specific behavior

- **SMT** is disabled (ARM processors don't support simultaneous multi-threading)
Copy link
Copy Markdown

@claude claude bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code doesn't actually disable SMT — fc/client.go:329 sets smt := true unconditionally with no ARM64 branch. Either the behavior isn't implemented yet or this claim is incorrect.

Copy link
Copy Markdown
Member Author

@tomassrnka tomassrnka Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applicable — this is a docs-only PR, split from the arm64-support branch. The code implementing these features (TARGET_ARCH, SMT, fetch-busybox, OCI platform) is in separate PRs (#2258, #2259, #2260, #2257). Bots are comparing against this branch alone; the docs will be accurate once all PRs merge.

- **CPU detection** uses fallback values since `gopsutil` doesn't populate Family/Model on ARM64
Copy link
Copy Markdown

@claude claude bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no fallback values in the current code. machineinfo/main.go:25-26 returns an error when Family or Model is empty — it does not fall back to any default values. This will cause the orchestrator to fail on ARM64 with gopsutil if this behavior isn't implemented yet.

Copy link
Copy Markdown
Member Author

@tomassrnka tomassrnka Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applicable — this is a docs-only PR, split from the arm64-support branch. The code implementing these features (TARGET_ARCH, SMT, fetch-busybox, OCI platform) is in separate PRs (#2258, #2259, #2260, #2257). Bots are comparing against this branch alone; the docs will be accurate once all PRs merge.

- **OCI platform** is set to the target architecture instead of hardcoded `amd64`
Copy link
Copy Markdown

@claude claude bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oci/oci.go:59-62 still hardcodes Architecture: "amd64" in DefaultPlatform, and both GetPublicImage and GetImage use this without any target-arch override. The documented behavior doesn't match the code.

Copy link
Copy Markdown
Member Author

@tomassrnka tomassrnka Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applicable — this is a docs-only PR, split from the arm64-support branch. The code implementing these features (TARGET_ARCH, SMT, fetch-busybox, OCI platform) is in separate PRs (#2258, #2259, #2260, #2257). Bots are comparing against this branch alone; the docs will be accurate once all PRs merge.

- **Busybox binaries** are committed for both architectures and selected automatically via Go build tags

### Cross-architecture deployment

`TARGET_ARCH` is a **runtime** environment variable that overrides the architecture used for path resolution and OCI image pulls. When unset, defaults to the host architecture (`runtime.GOARCH`).
Copy link
Copy Markdown

@claude claude bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TARGET_ARCH is not read anywhere in the Go source code — no os.Getenv("TARGET_ARCH") call exists. The env var section and the environment variables table entry (line 282) document behavior that isn't implemented yet, which could mislead users.

Copy link
Copy Markdown
Member Author

@tomassrnka tomassrnka Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applicable — this is a docs-only PR, split from the arm64-support branch. The code implementing these features (TARGET_ARCH, SMT, fetch-busybox, OCI platform) is in separate PRs (#2258, #2259, #2260, #2257). Bots are comparing against this branch alone; the docs will be accurate once all PRs merge.


```bash
# Run orchestrator targeting amd64 paths from an arm64 host
TARGET_ARCH=amd64 ./bin/orchestrator
Comment on lines +258 to +262
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove unsupported TARGET_ARCH override instructions

The new section documents TARGET_ARCH as a runtime override for binary path resolution and OCI pulls, but the implementation does not read this env var (pkg/cfg/model.go has no TARGET_ARCH field), still builds Firecracker/kernel paths without an architecture segment (pkg/sandbox/fc/config.go), and keeps OCI pulls pinned to amd64 via DefaultPlatform (pkg/template/build/core/oci/oci.go). Users following this guidance on mixed-arch hosts will expect overrides to work, but runtime behavior will not match the docs and can lead to missing binaries or wrong image architecture pulls.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

@tomassrnka tomassrnka Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applicable — this is a docs-only PR, split from the arm64-support branch. The code implementing these features (TARGET_ARCH, SMT, fetch-busybox, OCI platform) is in separate PRs (#2258, #2259, #2260, #2257). Bots are comparing against this branch alone; the docs will be accurate once all PRs merge.


# Or in .env file (read at runtime)
echo "TARGET_ARCH=amd64" >> .env.local
```

`TARGET_ARCH` affects:
- Firecracker and kernel binary path resolution (`{version}/{arch}/...`)
- OCI image platform for container pulls

It does **not** affect:
- Makefile compilation (see `BUILD_ARCH` below)
- Hardware-dependent runtime behavior (SMT detection, CPU info) which always uses the actual host architecture

### Building for ARM64

`BUILD_ARCH` defaults to `amd64`. To build ARM64 binaries:

```bash
# Single command
BUILD_ARCH=arm64 make build-local

# Or set in .env.local for persistent override
echo "BUILD_ARCH=arm64" >> .env.local
```

This applies to all services: orchestrator, envd, client-proxy.

---

## Environment Variables

Automatically set in local mode. Set before running to override:

- `TARGET_ARCH` - Target architecture override (`amd64` or `arm64`; default: host architecture)
- `HOST_ENVD_PATH` - Envd binary path (default: `../envd/bin/envd`)
- `HOST_KERNELS_DIR` - Kernel versions dir (local: `{storage}/kernels`, prod: `/fc-kernels`)
- `FIRECRACKER_VERSIONS_DIR` - Firecracker versions dir (local: `{storage}/fc-versions`, prod: `/fc-versions`)
Expand Down
Loading