fs

Simple S3-compatible storage server for development and testing.

Features

S3-Compatible Storage Server

A lightweight S3-compatible storage server for development and testing.

Quick Start:

# Install
go install github.com/go-faster/fs/cmd/fs@latest

# Start the server
fs s3

# Or with custom configuration
fs s3 --addr :9000 --root /data/s3

Features:

• Bucket operations (create, delete, list)
• Object operations (put, get, delete, list, copy, tagging, metadata)
• Multipart uploads
• File system-based storage
• Compatible with AWS CLI, MinIO client, and other S3 clients
• Health check endpoint

See COMPATIBILITY.md for the full compatibility statement (what's implemented, what returns NotImplemented, what's planned, and the durability & failure model). Compatibility is measured against the upstream ceph/s3-tests suite and real S3 clients — the machine-generated breakdown is in the S3 conformance report.

Example Usage:

# Using AWS CLI
export AWS_ENDPOINT_URL=http://localhost:8080
aws s3 mb s3://mybucket --endpoint-url $AWS_ENDPOINT_URL
aws s3 cp file.txt s3://mybucket/ --endpoint-url $AWS_ENDPOINT_URL

# Using cURL
curl -X PUT http://localhost:8080/mybucket
curl -X PUT -d "Hello!" http://localhost:8080/mybucket/hello.txt
curl http://localhost:8080/mybucket/hello.txt

Authentication & TLS

The binary authenticates requests with AWS Signature V4 by default. Provide a root credential and (optionally) TLS:

export FS_ROOT_ACCESS_KEY=AKIAEXAMPLE
export FS_ROOT_SECRET_KEY=exampleSecretKey
fs s3 --tls-cert cert.pem --tls-key key.pem

Additional keys, per-bucket grants (read/write/admin), and public-read buckets are configured under auth: in the config file. To run without any authentication (development only), pass --insecure-no-auth.

SigV4 header auth, presigned URLs (≤7-day expiry) and streaming uploads are all verified; TLS certificates hot-reload without dropping connections. As a library, enable it with server.WithAuth(store) / server.WithCORS(cfg) — the bare handler stays anonymous unless you opt in.

Admin API & access-key dashboard

Multiple access-key/secret credentials can be managed at runtime — without a restart — through a separate admin listener that also serves a small web dashboard. Config-defined keys stay read-only; keys created through the admin API are persisted (<root>/.access-keys.json, mode 0600) and survive restarts and SIGHUP reloads.

admin:
  enabled: true
  addr: "localhost:8090"   # keep bound to localhost or behind a proxy
  token: "change-me"       # or set FS_ADMIN_TOKEN

The dashboard (open http://localhost:8090/, paste the token) lists credentials and their grants, creates keys (generating the access key and secret, shown once), and deletes runtime keys. The same operations are available as a JSON API under /api/v1 (bearer-token protected), generated from _oas/admin.yml with ogen; the dashboard is a TypeScript/React SPA whose typed client is generated from the same spec with Orval:

# List credentials
curl -H "Authorization: Bearer $FS_ADMIN_TOKEN" localhost:8090/api/v1/access-keys

# Create a credential scoped to buckets matching "uploads-*"
curl -H "Authorization: Bearer $FS_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"grants":[{"bucket":"uploads-*","permission":"write"}]}' \
  localhost:8090/api/v1/access-keys

Run as a systemd service

Generate a unit for fs s3 — a per-user service by default, or a hardened system service with --user=false:

# Install and enable a per-user service
fs systemd --install --config ~/fs.yaml
systemctl --user daemon-reload
systemctl --user enable --now fs
loginctl enable-linger "$USER"   # keep it running after logout

# Or emit a system unit
fs systemd --user=false --config /etc/fs/config.yaml | sudo tee /etc/systemd/system/fs.service

The unit wires ExecReload to SIGHUP, so systemctl --user reload fs performs the hot credential/TLS reload.

Operations

• Durability — storage.fsync (none / file / file+dir, default file) controls fsync aggressiveness; writes are always crash-atomic (no torn object). A background scrubber (integrity.scrub_interval) detects bit-rot and can quarantine corrupt objects; integrity.verify_on_read checks each object before serving.
• Health & readiness — /health (liveness: the process is up) and /ready (readiness: storage is reachable, 503 otherwise). Prometheus /metrics and pprof are served on a separate listener (default localhost:9464, METRICS_ADDR to change).
• Hot reload — send SIGHUP to reload credentials and the TLS certificate from disk without a restart.

Installation

go install github.com/go-faster/fs/cmd/fs@latest

Or build from source:

git clone https://github.com/go-faster/fs
cd fs
go build -o bin/fs ./cmd/fs

Usage

Quick Start

# Start S3 server with defaults
fs s3

# Show help
fs s3 --help

Configuration

The server supports both YAML configuration files and command-line flags:

# Using YAML configuration
fs s3 --config config.yaml

# Using command-line flags
fs s3 --addr :9000 --root /var/lib/s3data

# Mix both (flags override config file)
fs s3 --config config.yaml --addr :9000

# Generate example configuration
fs s3 --generate-config > my-config.yaml

Run fs s3 --generate-config to produce a fully commented configuration template, and fs s3 --help for the list of flags.

Example Configuration

server:
  addr: ":8080"
  read_timeout: 30s
  write_timeout: 30s
  idle_timeout: 120s
  health_path: "/health"

storage:
  root: ".s3data"
  type: "filesystem"

observability:
  service_name: "go-faster/fs"
  enable_request_logging: true
  enable_metrics: true
  enable_tracing: true

Use as a library

The S3 server is embeddable. Install the module and pick a storage backend — storagefs for the filesystem, storagemem for in-memory, or your own implementation of the fs.Storage interface:

go get github.com/go-faster/fs

The library core pulls in no observability stack — wrap the handler yourself (e.g. with otelhttp) via server.NewHandler or Config.WrapHandler.

Custom backends can verify themselves against the storage contract with the storagetest conformance suite:

func TestStorage(t *testing.T) {
	storagetest.Run(t, func(t testing.TB) fs.Storage {
		return mybackend.New(t.TempDir())
	})
}

Mount the handler into your own server

Use server.NewHandler when you already run an http.Server or mux and just want to expose the S3 API (optionally under a path prefix):

package main

import (
	"net/http"

	"github.com/go-faster/fs/server"
	"github.com/go-faster/fs/storagefs"
)

func main() {
	store, err := storagefs.New("/data")
	if err != nil {
		panic(err)
	}

	mux := http.NewServeMux()
	mux.Handle("/s3/", http.StripPrefix("/s3", server.NewHandler(store)))

	http.ListenAndServe(":8080", mux)
}

Run the turnkey server

Use server.New for a managed server with a health endpoint, request timeouts, optional bucket pre-creation and graceful shutdown driven by a context:

package main

import (
	"context"
	"os/signal"
	"syscall"

	"github.com/go-faster/fs/server"
	"github.com/go-faster/fs/storagemem"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	srv, err := server.New(server.Config{
		Storage: storagemem.New(),
		Addr:    ":9000",
		Buckets: []string{"uploads"}, // pre-created if absent
	})
	if err != nil {
		panic(err)
	}

	// Serves until ctx is canceled, then drains in-flight requests.
	if err := srv.ListenAndServe(ctx); err != nil {
		panic(err)
	}
}

server.Config

Field	Default	Description
Storage	— (required)	Backend serving S3 operations (fs.Storage).
Addr	:8080	TCP address to listen on.
ReadTimeout / WriteTimeout / IdleTimeout	30s / 30s / 120s	Underlying http.Server timeouts.
HealthPath	/health	Plaintext liveness endpoint; "-" disables it.
ReadyPath / Ready	/ready / —	Readiness endpoint and its probe; a non-nil probe error returns 503.
Buckets	—	Buckets created (idempotently) before serving.
Auth / CORS / TLS	—	SigV4 auth store, per-bucket CORS, and hot-reloadable TLS.
WrapHandler	—	Wrap the handler with middleware/observability (e.g. otelhttp.NewHandler).

See the server package reference for the full API and runnable examples.

Development

# Run tests
go test ./...

# Build
go build ./cmd/fs

# Run with coverage
make coverage

License

Apache 2.0