AGENTS.md

AGENTS.md

This file provides guidance to coding agents collaborating on this repository.

Mission

Hann is an approximate nearest neighbor search library for Go. It provides a set of index data structures (HNSW, PQIVF, and RPT) behind one interface, with distance computation written in C and vectorized with AVX instructions. Priorities, in order:

1. Correctness of index operations: insertion, deletion, update, and search must keep the index consistent.
2. Search quality and speed, measured by recall and query latency on the example datasets.
3. Clean separation between the shared interface and helpers (core/) and the index implementations (hnsw/, pqivf/, and rpt/).
4. Safety of the cgo boundary: no out-of-bounds reads, and no pointers into Go memory that outlive the call.

Core Rules

• Use English for code, comments, docs, and tests.
• Prefer small, focused changes over large refactoring.
• Add comments only when they clarify non-obvious behavior.
• Do not add features, error handling, or abstractions beyond what is needed for the current task.
• Keep external dependencies minimal: do not add new go.mod entries without prior discussion.

Backward Compatibility

Hann is a public Go module that other programs import. The following must stay backward-compatible:

• The core.Index interface. Adding a method breaks every implementation outside this repository, so a new capability belongs on the concrete index types, or on a separate optional interface that callers can assert, like core.BulkIndex and core.Trainer.
• Exported types and constructor signatures (hnsw.New, pqivf.New, and rpt.New, each returning (*Index, error)). A new tuning parameter is a new functional option, never a change to an existing signature.
• The shapes of core.Neighbor and core.IndexStats: fields may be added, not removed or renamed.
• The gob encoding written by Save. An index file written by an older version must still load. The serializedIndex, serializedPQIVF, and rptSerialized structs are the on-disk format, so fields may be added with sensible zero values, but they may not be removed, renamed, or reordered in meaning.
• The names of the built-in metrics in the core registry and the names reported by IndexStats.Distance.
• The environment variables HANN_SEED and HANN_BENCH_NTRD, along with the values they accept. HANN_LOG is accepted and ignored, because the library no longer logs.
• The minimum Go version declared in go.mod. Raising it drops users, so it is a deliberate decision, not a side effect of using a newer standard library function.

Writing Style

• Use Oxford commas in inline lists: "a, b, and c" not "a, b, c".
• Do not use em dashes, in documentation or in code comments. Restructure the sentence, or use a colon or semicolon instead.
• Avoid colorful adjectives and adverbs. Write "rate limiter" not "smart rate limiter".
• Prefer noun phrases for checklist items over imperative verbs. Write "rate limit enforcement" not "enforce rate limits".
• Headings in Markdown files must be in title case: "Build from Source" not "Build from source". Minor words stay lowercase unless they are the first word: the articles (a, an, the), the coordinating conjunctions (and, but, or, nor, so, yet, for), and the short prepositions (in, on, at, to, by, of, up, as, from, with, into, over). The prepositions are named because "from" has to be lowercase for "Build from Source" to be correct.
• Do not bold the lead-in of a list item. Write "Unit tests: ..." not "Unit tests: ...".
• Use sentence case for the lead-in of a list item. Write "Seed selection: ..." not "Seed Selection: ...". Proper nouns keep their capitals.
• Capitalize only the first part of a hyphenated compound: "Nearest-neighbor Search" in a heading, "Nearest-neighbor" at the start of a sentence, and "nearest-neighbor search" elsewhere. Never write "Nearest-Neighbor".
• Start each sentence with a capital letter, capitalize proper nouns (Go, AVX, SIMD, HNSW, PQIVF, RPT), and leave common nouns lowercase in the middle of a sentence.
• Write correct and complete sentences.
• Avoid made-up words.
• Do not use a colon in place of a verb. Three uses are fine: joining two clauses inside a complete sentence (the replacement the em-dash rule above calls for), introducing the gloss of a list item, and introducing an enumeration, whether as a list or inline ("Targets: make test, make lint, ..."). What a colon must not do is turn a sentence into a label and a definition: write "Splits a vector into subspaces, then quantizes each one" rather than "Product quantization: splits a vector into subspaces". That shape belongs to a list item, and carrying it into prose (a doc comment summary, a paragraph) leaves a fragment where a sentence was required.
• Use participial phrases and abbreviations scarcely.

Repository Layout

• core/: the shared interface and helpers. index.go declares Index, the optional BulkIndex and Trainer interfaces, Neighbor, and IndexStats; distance.go wraps the C distance functions; metric.go declares Metric and the metric registry; vector_ops.go holds normalization, single and batched; cpu_check.go detects AVX and AVX2 on x86, selects NEON on arm64, and tells the C side which implementation to install; utils.go reads HANN_SEED.
• core/*.c and core/*.h: the C implementations. simd_distance.c holds Euclidean, squared Euclidean, Manhattan, and cosine distance; simd_ops.c holds normalization and the hann_cpu_init entry point. Each function has a fallback variant, AVX and AVX2 variants on x86_64, and a NEON variant on arm64, selected once through a function pointer. The vector bodies are written once in simd_kernels.inc.h against a macro vocabulary, simd_kernels_avx.inc.h, simd_kernels_avx2.inc.h, and simd_kernels_neon.inc.h instantiate them per ISA, and simd_isa.h declares the target attribute macros and the shared reduction helper.
• hnsw/index.go: the HNSW graph index, its layered neighbor lists, and its gob codec.
• pqivf/index.go: the PQIVF index, coarse clustering, product quantization, and Train.
• rpt/index.go: the RPT index and its random projection tree.
• example/: dataset loading (load_data.go), shared helpers and recall computation (utils.go), and the index runners (run_datasets.go).
• example/cmd/: one main per example and per benchmark, run with go run. This directory is excluded from the test and lint targets.
• example/data/: the dataset download script and notes on the datasets.
• .github/workflows/: CI workflows for tests and lints.
• Makefile: all developer tasks (format, test, lint, examples, benchmarks, and datasets).

Architecture

Layers

Hann is organized into three layers that should not have upward dependencies:

1. core/: the interface, distance functions, and vector operations; no knowledge of any index.
2. hnsw/, pqivf/, and rpt/: index implementations; each depends on core/ and on nothing else in the repository, and the three never import each other.
3. example/ and example/cmd/: programs that exercise the indexes; nothing imports them.

Boundaries Worth Keeping

• An index is reached through core.Index. Code that is written against a concrete type gives up the ability to swap indexes, so keep example and benchmark code on the interface where the operation is part of it. Capabilities outside the interface are reached through the optional interfaces: core.Trainer for training, and core.BulkIndex (or the core.BulkAdd, core.BulkDelete, and core.BulkUpdate helpers) for batches.
• A metric travels as one core.Metric value that bundles the name, the distance function, and the normalization requirement. The name is what Stats reports and what the gob codec stores, and loading resolves it through the registry, so a custom metric must be registered with core.RegisterMetric before Load. Never switch behavior on a metric's name; use Metric.Normalizes.
• Each index owns a mutex and is safe for concurrent use through its exported methods. The unexported helpers assume the lock is already held. Keep that split: a helper must not take the lock itself, and an exported method must not call another exported method of the same index while holding it.
• Random behavior goes through the package-level seededRand, guarded by seededRandMu, so a run with HANN_SEED set is reproducible. Do not call the global math/rand functions, and do not create a new generator per operation.
• The library does not log. Failures are reported through returned errors, and conditions the caller cannot see otherwise, such as a search falling back to a brute-force scan, are counted in IndexStats. Do not add log lines; extend the stats instead.

The cgo Boundary

core/distance.go and core/vector_ops.go pass &slice[0] into C together with the length. The rules that keep this sound:

• Check for an empty slice before taking the address of its first element, and check that both operands have the same length before the call. The C side trusts the length it is given.
• Do not retain a Go pointer on the C side. Every call reads or writes the vector and returns.
• A new distance function needs a scalar fallback, a kernel body in simd_kernels.inc.h whose instantiations provide the AVX, AVX2, and NEON variants, an entry in the function pointer table in init_distance_functions, a declaration in the header, and a core.Metric value pre-registered in core/metric.go. The same applies to its batch variant, which computes the distances from one query to a flat buffer of candidate vectors and backs Metric.DistanceBatch and Metric.RankBatch; the batch loop lives once in simd_kernels.inc.h and calls the per-pair kernel of the same instantiation.
• The AVX variants carry per-function target attributes behind the HANN_TARGET_AVX macro and are selected at runtime by hann_cpu_init. A machine without AVX must still build and run through the fallback path, which is why no -m ISA flag may appear in the cgo CFLAGS. The NEON variants compile behind the architecture guard alone, because every arm64 CPU has NEON, and hann_cpu_init installs them unconditionally on arm64.
• NormalizeBatch fans out over a worker pool, so each worker must own its own vector. Do not share a slice between tasks.

Persistence

Save and Load are gob over an io.Writer and an io.Reader. Each index has a serialized form (a plain struct of exported fields) and a GobEncode/GobDecode pair that converts between it and the live index, which is what lets pointer-linked structures such as the HNSW graph and the RPT tree round-trip. Types are registered in each package's init. A change to a serialized struct is a change to the on-disk format, so read the backward compatibility section above before making one.

Go Conventions

• Go version: the minimum is declared in go.mod, and CI runs the test suite against every release from that version onward.
• A C compiler is needed, because core/ uses cgo. CGO_ENABLED=0 does not produce a working build.
• Formatting is enforced by gofmt (via make format). Run it before committing.
• Naming follows Go standard conventions: PascalCase for exported identifiers, camelCase for unexported identifiers and local variables, and SCREAMING_SNAKE_CASE for top-level constants where idiomatic.
• Errors are returned, never logged and swallowed. Wrap with context using fmt.Errorf("…: %w", err) so callers can use errors.Is/errors.As.
• The index packages and core must not log or print; diagnostic state belongs in returned errors and IndexStats. The example programs use the standard library log package.
• Every exported identifier carries a doc comment that starts with its name.

Required Validation

Run the relevant targets for any change:

Target	Command	What It Runs
Format	make format	go fmt ./...
Unit tests	make test	go test with coverage and the race detector
Lint	make lint	golangci-lint run ./...
Coverage report	make showcov	Displays per-function coverage after running the tests
Examples	make run-examples	Runs the examples that use the small datasets
Large examples	make run-examples-large	Runs the examples that use the large datasets
Benchmarks	make run-benches	Runs the local benchmarks
Go benchmarks	make bench	Runs the Go benchmarks for the kernels and the indexes
Datasets	make download-data	Downloads the datasets the examples use
Large datasets	make download-data-large	Downloads the large datasets
Git hooks	make setup-hooks	Installs the pre-commit and pre-push hooks

The examples and the benchmarks need the datasets, so run make download-data first. The large variants need a machine with a lot of memory (32 GB or more).

First Contribution Flow

1. Read core/index.go to see the contract, then the index package the change touches.
2. Add or update _test.go files in the changed package to describe the new behavior.
3. Run make test and watch the new test fail.
4. Implement the smallest change that makes it pass.
5. Run make test and make lint again, then refactor with the tests green.
6. If the change touches search, distance computation, or serialization, also run make run-examples and check that the reported recall has not dropped.

Good first tasks:

• New unit test for an existing untested helper in core/.
• Error message refinement in an index package, paired with a test that asserts the returned error.
• New make target or script improvement in Makefile or example/data/download_datasets.sh.
• A doc comment for an exported identifier that lacks one.

Testing Expectations

Follow a red-green cycle. Write the test first, run it, and see it fail for the reason you expect; a test that passes before the change is not testing the change. Then write the smallest change that makes it pass, and refactor with the test green. A bug fix starts with a test that reproduces the bug, so the failure is captured before it disappears.

• Unit tests live in _test.go files alongside the package they cover. The index packages are tested from outside (package hnsw_test), so the tests exercise the exported surface the same way a user does.
• Each index package splits its tests into six files by kind, and a new test belongs in the matching file: index_test.go for behavioral unit tests and shared helpers, quality_test.go for recall and differential tests, concurrency_test.go for stress and race tests, property_test.go for property-based tests, golden_test.go for the golden fixture test and the -update flag, and bench_test.go for benchmarks.
• Every new exported function or behavior change must ship with at least one test that exercises it, including error paths where applicable.
• Test the interface, not the internals. An index test asserts on Search results, on Stats, and on returned errors, not on the shape of the graph or the tree.
• Cover the four operations that can leave an index inconsistent: delete, bulk delete, update, and bulk update. Each must be followed by a search that shows the removed ids are gone and the surviving ids are still reachable.
• Serialization round-trips are tested by saving to a buffer or a t.TempDir() file, loading into a fresh index, and comparing search results with the original. Do not write into the repository.
• Set HANN_SEED in a test that depends on the outcome of a random choice, and use the returned error rather than an exact distance value where the result is approximate.
• Run the race detector on anything that touches goroutines, which includes bulk operations, parallel search, and NormalizeBatch: go test -race -count=1 on the affected packages, more than once.
• A test must not depend on the example datasets, because they are downloaded separately and are not present in CI.
• Tests for the AVX paths must still pass on a machine without AVX, so assert on distance values with a tolerance rather than on bit-exact equality.

Change Design Checklist

Before coding:

1. Packages affected by the change (core, hnsw, pqivf, rpt, or example).
2. Whether the change alters the core.Index interface, an exported signature, or the gob format.
3. Whether the change touches C code, and if so, whether every one of the fallback, AVX, AVX2, and NEON paths was updated.
4. Whether a new external dependency is required, and if so, whether it has been discussed.
5. Whether the change affects recall or query latency, and how that will be measured.

Before submitting:

1. make format passes (no diff).
2. make test passes with the race detector enabled.
3. make lint passes.
4. The examples run locally if the change touches search, distance computation, or serialization.

Commit and PR Hygiene

• Keep commits scoped to one logical change.
• PR descriptions should include:

1. Behavioral change summary.
2. Tests added or updated.
3. Whether the examples or the benchmarks were run locally (yes/no), and on which CPU, since the SIMD path taken depends on it.