Merge pull request #19 from iscc/develop

Develop → Main: CDC optimization, docs, CI fixes

Author: titusz

.claude/agent-memory/advance/MEMORY.md

@@ -24,6 +24,7 @@ iterations.
 ## Build and Tooling
 
 - `cargo build -p iscc-jni` must run before `mvn test` (native library prerequisite)
+- Maven POM is at `crates/iscc-jni/java/pom.xml` — run `mvn test` from `crates/iscc-jni/java/`
 - CI workflow at `.github/workflows/ci.yml` has 9 jobs: version-check, rust, python, nodejs, wasm,
     c-ffi, java, go, bench. The `bench` job runs `cargo bench --no-run` (compile-only, no execution)
 - `version-check` job: lightweight (checkout + setup-python only), runs
@@ -40,66 +41,57 @@ iterations.
 - wasm-opt release flags: `[package.metadata.wasm-pack.profile.release]` with
     `wasm-opt = ["-O", "--enable-bulk-memory", "--enable-nontrapping-float-to-int"]`
 
-## Go Pure Go Rewrite
-
-- Pure Go codec: `packages/go/codec.go` — type enums (`MainType`, `SubType`, `Version` with `iota`),
-    varnibble header encoding/decoding, base32/base64, `EncodeComponent`, `IsccDecompose`,
-    `IsccDecode`. Zero external dependencies
-- Go type naming: `MTMeta`..`MTFlake`, `STNone`..`STWide`, `STText = STNone`, `VSV0 Version = 0`
-- Internal helpers are unexported (lowercase): `encodeHeader`, `decodeHeader`, etc.
-- `IsccDecode` uses `DecodeResult` struct defined in `codec.go`
-- Base32: `base32.StdEncoding.WithPadding(base32.NoPadding)`. Base64: `base64.RawURLEncoding`
-- Pure Go text utils: `TextClean` (NFKC + control-char + empty-line collapse), `TextCollapse` (NFD +
-    lowercase + filter C/M/P + NFKC), `TextTrim` (UTF-8 byte-boundary), `TextRemoveNewlines`
-    (strings.Fields join). Uses `golang.org/x/text/unicode/norm`
-- CDC: `cdcGear` table is `var` not `const` (Go no const arrays). `min()` builtin Go 1.21+
-- MinHash: `minhashFn` naming (avoids conflict). `maxi64`/`mprime`/`maxH` are `var` not `const`
-- SimHash: `AlgSimhash` returns `([]byte, error)`, `SlidingWindow` returns `([]string, error)`. Uses
-    `[]rune` for Unicode-correct SlidingWindow
-- CDC integer ceiling: `(minSize + 1) / 2` (Go has no div_ceil method)
-- DCT: `algDct` (unexported) + `dctRecursive` helper. Only uses `math` stdlib. Nayuki recursive
-    divide-and-conquer. Input must be power of 2 — checked via `n > 0 && n&(n-1) == 0`
-- WTA-Hash: `AlgWtahash` (exported) + `wtaVideoIdPermutations` `[256][2]int` table. No external deps
-- Gen functions: `code_content_text.go` (GenTextCodeV0 + softHashTextV0), `code_meta.go`
-    (GenMetaCodeV0 + metaNameSimhash + softHashMetaV0 + softHashMetaV0WithBytes + interleaveDigests
-    \+ slidingWindowBytes + decodeDataURL + parseMetaJSON + jsonHasContext + buildMetaDataURL +
-    multiHashBlake3), `code_data.go` (GenDataCodeV0 + DataHasher with Push/Finalize),
-    `code_instance.go` (GenInstanceCodeV0 + InstanceHasher with Push/Finalize),
-    `code_content_image.go` (GenImageCodeV0 + softHashImageV0 + transposeMatrix + flatten8x8 +
-    computeMedian), `code_content_audio.go` (GenAudioCodeV0 + softHashAudioV0 + arraySplit[T]).
-    Result types: `TextCodeResult`, `MetaCodeResult`, `DataCodeResult`, `InstanceCodeResult`,
-    `ImageCodeResult`, `AudioCodeResult`, `VideoCodeResult`, `MixedCodeResult`, `IsccCodeResult`
-- xxh32: `xxh32.go` — standalone xxHash32 impl (~80 lines). Used by softHashTextV0 for n-gram
-    feature hashing. Unexported: `xxh32(data, seed)`, `xxh32Round`, `rotl32`, `readU32LE`
-- JCS canonicalization: uses Go stdlib `json.Marshal` (sorts keys, compact format). Works for
-    string/null values in conformance vectors. For full RFC 8785 float compliance, would need a
-    dedicated library
-- BLAKE3 dependency: `github.com/zeebo/blake3` (SIMD-optimized). `blake3.Sum256(data)` returns
-    `[32]byte`
-- Test naming for gen functions: `TestPureGo*` prefix (historical — could be renamed to `Test*` in
-    future cleanup)
-- Go docs: `packages/go/README.md` and `docs/howto/go.md` describe pure Go API (no WASM/wazero).
-    Examples use `iscc.Function(...)` pattern with typed result structs (`*MetaCodeResult`, etc.)
-- Image-Code helpers: `transposeMatrix`, `flatten8x8`, `computeMedian` are unexported in
-    `code_content_image.go`. `bitsToBytes` reused from `codec.go`
-- Audio-Code: `arraySplit[T any]` is generic (Go 1.18+), used for splitting digests into quarters/
-    thirds. `AlgSimhash` on 4-byte digests returns 4 bytes (output = input digest length)
-- `sort.Slice` for int32: `func(i, j int) bool { return s[i] < s[j] }` (no built-in int32 sort)
-- Video-Code: `SoftHashVideoV0` exported (matching Rust `pub fn`). Dedup via
-    `fmt.Sprintf("%v", sig)` string keys in `map[string][]int32`. Column-wise int64 sums →
-    `AlgWtahash`
-- Mixed-Code: `softHashCodesV0` unexported (matching Rust non-pub). Preserves first header byte for
-    type info in SimHash entries. Uses `decodeHeader`/`decodeLength` to validate Content MainType
-    and bit length. `AlgSimhash` error safely discarded (all entries identical length)
-- Go module dependencies: `github.com/zeebo/blake3` (BLAKE3, SIMD), `golang.org/x/text` (Unicode).
-    No wazero or WASM dependencies. `github.com/klauspost/cpuid/v2` indirect (blake3 SIMD detection)
-- Test naming: `TestCodec*`, `TestUtils*`, `TestCdc*`, `TestMinhash*`, `TestSimhash*`,
-    `TestAlgDct*`, `TestAlgWtahash*`, `TestPermutation*`
-- Conformance tests (per-function): `os.ReadFile("../../crates/iscc-lib/tests/data.json")`
-- Conformance selftest: `//go:embed testdata/data.json` in conformance.go.
-    `ConformanceSelftest()   (bool, error)` — package-level function (no receiver). Uses
-    `vectorEntry` struct + 9 `run*Tests` section runners. `decodeStream` shared helper for
-    Data/Instance hex decoding
+## Go Pure Go Rewrite (Summary)
+
+- Pure Go in `packages/go/` — all 10 gen functions + codec + algorithms. Zero WASM deps
+- Dependencies: `github.com/zeebo/blake3`, `golang.org/x/text`. Indirect: `cpuid/v2`
+- Go idioms: unexported helpers (lowercase), `var` for arrays/large uint64 (Go const limitations),
+    `[]rune` for Unicode SlidingWindow, generics for `arraySplit[T]`
+- Conformance: `//go:embed testdata/data.json`, per-function tests use
+    `os.ReadFile("../../crates/iscc-lib/tests/data.json")`
+- 151 Go tests total. CI: 4 steps (checkout, setup-go, test, vet) — no Rust deps
+
+## gen_sum_code_v0
+
+- `gen_sum_code_v0(path: &Path, bits: u32, wide: bool) -> IsccResult<SumCodeResult>` in `lib.rs`
+- Single-pass file I/O: opens file, reads in `IO_READ_SIZE` chunks, feeds both `DataHasher` and
+    `InstanceHasher`, composes ISCC-CODE via `gen_iscc_code_v0`
+- `SumCodeResult { iscc, datahash, filesize }` in `types.rs` — same `#[non_exhaustive]` pattern
+- File I/O errors mapped to `IsccError::InvalidInput("Cannot open/read file: {e}")`
+- `units: Vec<String>` field deferred (not in scope for initial core implementation)
+- 32nd and final Tier 1 symbol for Rust core — all 32 symbols now implemented
+- Python binding: PyO3 wrapper in `crates/iscc-py/src/lib.rs` accepts `&str` path, `SumCodeResult`
+    class in `__init__.py`, public wrapper accepts `str | os.PathLike` via `os.fspath()`, 6 tests in
+    `tests/test_smoke.py`
+- Node.js binding: `NapiSumCodeResult` struct (`#[napi(object)]`) + `gen_sum_code_v0` napi fn in
+    `crates/iscc-napi/src/lib.rs`. Uses `i64` for `filesize` (napi-rs no u64 support). 6 tests in
+    `__tests__/functions.test.mjs`
+- WASM binding: `WasmSumCodeResult` struct (`#[wasm_bindgen(getter_with_clone)]`) +
+    `gen_sum_code_v0` fn in `crates/iscc-wasm/src/lib.rs`. Accepts `&[u8]` (no filesystem in WASM).
+    Uses `f64` for `filesize` (wasm-bindgen `u64` maps to `BigInt`, awkward for JS). Composes
+    internally via `DataHasher` + `InstanceHasher` + `gen_iscc_code_v0`. 6 tests in `tests/unit.rs`,
+    75 total WASM tests (9 conformance + 66 unit; 1 behind `conformance` feature gate)
+- C FFI binding: `IsccSumCodeResult` repr(C) struct with `ok: bool`, `iscc: *mut c_char`,
+    `datahash: *mut c_char`, `filesize: u64`. `iscc_gen_sum_code_v0(path, bits, wide)` extern "C"
+    function + `iscc_free_sum_code_result` free function in `crates/iscc-ffi/src/lib.rs`. Follows
+    `IsccDecodeResult` struct-return pattern. 4 Rust tests + 3 C tests. 82 total Rust tests, 57
+    total C test assertions
+- JNI binding: `SumCodeResult.java` (immutable, `String iscc`, `String datahash`, `long filesize`)
+    - `Java_io_iscc_iscc_1lib_IsccLib_genSumCodeV0` in `crates/iscc-jni/src/lib.rs`. Returns `jobject`
+        via `env.find_class("io/iscc/iscc_lib/SumCodeResult")` + `env.new_object()` with signature
+        `(Ljava/lang/String;Ljava/lang/String;J)V`. 4 Maven tests. 62 total Maven tests
+- Go binding: `packages/go/code_sum.go` — `SumCodeResult` struct (`Iscc`, `Datahash`, `Filesize`) +
+    `GenSumCodeV0(path string, bits uint32, wide bool)`. Single-pass file I/O with `os.Open` +
+    `DataHasher` + `InstanceHasher` + `GenIsccCodeV0`. 4 tests in `code_sum_test.go`. 151 total Go
+    tests. ALL 7 bindings complete for issue #15
+
+## Benchmarks
+
+- `crates/iscc-lib/benches/benchmarks.rs` — all 10 `gen_*_v0` + DataHasher streaming + CDC chunks
+- `bench_sum_code` uses `tempfile::NamedTempFile` since `gen_sum_code_v0` takes `&Path` (not
+    `&[u8]`)
+- Temp files created outside bench closure (setup cost excluded from measurement)
+- `tempfile` is a dev-dependency only (workspace dep `tempfile = "3"`)
 
 ## Codec Internals
 
@@ -148,9 +140,37 @@ iterations.
 
 - All 4 Reference pages complete: Rust API, Python API, C FFI, Java API
 
+## Binding Constant Export Patterns
+
+- NAPI: `#[napi(js_name = "CONST_NAME")] pub const CONST_NAME: u32 = iscc_lib::CONST_NAME as u32;`
+- WASM: `#[wasm_bindgen(js_name = "CONST_NAME")] pub fn const_name() -> u32 { ... }` (getter fn, not
+    const — wasm-bindgen limitation)
+- C FFI: `#[unsafe(no_mangle)] pub extern "C" fn iscc_const_name() -> u32 { ... }` + inline
+    `#[test]` in same file. cbindgen auto-generates the C header
+- NAPI JS tests: `describe('CONST_NAME', () => { it('equals X'); it('is a number'); })`
+- WASM tests: `#[wasm_bindgen_test]` in `tests/unit.rs` (requires wasm-pack to run)
+- C tests: `ASSERT_EQ(iscc_const_name(), value, "label")` in `tests/test_iscc.c`
+- 5 constants currently exported: META_TRIM_NAME, META_TRIM_DESCRIPTION, META_TRIM_META,
+    IO_READ_SIZE, TEXT_NGRAM_SIZE
+
+## Documentation Sweep Patterns
+
+- "N gen" count references exist in: READMEs (9 files), docs/ (14 files), howto/ (6 files), crate
+    CLAUDE.md files (5), notes/ (2), source comments (.rs, .py, .mjs, .pyi), benchmarks/ (2)
+- The Edit tool requires a full Read call (not offset/limit) before the first edit per file
+- mdformat auto-reformats after edits — always run `mise run format` twice after doc changes
+- iscc-core-ts is external and may have different function counts than iscc-lib
+
 ## Gotchas
 
 - JNI package underscore encoding: `iscc_lib` → `iscc_1lib` in function names
 - mdformat auto-formats markdown — keep backtick expressions short to avoid wrapping crashes
 - `from __future__ import annotations` in `__init__.py` — use `|` union syntax, not `Union`
-- Python `__all__` has 45 entries (30 API + 10 result types + `__version__` + MT, ST, VS, core_opts)
+- Python `__all__` has 48 entries (32 API + 11 result types + `__version__` + MT, ST, VS, core_opts)
+- `gen_sum_code_v0` wide mode only differs from normal when `bits >= 128` (wide requires 128-bit+
+    codes)
+- After adding new symbols to `crates/iscc-py/src/lib.rs`, MUST rebuild the `.so` with
+    `uv run maturin develop -m crates/iscc-py/Cargo.toml` before `pytest` will work
+- JSON `{"x":""}` overhead is 8 bytes (not 7) — relevant for boundary tests on META_TRIM_META
+- META_TRIM_META validation: pre-decode check uses `META_TRIM_META * 4/3 + 256` (base64 inflation +
+    media type header), post-decode check uses `META_TRIM_META` directly

.claude/agent-memory/define-next/MEMORY.md

@@ -7,80 +7,57 @@ iterations.
 
 ## Scope Calibration Principles
 
-- CI job additions are small, single-file changes that provide high value. Pattern: copy existing
-    job structure, swap language-specific setup action and build/test commands
 - Critical issues always take priority regardless of feature trajectory
 - Multiple small issues in the same crate are a natural batch (e.g., 3 fixes touching 2 files)
-- README files are "create" operations — less risky than code changes. Doc files excluded from
-    3-file limit
+- Doc files are excluded from the 3-file modification limit — can batch all 6 howto guides in one
+    step since they follow identical patterns
 - When CI is red, formatting/lint fixes are always the first priority regardless of handoff "Next"
 - Prefer concrete deliverables over research tasks when both are available
-- File deletions don't count toward the 3-file modification limit — they are simpler than edits
-- After a major rewrite (e.g., Go pure rewrite), docs/CI lag behind — schedule a cleanup step to
-    bring all stale references in sync before moving to the next feature
-- State assessments can go stale — always verify claimed gaps by reading the actual files. The state
-    may say "met" for something that still has stale content
-- When state says "all automatable work complete," cross-check the spec's verification criteria
-    against actual files — state assessment may miss spec requirements that were never implemented
-    (e.g., missing Reference pages)
+- State assessments can go stale — always verify claimed gaps by reading the actual files
+- New Tier 1 symbols: always implement in Rust core first, then propagate to bindings in separate
+    steps. Core + tests in one step, bindings in subsequent steps
+- When previous next.md already contains correct scoping, verify line references are still accurate
+    and refresh rather than rewrite from scratch — avoid unnecessary churn
+- Repetitive doc additions across language guides: all 6 howto files follow identical structure
+    (heading, 1-line description, fenced code block). Safe to batch all in one step
 
 ## Architecture Decisions
 
-- Java conformance tests use `data.json` via relative path from Maven's working directory
-- Maven Surefire `-Djava.library.path` points to `target/debug/` for finding native cdylib
 - Go bindings are pure Go (no WASM, no wazero, no binary artifacts)
 - All binding conformance tests follow the same structure: load data.json, iterate per-function
     groups, decode inputs per signature, compare `.iscc` output
 - `gen_iscc_code_v0` test vectors have no `wide` parameter — always pass `false`
 - `"stream:<hex>"` prefix denotes hex-encoded byte data for Data/Instance-Code tests
 
-## Documentation Reference Pages Status (Iteration 19)
+## Benchmark Patterns
 
-Documentation spec requires 4 Reference pages:
+- `benchmarks.rs` uses `criterion_group!` macro to register all bench functions
+- Data/Instance/ISCC-Code benchmarks use `BenchmarkId` + `Throughput::Bytes` for throughput metrics
+- `deterministic_bytes(size)` helper generates reproducible test data
+- `gen_sum_code_v0` requires `&Path` (temp file needed) — unlike `gen_data_code_v0` which takes
+    `&[u8]` directly. Temp file must be created OUTSIDE the bench closure
 
-1. Rust API (`rust-api.md`) — ✓ exists
-2. Python API (`api.md`) — ✓ exists
-3. C FFI reference (`c-ffi-api.md`) — ✓ exists (added iteration 18, 694 lines)
-4. Java API — ✗ missing (scoped for iteration 19)
+## Documentation Sweep Patterns
 
-## Java API Reference Page Facts
-
-- `IsccLib.java`: 382 lines, 30 `public static native` methods + 4 `public static final int`
-    constants + private constructor + static NativeLoader block
-- `IsccDecodeResult.java`: 42 lines, 5 `public final` fields (maintype, subtype, version, length,
-    digest)
-- Streaming hashers use opaque `long` handles (JNI pointers) — must document free lifecycle
-- All methods throw `IllegalArgumentException` on invalid input
-- Package: `io.iscc.iscc_lib`, Maven coordinates: `io.iscc:iscc-lib:0.0.2`
+- `crates/iscc-wasm/pkg/README.md` must always be identical to `crates/iscc-wasm/README.md` — both
+    are published to npm
+- When updating "9 gen functions" to "10", distinguish context: data.json has 9 function sections
+    (no gen_sum_code_v0), so conformance/benchmark code correctly says "9"
+- Two docs pages (architecture.md, development.md) share identical directory tree and crate summary
+    table — edits must be synced between them
 
 ## CI/Release Patterns
 
-- Release workflow has `workflow_dispatch` inputs: `crates-io`, `pypi`, `npm`, `maven` (booleans)
-- All publish jobs have idempotency checks (version-existence pre-check, `skip` output)
-- `scripts/version_sync.py` uses only stdlib — can run as `python scripts/version_sync.py --check`
-
-## Project Near-Completion State (Iteration 19)
-
-All 7 bindings at 30/30, CI green with 9 jobs. PR #10 exists from develop→main.
-
-**Remaining automated gaps:**
-
-1. Java API reference page — SCOPED (iteration 19, LAST automatable task)
-2. Tab order standardization — LOW priority, needs human review
-
-**Human-only tasks remaining after iteration 19:**
-
-- Merge PR #10 (develop → main)
-- Trigger releases for npm, PyPI
-- Maven Central publishing (GPG, Sonatype)
-- Tab order decision
-- OIDC for crates.io
+- v0.0.3 released to all registries. Next release after remaining gaps closed.
+- Release workflow has `workflow_dispatch` with per-registry checkboxes
 
 ## Gotchas
 
 - JNI function names encode Java package underscores as `_1`
 - WASM howto uses `@iscc/wasm` (not `@iscc/iscc-wasm`). npm lib is `@iscc/lib`
-- ISCC Foundation URL is `https://iscc.io`
 - Java `byte` is signed — values 128-255 wrap, JNI handles correctly
-- Two docs pages (architecture.md, development.md) share identical directory tree and crate summary
-    table — edits must be synced between them
+
+## Project Status
+
+- Iteration 16: Only gap is bench_sum_code. After that, only issue #16 (low priority, feature flags)
+    remains. Project approaching full target completion.