Document persistent designs and factor encoding

[leostimpfle]

Hi @schroedk ! I'm on holiday this week so had too much time to create too many pull requests 🙂

This PR concludes the stack #326 to address #268 and #269. I'll try to outline the stack's structure with the hope that it will make it easier to follow what's going on. Although there are many PRs (15 including this one), each individual diff is small and the core changes are in four PRs with the remaining ones being mostly mechanical refactors (see below).

This PR stack introduces two core features

• A persistent Design API: Persistent Design #269
• Internal factor encoding: Internal factor encoding #268

A core design decision is the separation of an immutable Design and a solver-specific SolverState:

within/crates/within-py/src/api.rs

Lines 325 to 328 in a5e9282

	pub struct PySolver {
	design: Py<PyDesign>,
	state: SolverState,
	}

This separates a persistent Design's ownership from a Solver's local state (e.g., weighting, slope whitening, ...). I have landed on this separation after encountering some ownership awkwardness in #320. Note that the current implementation does not ensure that a SolverState is consistent with a Design, so if we end up with this implementation, we may want to add additional guardrails (similar to #270).

Illustration of new features

The changes allow us to do something like this:

import numpy as np
from within import Design, Solver, solve

# Caller labels are deliberately non-contiguous.
categories = np.asfortranarray(
    np.array(
        [[10, 100], [20, 100], [10, 900], [20, 900]],
        dtype=np.uint32,
    )
)
y = np.array([1.0, 2.0, 3.0, 4.0])

# Build and compact the design once, then share it across solver states.
design = Design(categories)
unweighted = Solver(design)
weighted = Solver(design, weights=np.array([1.0, 2.0, 1.0, 2.0]))

result = unweighted.solve(y)
weighted_result = weighted.solve(y)
one_shot_result = solve(design, y)

# Internal factor positions stay private; result lookup uses caller labels.
slot = result.layout.index(term=1, level=900, column=0)
assert result.layout.address(slot) == (1, 900, 0)
assert result.x.shape == (4,)  # two observed levels in each of two factors

---

Guide to PR stack

The core changes are in four PRs:

PR	Outcome
#316	Adds FactorLabel abstraction to make coefficient layouts and unidentified directions use caller-visible factor labels.
#324	Decomposes a Solver into a design-specific SolverState and an immutable Design.
#329	Exposes the persistent Design in Python and allows multiple Python solvers to share it.
#333	Compacts arbitrary observed u32 labels into dense internal positions while preserving caller-label result lookup.

The remaining PRs have a mostly supporting role to enable the four core changes above:

PR	Outcome
#309	Encapsulates the design frame before its categorical columns can be rewritten internally.
#314	Introduces the internal FactorEncoding abstraction used by caller-label translation.
#318	Separates the immutable design from solver-local loading transformations.
#319	Adds construction of a reusable Rust Design from a categories matrix.
#323	Makes that solver-local view borrow the persistent design.
#327	Adds Solver::from_design, allowing multiple Rust solvers to borrow one persistent design.
#328	Makes SolverState public so the Python facade can own it alongside a persistent Design.
#332	Accepts a persistent Python Design in the one-shot solve and solve_batch functions as well.
#334	Removes an argsort branch made unreachable by compact internal codes.
#335	Removes active-level scanning and remapping made redundant by compact internal codes.

---

Potential clean-ups (if we settle on the basic implementation in this stack)

• Enable Solver construction from borrowed design #327 introduced Solver construction from a borrowed &'a Design<a'> without touching the existing owned construction to keep the diff small. We may want to remove the "owned" and only keep the "borrowed" construction.
• The current API does not prevent usage of a Design with a SolverState constructed from a different Design. Although SolverState is public, it is not intended be part of the main user-facing API. It only needs to be public for within-py.