…ning

- Use header.HugepageSize for uncompressed fetch alignment (semantically correct)
- Stream NFS cache hits directly into ReadFrame instead of buffering in memory
- Fix timer placement to cover full GetFrame (read + decompression)
- Fix onRead callback: nil for compressed inner calls (prevents double-invoke),
  pass through for uncompressed (bytes are final)
- Remove panic recovery from runFetch (never in main)
- Remove low-value chunker tests subsumed by ConcurrentStress
- Remove 4MB frame configs from benchmarks (targeting 2MB only)
- Remove unused readCacheFile function

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ble NFS cache

- Remove dead flagsClient chain through chunker/build/template layers (~15 files)
- Delete ChunkerConfigFlag (unused after flagsClient removal)
- Delete mock_flagsclient_test.go
- Simplify GetUploadOptions: remove redundant intOr/strOr fallbacks (flags have defaults)
- Add GetCompressionType helper to frame_table.go, deduplicate compression type extraction
- Replace [16]byte{} with uuid.Nil and "rootfs.ext4" with storage.RootfsName in inspect-build
- Simplify UploadV4Header return pattern
- Remove onRead callback from legacy fullFetchChunker (FullFetch should not use progressive reads)
- Re-enable NFS cache in template cache.go
- Remove all fmt.Printf debug instrumentation from orchestrator

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…sionType threading

Add per-build file size and SHA-256 checksum to V4 headers, eliminating
the redundant Size() network call when opening upstream data files on
the read path. Checksums are computed for free by piggybacking on
CompressStream's existing frame iteration.

Remove the separate compressionType parameter threaded through
getBuild → newStorageDiff → NewChunker; the read path now derives
compression state from the per-mapping FrameTable directly.

V4 binary format change (not yet deployed):
  [Metadata] [LZ4: numBuilds, builds(uuid+size+checksum),
              numMappings, mappings...]

V3 path unchanged — falls back to Size() call when size is unknown.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… lev-compression-final

GetUploadOptions now accepts fileType and useCase parameters, enriching
the LD evaluation context so dashboard targeting rules can differentiate
(e.g. compress memfile but not rootfs, or builds but not pauses).
TemplateBuild accepts per-file opts directly instead of holding an ff
reference.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…rtUploader interface

CompressConfig with LaunchDarkly feature flag support, compressStream
pipeline (parallel frame compression → ordered emit → concurrent upload),
and GCS multipart partUploader implementation with zero-copy slice uploads.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…leInfo, and validation

V4 headers store per-mapping FrameTables and per-build file metadata
(size + SHA-256 checksum). The mappings block is LZ4-compressed with a
uint32 size prefix for exact-size decompression.

Adds: SerializeHeader, DeserializeBytes (auto-detecting V3/V4),
LoadHeader, StoreHeader, ValidateHeader, CloneForUpload, AddFrames,
mergeFrameTables, and compressed path helpers on TemplateFiles.

Existing Serialize/Deserialize/DeserializeBytes APIs preserved for
backward compatibility — signature changes deferred to read-path PR.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove unused OverrideJSONFlag, propagate Subset errors through
MergeMappings, align newCompressorPool signature with final, and
trim redundant/trivial tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Resolve conflicts: keep final's behavior/API (Serialize/Deserialize,
LoadHeader, StoreHeader, show-build-diff), apply primitives cleanup
(assert→require, removed duplicate tests, deduplicated CompressConfigFlag).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… cleanup, deterministic serialization

- Switch back to LZ4 streaming API: handles incompressible data
  automatically, adds per-block xxHash32 checksums, pool encoders
  and decoders symmetrically with zstd
- Move header LZ4 compress/decompress into header package (V4 wire format)
- Always wait on uploadEG before returning (errors.Join)
- Sort BuildFiles by UUID for deterministic V4 serialization
- Remove dead decoderConcurrency key from CompressConfigFlag
- Fix CompressConfigFromLDValue ctx parameter order

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… into lev-compression-primitives

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Compression read/write path changes on top of the primitives merge:

  Bug fixes:
  - P2P peer transition now sends V4 headers (was sending V3 from snapshot)
  - FinalizeHeaders returns serialized bytes; StoreHeader returns ([]byte, error)
  - peerFramedFile.GetFrame requests len(buf) bytes (was requesting blockSize)

  Diff minimization:
  - Restore OpenBlob 3-arg signature to match main
  - Restore main's variable style, comments, span names throughout
  - Revert gratuitous changes: storage_fs_test.go, cache.go, template_metadata.go
  - Inline createChunker into Init (mirrors main's structure)
  - Chunker takes storagePath instead of buildID+fileType
  - Resolve CompressConfig at call site, not in LayerExecutor
  - storagePath private (matches main), remove buildFileSize verbosity

  Code quality:
  - Rename chunk_framed.go → chunk.go (only implementation)
  - Consolidate storage mocks in-package, delete mocks/ subdir
  - Move PeerTransitionedError into storage.go
  - Precomputed metrics as two plain vars (chunkerAttrs, chunkerAttrsCompressed)
  - assert→require in chunk_test.go, deterministic test data, remove unused pipelinedReader
  - Delete redundant TestDiffStoreConcurrentInitAndAccess
  - Simplify StoreFile — explicit returns instead of named return dance

  Integration tests:
  - Merge compression tests into regular suite (remove build tag + separate target)
  - Move TestLargeMemoryPauseResume into sandbox_pause_test.go
  - GHA workflow always runs with LZ4 level 0, 8 workers
  - Strip compression suffix at server (resolve.go) not client

Addresses dobrac's review comments on the compression primitives PR:

  Serialization:
  - Split serialization.go into serialization_v3.go and serialization_v4.go
  - Each version's wire format is self-contained in one file
  - Move Metadata types/constants to metadata.go
  - Remove unused exported Serialize() wrapper

  V4 wire format:
  - Replace packed CompressionTypeNumFrames uint64 with separate
    CompressionType uint32 + NumFrames uint32 (no bit-shifting)
  - Remove MaxCompressedHeaderSize limit (uint32 prefix + LZ4 frame
    boundary are sufficient)

  Naming and reuse:
  - Rename AddFrames → SetFrames (replaces, not appends)
  - Use SetFrames in MergeMappings instead of inline Subset calls
  - Remove unnecessary maps.Clone in ToDiffHeader

  Formatting:
  - Use decimal (%d) instead of hex (%#x) in error messages and
    String() methods for consistency with the rest of the codebase

  Tests:
  - Add test for uploadPartSlices (MD5 hashing, body concatenation)

  Documentation:
  - Flag BuildFiles incompleteness: only contains builds from current
    upload session, not upstream dependencies

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Rename TemplateFiles → Paths (it maps a build ID to storage paths)
- Remove generic string-accepting methods (DataPath, HeaderPath,
  CompressedDataPath) that allowed callers to bypass constants
- Add explicit compressed path methods (MemfileCompressed,
  RootfsCompressed) for the new compression write path
- Rename free functions for consistency: ParseStoragePath → SplitPath,
  BaseFileName → StripCompression, CompressedPath → AppendCompression
- Fix manual string concatenation in template/storage.go to use Paths
- Simplify build_upload helpers to take resolved storage paths

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… lev-compression-final

- Remove ValidateHeader from NewHeader — avoids rejecting zero-size
  templates on deserialization and removes O(n log n) sort from the
  hot read path. ValidateHeader remains exported for CLI/diagnostic use.
- Fix O(N²) in applyToHeader: add SubsetFrom/SetFramesFrom with cursor
  so consecutive sorted mappings walk the FrameTable once instead of
  rescanning from the start for each mapping.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…final

…ingReader

Replace the monolithic FramedFile.GetFrame() callback-based API with a
composable Seekable interface backed by io.ReadCloser streams. This
eliminates the ReadFrame/readInto/readFrameDecompress plumbing from the
storage layer and pushes progressive read logic into the Chunker.

Storage interfaces:
- Remove FramedFile (GetFrame + Size + StoreFile) and RangeReadFunc
- Add Seekable (StreamingReader + SeekableWriter + Size) with
  OpenRangeReader returning io.ReadCloser for both compressed and
  uncompressed paths
- Add SeekableReader interface (ReadAt + Size) used by build.Diff
- Add NewDecompressingReader for pooled LZ4/zstd decompression
- Split compress_pool.go into compress_encode.go + compress_decode.go
- StorageProvider.OpenFramedFile → OpenSeekable throughout

Chunker (streaming_chunk.go):
- Accept StreamingReader instead of StorageProvider + storagePath
- Replace GetFrame callback with io.ReadFull loop in progressiveRead()
- Feature-flag-tunable min read batch size (MinChunkerReadSizeKB)
  replacing the removed ChunkerConfigFlag JSON flag
- Rename ReadBlock→ReadAt, SliceBlock→Slice
- Split FramedBlockReader into FramedReader + FramedSlicer interfaces

NFS cache (storage_cache_seekable.go → + storage_cache_compressed.go):
- Rewrite cachedFramedFile as cachedSeekable implementing Seekable
- Extract compressed cache path into storage_cache_compressed.go with
  TeeReader-based write-through on Close
- Uncompressed path uses write-through NFS caching with isCompleteRead
  validation

P2P / gRPC:
- Reduce diff with main to the essential: peerserver data-serving path
  is unchanged (renames only: FramedSource→SeekableSource, framed.go→
  seekable.go, SliceBlock→Slice). Proto RPC renamed GetBuildFrame →
  ReadAtBuildSeekable.
- Peerclient: replace peerFramedFile with peerSeekable, adding
  OpenRangeReader for streaming peer reads and fixing peerStreamReader
  to fill the caller's buffer across gRPC message boundaries.

Build path:
- StorageDiff.Init opens Seekable once, passes it to NewChunker
- getBuild receives CompressionType to construct the correct data path
- Diff interface now embeds SeekableReader + FramedSlicer (was
  FramedBlockReader)
- NoDiff and localDiff implement Size() for SeekableReader conformance

Tests:
- Rewrite chunk tests using fakeSeekable + controlledChunker with
  channel-based flow control (no time.Sleep)
- Add TestChunker_CacheHit, TestChunker_PanicRecovery,
  TestChunker_ConcurrentSameChunk
- Rewrite peerclient tests for peerSeekable (ReadAt, OpenRangeReader,
  header transition, upload-skips-peer)
- Regenerate mocks (mockdiff, mock_seekable, orchestrator mocks)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Minimize diff with lev-paths-refactor base branch:
- Restore SeekableObjectType parameter to OpenSeekable interface,
  implementations, mocks, and all call sites
- Restore storageHeaderObjectType/storageObjectType helpers and
  struct fields in template/storage.go and build/storage_diff.go
- Restore peerStreamReader.Read to base's one-message-per-Read
- Restore StreamingReader type assertions on gcpObject and fsObject
- Restore isCompleteRead 3-param signature with err/EOF handling
- Restore cache_seekable tests via testReadAt helper preserving
  base test names and structure
- Revert cosmetic renames (files→paths, variable names, comments)
- Restore base interface comments on SeekableReader, SeekableWriter
- Remove redundant TestLargeMemoryPauseResume integration test

Fix compression bugs:
- Fix nil FrameTable panic: move cfg.IsEnabled() check before
  small-file optimization in GCP StoreFile
- Add nil FrameTable guard in compressedUploader.UploadData

Clean up storage package:
- Replace compositeReadCloser with newDecompressingReadCloser
- Privatize internal symbols: parseCompressionType,
  newDecompressingReadCloser, compressConfigFromLDValue (inlined)
- Unify openRangeReader length param to int64 across providers
- Simplify AWS: inline OpenRangeReader, error on compressed
- Rename compressedCacheReader → decompressingCacheReader
- Use isCompleteRead consistently in both cache write-through paths
- Idiomatic error handling in decompressingCacheReader.Close

Add OTEL instrumentation to read paths:
- Span ("read") on cachedSeekable.OpenRangeReader covering full
  open-to-close lifecycle via withSpan wrapper, with offset/length/
  compressed attributes
- NFS cache read timer on both uncompressed and compressed paths
  with compressed/compression_type attributes
- Write-back span on decompressingCacheReader.Close matching
  existing span on cacheWriteThroughReader.Close
- Extract constructors: withSpan, newCacheWriteThroughReader,
  newDecompressingCacheReader — sequential wrapping, no nesting

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>