fix(stats): perf regression because of memory-aware chunking logic error

[jqnatividad]

No description provided.

[Copilot]

Pull request overview

Fixes a performance regression in stats chunk sizing by tightening when memory-aware chunking is used and adjusting chunk-size selection logic intended to balance memory constraints vs parallelism.

Changes:

• Switch stats to only do sampling/memory-aware chunk sizing when non-streaming stats are enabled (avoids overhead for streaming-only stats).
• Modify dynamic chunk-size selection logic in util::calculate_dynamic_chunk_size.
• Refactor mode/cardinality updates into a helper and adjust typesonly + infer_boolean behavior to compute only what boolean inference needs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/util.rs	Updates dynamic chunk-size selection criteria (memory-based vs CPU-based).
src/cmd/stats.rs	Limits memory-aware chunking to non-streaming stats; refactors mode updates; tweaks typesonly/infer_boolean stats selection.

Comments suppressed due to low confidence (2)

src/util.rs:1204

• In calculate_dynamic_chunk_size, the new preference condition selects cpu_based_chunk_size whenever memory_based_chunk_size <= cpu_based_chunk_size. That’s the “memory constrained” case where we generally need the smaller, memory-based chunk size to avoid high peak memory for non-streaming stats. As written, this can negate memory-aware chunk sizing and potentially reintroduce OOM risk. Consider restoring logic that prefers memory_based_chunk_size when it is smaller, and only prefers cpu_based_chunk_size when memory_based_chunk_size is at least as large (i.e., memory permits CPU-based chunking) and CPU-based yields better parallelization.

        // Prefer CPU-based chunking if:
        // 1. Memory-based chunk size is smaller than CPU-based (degenerate/low memory), OR
        // 2. It creates more chunks (better parallelization)
        if memory_based_chunk_size <= cpu_based_chunk_size || cpu_based_chunks > memory_based_chunks
        {
            cpu_based_chunk_size
        } else {
            memory_based_chunk_size
        }

src/cmd/stats.rs:3200

• The new update_modes() helper is inserted between the doc comment for add() and the add() method itself, so Rust will attach the long "Adds a sample value…" doc comment to update_modes() instead of add(). As a result, add() now only has the trailing bullet doc lines and its main documentation is misplaced. Move update_modes() above the doc comment or below add(), and give update_modes its own brief doc comment (or a regular // comment) so docs render correctly.

    /// # Safety
    ///
    /// * Uses unsafe code for performance-critical operations
    /// Updates modes/cardinality trackers with a sample value.
    /// Weighted modes and unweighted modes are mutually exclusive.
    #[inline(always)]
    fn update_modes(&mut self, sample: &[u8], weight: f64) {
        if let Some(ref mut wm) = self.weighted_modes {
            // Weighted modes: accumulate weights per value
            // Use get_mut first to avoid heap-allocating sample.to_vec() when key already exists
            if let Some(val) = wm.get_mut(sample) {
                *val += weight;
            } else {
                wm.insert(sample.to_vec(), weight);
            }
        } else if let Some(v) = self.modes.as_mut() {
            v.add_bytes(sample);
        }
    }

    /// * Assumes valid UTF-8 input for string operations
    /// * Bounds checking is avoided where safe
    #[allow(clippy::inline_always)]
    #[inline(always)]
    fn add(