Performance: Eliminate unnecessary header processing in request hot path

[jeremyandrews]

GitHub Issue: Optimize Header Processing to Eliminate Unnecessary Allocations

Title

Performance: Eliminate unnecessary header processing in request hot path

Labels

• enhancement
• good first issue

Issue Description

Problem

Currently, every HTTP request processes headers into a Vec<String> even when request logging is disabled, causing unnecessary vector allocations and string formatting overhead. This happens in src/goose.rs in the request() method:

// Always processes headers even when not needed
let mut headers: Vec<String> = Vec::new();
for header in built_request.headers() {
    headers.push(format!("{header:?}"));  // Allocation + formatting for EVERY header
}

let raw_request = GooseRawRequest::new(
    method.clone(),
    &final_url,
    headers,  // Vec<String> always created, even when unused
    body,
);

Performance Impact

• Vector allocation + string formatting per header on every request
• Wasted CPU cycles processing headers when request logging is disabled
• Unnecessary memory pressure from header string allocations
• Scale: In a test with 1M requests × 10 headers each = 10M unnecessary string allocations
• Always active: This overhead exists even when users don't need request logging

Solution

Implement lazy header processing that only converts headers to strings when actually needed for logging.

Proposed Changes

1. Modify the request method to conditionally process headers:

fn create_raw_request(
    method: GooseMethod,
    url: &str, 
    headers: &HeaderMap,
    body: &str,
    config: &GooseConfiguration
) -> GooseRawRequest {
    let headers = if config.request_log.is_empty() {
        Vec::new()  // Don't process headers if not logging requests
    } else {
        // Only process headers when actually needed for logging
        headers.iter()
            .map(|(name, value)| format!("{}: {:?}", name, value))
            .collect()
    };
    
    GooseRawRequest::new(method, url, headers, body)
}

2. Update the main request method:

// In src/goose.rs request() method, replace the current header processing:
let raw_request = create_raw_request(
    method.clone(),
    &final_url,
    built_request.headers(),
    body,
    &self.config,
);

3. Alternative approach - lazy evaluation with closure:

// Store a closure instead of processed headers for even better performance
pub struct GooseRawRequest {
    pub method: GooseMethod,
    pub url: String,
    pub headers: Box<dyn Fn() -> Vec<String> + Send + Sync>, // Lazy evaluation
    pub body: String,
}

// Only evaluate when needed (e.g., during logging)
impl GooseRawRequest {
    fn get_headers(&self) -> Vec<String> {
        (self.headers)()
    }
}

Expected Performance Improvement

• 70-90% reduction in header processing overhead when logging disabled
• Zero header allocations when request logging is not enabled
• Faster request processing due to eliminated string formatting
• Reduced memory pressure and better cache utilization
• Immediate benefit for most users who don't enable request logging

Implementation Effort

• Difficulty: Low
• Files to modify: src/goose.rs (primarily the request() method)
• Breaking changes: None (internal implementation detail)
• Testing: Can be validated with existing test suite + performance benchmarks

Acceptance Criteria

• Headers are not processed when request_log is empty
• Headers are correctly processed and formatted when request logging is enabled
• No performance regression when request logging is enabled
• Zero header-related allocations when request logging is disabled
• All existing functionality preserved
• Performance benchmarks show improvement
• Tests pass

Testing Strategy

1. Unit tests: Verify header processing logic works correctly in both modes
2. Integration tests: Ensure request logging still captures headers properly
3. Performance tests: Benchmark header processing overhead before/after
4. Memory profiling: Confirm allocation elimination when logging disabled
5. Functional tests: Validate that request logs contain proper header information

Performance Test Cases

1. High-traffic test with logging disabled: Should show significant allocation reduction
2. High-traffic test with logging enabled: Should maintain same functionality
3. Mixed scenario: Enable/disable logging during test to verify dynamic behavior

Implementation Notes

• This optimization provides immediate benefit since most users don't enable request logging
• The change is completely backward compatible
• Consider making the header processing configurable beyond just logging (e.g., metrics collection)
• Ensure thread safety if implementing lazy evaluation approach

Code Locations

• Primary file: src/goose.rs
• Method: request() around lines where headers are processed
• Struct: GooseRawRequest::new() constructor

Related Optimizations

This optimization works well in combination with:

• Issue Performance: Eliminate string formatting in metrics aggregation hot path #637: Metrics key optimization
• Issue Performance: Eliminate string allocations in request metrics hot path #638: String allocation elimination
• Future header-related optimizations (e.g., header pooling)