Skip to content

architecture: refactor to 3-stage pipeline with separated Reader, Decoder, Encoder goroutines #4

@more-shubham

Description

@more-shubham

Currently processImage does file I/O, decode, and encode sequentially in one function inside a single worker goroutine. Separating into 3 pipeline stages allows each stage to run concurrently and makes bottleneck identification precise.

Scope:

  • internal/pipeline/reader.go — file open and raw bytes read, outputs to decode channel
  • internal/pipeline/decoder.go — JPEG/PNG decode from raw bytes, outputs image.Image to encode channel
  • internal/pipeline/encoder.go — WebP encode and file write, outputs Result
  • Each stage has its own bounded channel and worker count
  • Add stage-level metrics: time spent in read vs decode vs encode separately

Architecture:

JobsChan → [Reader goroutines] → rawChan → [Decoder goroutines] → imgChan → [Encoder goroutines] → ResultsChan

Acceptance Criteria:

  • Each stage independently configurable in worker count
  • Stage timing visible in summary.json under stage_metrics
  • Memory profile shows decode buffer reuse across pipeline
  • All existing tests pass against new architecture
  • BENCHMARKS.md updated with stage-level timing breakdown

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions