Prevent heap buffer overflow from Y4M dimension overflow

[Naveed8951]

Summary

This PR fixes a verified memory-corruption vulnerability in the Y4M input parser caused by unsafe signed integer arithmetic when computing frame buffer sizes from attacker-controlled header dimensions. The fix enforces strict dimension validation and converts all size computations to checked size_t arithmetic to prevent heap buffer overflows.

Severity

S0 – heap buffer overflow reachable via crafted input.

Affected Components

• Y4M input parsing and buffering
  • y4m_input_open()
  • y4m_parse_tags()
  • y4m_input_fetch_frame()

Vulnerability Description

Y4M header tags, notably W (width) and H (height), are attacker-controlled. Previously, these values were parsed into signed int fields and used in int arithmetic when computing frame buffer sizes such as dst_buf_read_sz, particularly for high-bit-depth formats (e.g. C420p10, C420p12).

For sufficiently large dimensions, signed integer multiplication could overflow, producing undefined behavior and negative intermediates. When later converted to size_t, these values became very large positive sizes. The frame read path would then attempt to read that many bytes into a heap buffer allocated using a different, non-overflowed size computation, resulting in a heap buffer overflow.

Fix Description

• Enforces strict positive-integer parsing for W and H tags and rejects invalid, zero, negative, or overflow-prone values.
• Converts all frame size and read size computations to checked size_t arithmetic.
• Adds explicit invariants ensuring:
  • dst_buf_read_sz never exceeds the allocated destination buffer size.
  • aux_buf_read_sz never exceeds aux_buf_sz.
• Switches Y4M buffer allocations to vpx_malloc / vpx_free for consistent bounded allocation behavior.
• Hardens plane and stride computations in y4m_input_fetch_frame() by validating size calculations and casts.

Before vs After

Before:
Large or malformed W / H values could trigger signed integer overflow, leading to incorrect read sizes and heap buffer overflows during frame reads.

After:
Invalid or overflow-inducing dimensions are rejected early, and all derived sizes are computed using checked arithmetic and validated against allocations, preventing memory corruption.

Testing

• Verified clean builds after changes.
• Recommended runtime validation includes running vpxenc against valid Y4M inputs across supported chroma types and bit depths, and fuzzing Y4M headers focused on W, H, and chroma tag combinations.