fix potential integer overflow in raw API

[dloebl]

Regression introduced with v0.5.1/#84.
There is potentially a small integer overflow in the LZW encoding logic affecting very large GIFs (65.535 x 65.535):

pContext->pLZWData = malloc(sizeof(uint16_t) * (numPixel + 2 + maxResets));

u32:numPixel is at max 4.294.836.225: (2^16-1)^2
u32:maxResets is at max 1.119.029: 4.294.836.225/3838

And now 4.294.836.225 + 1.119.029 + 2 is 4.295.955.256 - which is above the maximum value a 32-bit unsigned integer can hold 4.294.967.295 (2^32-1).

The fix is to simply cast to size_t before the calculation:

pContext->pLZWData = malloc(sizeof(uint16_t) * ((size_t)numPixel + 2 + maxResets));

This is only affecting very large GIFs (at the very end of the dimension limit), with small color palettes.
In a follow-up, we can also make this handling more gracefully on 32-bit systems - but for now it's fine.

[Copilot]

Pull request overview

This pull request fixes a potential integer overflow in the LZW encoding logic for very large GIFs (65,535 x 65,535 pixels). The overflow occurs when calculating the malloc size for LZW data, where the sum of numPixel + 2 + maxResets can exceed the maximum value of a 32-bit unsigned integer (4,294,967,295). The fix casts numPixel to size_t before the arithmetic, ensuring the entire expression is evaluated using the larger type.

Changes:

• Cast numPixel to size_t in malloc size calculation to prevent integer overflow for maximum-sized GIFs

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.