fbcon: fix integer overflow in fbcon_do_set_font

commit 1a194e6 upstream.

Fix integer overflow vulnerabilities in fbcon_do_set_font() where font
size calculations could overflow when handling user-controlled font
parameters.

The vulnerabilities occur when:
1. CALC_FONTSZ(h, pitch, charcount) performs h * pith * charcount
   multiplication with user-controlled values that can overflow.
2. FONT_EXTRA_WORDS * sizeof(int) + size addition can also overflow
3. This results in smaller allocations than expected, leading to buffer
   overflows during font data copying.

Add explicit overflow checking using check_mul_overflow() and
check_add_overflow() kernel helpers to safety validate all size
calculations before allocation.

Signed-off-by: Samasth Norway Ananda <[email protected]>
Reviewed-by: Thomas Zimmermann <[email protected]>
Fixes: 39b3cff ("fbcon: prevent user font height or width change from causing potential out-of-bounds access")
Cc: George Kennedy <[email protected]>
Cc: stable <[email protected]>
Cc: [email protected]
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Simona Vetter <[email protected]>
Cc: Helge Deller <[email protected]>
Cc: Thomas Zimmermann <[email protected]>
Cc: "Ville Syrjälä" <[email protected]>
Cc: Sam Ravnborg <[email protected]>
Cc: Qianqiang Liu <[email protected]>
Cc: Shixiong Ou <[email protected]>
Cc: Kees Cook <[email protected]>
Cc: <[email protected]> # v5.9+
Signed-off-by: Thomas Zimmermann <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Greg Kroah-Hartman <[email protected]>

Author: samasth-norway

drivers/video/fbdev/core/fbcon.c

@@ -2509,9 +2509,16 @@ static int fbcon_set_font(struct vc_data *vc, struct console_font *font,
 	if (fbcon_invalid_charcount(info, charcount))
 		return -EINVAL;
 
-	size = CALC_FONTSZ(h, pitch, charcount);
+	/* Check for integer overflow in font size calculation */
+	if (check_mul_overflow(h, pitch, &size) ||
+	    check_mul_overflow(size, charcount, &size))
+		return -EINVAL;
+
+	/* Check for overflow in allocation size calculation */
+	if (check_add_overflow(FONT_EXTRA_WORDS * sizeof(int), size, &size))
+		return -EINVAL;
 
-	new_data = kmalloc(FONT_EXTRA_WORDS * sizeof(int) + size, GFP_USER);
+	new_data = kmalloc(size, GFP_USER);
 
 	if (!new_data)
 		return -ENOMEM;