KSES: Prevent normalization from unescaping escaped numeric character references.

Fixes an issue where `wp_kses_normalize_entities` would transform inputs like "'" into "'", changing the intended HTML text.

This behavior has present since the initial version of KSES was introduced in [649].

[2896] applied the normalization to post content for users without the "unfiltered_html" capability.

Developed in WordPress/wordpress-develop#9099.

Props jonsurrell, dmsnell, sirlouen.
Fixes #63630.

Built from https://develop.svn.wordpress.org/trunk@60616


git-svn-id: http://core.svn.wordpress.org/trunk@59952 1a063a9b-81f0-0310-95a4-ce76da25c4cd

Author: sirreal

wp-includes/kses.php

@@ -1958,14 +1958,45 @@ function wp_kses_normalize_entities( $content, $context = 'html' ) {
 	// Disarm all entities by converting & to &
 	$content = str_replace( '&', '&', $content );
 
-	// Change back the allowed entities in our list of allowed entities.
+	/*
+	 * Decode any character references that are now double-encoded.
+	 *
+	 * It's important that the following normalizations happen in the correct order.
+	 *
+	 * At this point, all `&` have been transformed to `&`. Double-encoded named character
+	 * references like `&` will be decoded back to their single-encoded form `&`.
+	 *
+	 * First, numeric (decimal and hexadecimal) character references must be handled so that
+	 * `	` becomes `	`. If the named character references were handled first, there
+	 * would be no way to know whether the double-encoded character reference had been produced
+	 * in this function or was the original input.
+	 *
+	 * Consider the two examples, first with named entity decoding followed by numeric
+	 * entity decoding. We'll use U+002E FULL STOP (.) in our example, this table follows the
+	 * string processing from left to right:
+	 *
+	 * | Input        | &-encoded        | Named ref double-decoded  | Numeric ref double-decoded |
+	 * | ------------ | ---------------- | ------------------------- | -------------------------- |
+	 * | `.`     | `.`     | `.`              | `.`                   |
+	 * | `.` | `.` | `.`              | `.`                   |
+	 *
+	 * Notice in the example above that different inputs result in the same result. The second case
+	 * was not normalized and produced HTML that is semantically different from the input.
+	 *
+	 * | Input        | &-encoded        |  Numeric ref double-decoded | Named ref double-decoded |
+	 * | ------------ | ---------------- | --------------------------- | ------------------------ |
+	 * | `.`     | `.`     | `.`                    | `.`                 |
+	 * | `.` | `.` | `.`            | `.`             |
+	 *
+	 * Here, each input is normalized to an appropriate output.
+	 */
+	$content = preg_replace_callback( '/&#(0*[0-9]{1,7});/', 'wp_kses_normalize_entities2', $content );
+	$content = preg_replace_callback( '/&#[Xx](0*[0-9A-Fa-f]{1,6});/', 'wp_kses_normalize_entities3', $content );
 	if ( 'xml' === $context ) {
 		$content = preg_replace_callback( '/&([A-Za-z]{2,8}[0-9]{0,2});/', 'wp_kses_xml_named_entities', $content );
 	} else {
 		$content = preg_replace_callback( '/&([A-Za-z]{2,8}[0-9]{0,2});/', 'wp_kses_named_entities', $content );
 	}
-	$content = preg_replace_callback( '/&#(0*[0-9]{1,7});/', 'wp_kses_normalize_entities2', $content );
-	$content = preg_replace_callback( '/&#[Xx](0*[0-9A-Fa-f]{1,6});/', 'wp_kses_normalize_entities3', $content );
 
 	return $content;
 }

wp-includes/version.php

@@ -16,7 +16,7 @@
  *
  * @global string $wp_version
  */
-$wp_version = '6.9-alpha-60615';
+$wp_version = '6.9-alpha-60616';
 
 /**
  * Holds the WordPress DB revision, increments when changes are made to the WordPress DB schema.