Consider decoding most frequent non-ASCII HTML entities

[forthrin]

It seems CGI decidedly decodes only ASCII entities.

Are you open to decoding a handful of the most frequent non-ASCII ones, in particular   and possibly a few others?

[duerst]

I wouldn't want to decode  , because that makes it very difficult to distinguish it from a simple space. Decoding some of the entities that produce characters that can be more easily distinguished should be fine. Regards, Martin.

…

On 2026-01-05 16:36, forthrin wrote: forthrin created an issue (ruby/cgi#104) It seems CGI decidedly decodes only ASCII entities. Are you open to decoding a handful of the most frequently used non-ASCII entities, in particular ` ` and possibly a few others?

[forthrin]

Thanks for your interest. Based on some heuristic tests on a semi-large HTML cache it seems that selected characters from the quarter-blocks U+0080 and U+2000 are the most prevalent. U+00C0 is also a candidate due to major use of Latin 1.

1. What would be your criteria for inclusion, except from frequency?

2. Are you adamant to pass   through verbatim to force programmers to deal with it explicitly in all cases? Elaborate.

[duerst]

This issue does not say where and for what purpose the escaping is supposed to take place. Is it showing things to a human viewer (what I assumed)? Further processing including markup (that would mean & and friends can't be unescaped)? Further processing not including markup (in which case even & could be unescaped)? The statistics below is definitely skewed towards Nordic languages, in particular Norwegian, but I don't think we should include/exclude entities based on frequency. The fullest list is at https://html.spec.whatwg.org/multipage/named-characters.html. P.S.: Your frequency table doesn't show up in the github issue.

…

On 2026-01-06 18:39, forthrin wrote: forthrin left a comment (ruby/cgi#104) Thanks for your interest. See enclosed a heuristic test done on a collection of HTML files to check for prevalence. This may obviously be skewed for the data set, but should at least give some idea as to which entities are worth converting. As you can see, all entities were either Latin 1 or General Punctuation, which is a pragmatic constraint. 1. What would be your criteria for inclusion, except from frequency? 2. Are you adamant to pass `&nbsp;` through verbatim to force programmers to deal with it explicitly in all cases? Elaborate. | **`FREQ`** | **`ASCII`** | **`ENTITY`** | **`CGI`** | |-----------:|-------------|--------------|-----------| | `648019` | ``U+0026`` | `&amp;` | `&` | | `417737` | ``U+0022`` | `&quot;` | `"` | | `13070` | ``U+003C`` | `&lt;` | `<` | | `13061` | ``U+003E`` | `&gt;` | `>` | | `1030` | ``U+0027`` | `&apos;` | `'` | | | **`LATIN`** | | | | `176311` | ``U+00A0`` | `&nbsp;` | | | `20192` | ``U+00A9`` | `&copy;` | | | `4336` | ``U+00E5`` | `&aring;` | | | `3659` | ``U+00F8`` | `&oslash;` | | | `173` | ``U+00E6`` | `&aelig;` | | | `143` | ``U+00BB`` | `&raquo;` | | | `132` | ``U+00AB`` | `&laquo;` | | | `67` | ``U+00E9`` | `&eacute;` | | | `54` | ``U+00B0`` | `&deg;` | | | `40` | ``U+00E4`` | `&auml;` | | | `17` | ``U+00D7`` | `&times;` | | | `15` | ``U+00E1`` | `&aacute;` | | | `12` | ``U+00BF`` | `&iquest;` | | | `10` | ``U+00F1`` | `&ntilde;` | | | `5` | ``U+00E8`` | `&egrave;` | | | `4` | ``U+00F2`` | `&ograve;` | | | `4` | ``U+00A6`` | `&brvbar;` | | | `3` | ``U+00AE`` | `&reg;` | | | `2` | ``U+00FA`` | `&uacute;` | | | `2` | ``U+00E7`` | `&ccedil;` | | | `1` | ``U+00FC`` | `&uuml;` | | | `1` | ``U+00F6`` | `&ouml;` | | | | **`PUNCT`** | | | | `5144` | ``U+2019`` | `&rsquo;` | | | `777` | ``U+2013`` | `&ndash;` | | | `743` | ``U+2022`` | `&bull;` | | | `222` | ``U+2026`` | `&hellip;` | | | `192` | ``U+20AC`` | `&euro;` | | | `113` | ``U+203A`` | `&rsaquo;` | | | `113` | ``U+2039`` | `&lsaquo;` | | | `37` | ``U+2014`` | `&mdash;` | | | `14` | ``U+2020`` | `&dagger;` | | | `3` | ``U+2122`` | `&trade;` | | | `1` | ``U+2030`` | `&permil;` | |

[forthrin]

Your two last question escape me. The first is timely. Except for <>&, entities were made pending Unicode. Notice however:

1. A few are still vastly frequent:

• Typography (quotes, dashes, footnotes, bullets)
• Business (copyright, trademark, currencies)
• Science (permil, degrees)

2. They prevail with good reason: They are frequently needed and notoriously difficult to enter on keyboards.

By supporting the clearly prevalent ones, libraries supporting the full W3 set (see below) can be avoided, when needing to prepare HTML text for human (or machine) consumption, where one obviously never wants the entities.

Do you think we can land a sensible selection of characters limited to a few Unicode blocks? Any possible issues?

[forthrin]

Range	Chars
Basic Latin (32) 0009-007D	!"#$%&'()*+,./ :;<=>? @ [\]^_ {} U+0009 U+000A U+0060 U+007C
Latin-1 Supplement (96) 00A0-00FF	¡¢£¤¥¦§¨©ª«¬­®¯ °±²³´µ¶·¸¹º»¼½¾¿ ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏ ÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß àáâãäåæçèéêëìíîï ðñòóôõö÷øùúûüýþÿ U+00A0
Latin Extended-A (120) 0100-017E	ĀāĂ㥹ĆćĈĉĊċČčĎď ĐđĒēĖėĘęĚěĜĝĞğ ĠġĢĤĥĦħĨĩĪīĮį İıIJijĴĵĶķĸĹĺĻļĽľĿ ŀŁłŃńŅņŇňʼnŊŋŌō ŐőŒœŔŕŖŗŘřŚśŜŝŞş ŠšŢţŤťŦŧŨũŪūŬŭŮů ŰűŲųŴŵŶŷŸŹźŻżŽž
Latin Extended-B (4) 0192-0237	ƒ Ƶ ǵ ȷ
Spacing Modifier Letters (8) 02C6-02DD	ˆˇ ˘˙˚˛˜˝
Combining Diacritical Marks (1) 0311-0311	̑
Greek and Coptic (59) 0391-03F6	ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟ ΠΡΣΤΥΦΧΨΩ αβγδεζηθικλμνξο πρςστυφχψω ϑϒϕϖϜϝ ϰϱϵ϶
Cyrillic (92) 0401-045F	ЁЂЃЄЅІЇЈЉЊЋЌЎЏ АБВГДЕЖЗИЙКЛМНОП РСТУФХЦЧШЩЪЫЬЭЮЯ абвгдежзийклмноп рстуфхцчшщъыьэюя ёђѓєѕіїјљњћќўџ
General Punctuation (48) 2002-2063	‐–—―‖‘’‚“”„ †‡•‥… ‰‱′″‴‵‹›‾ ⁁⁃⁄⁏ ⁗ U+2002 U+2003 U+2004 U+2005 U+2007 U+2008 U+2009 U+200A U+200B U+200C U+200D U+200E U+200F U+205F U+2060 U+2061 U+2062 U+2063
Currency Symbols (1) 20AC-20AC	€
Combining Diacritical Marks for Symbols (2) 20DB-20DC	⃛⃜
Letterlike Symbols (42) 2102-2148	ℂ℅ℊℋℌℍℎℏ ℐℑℒℓℕ№℗℘ℙℚℛℜℝ℞ ™ℤ℧ℨ℩ℬℭℯ ℰℱℳℴℵℶℷℸ ⅅⅆⅇⅈ
Number Forms (12) 2153-215E	⅓⅔⅕⅖⅗⅘⅙⅚⅛⅜⅝⅞
Arrows (77) 2190-21FF	←↑→↓↔↕↖↗↘↙↚↛↝↞↟ ↠↡↢↣↤↥↦↧↩↪↫↬↭↮ ↰↱↲↳↵↶↷↺↻↼↽↾↿ ⇀⇁⇂⇃⇄⇅⇆⇇⇈⇉⇊⇋⇌⇍⇎⇏ ⇐⇑⇒⇓⇔⇕⇖⇗⇘⇙⇚⇛⇝ ⇤⇥ ⇵⇽⇾⇿
Mathematical Operators (231) 2200-22FE	∀∁∂∃∄∅∇∈∉∋∌∏ ∐∑−∓∔∖∗∘√∝∞∟ ∠∡∢∣∤∥∦∧∨∩∪∫∬∭∮∯ ∰∱∲∳∴∵∶∷∸∺∻∼∽∾∿ ≀≁≂≃≄≅≆≇≈≉≊≋≌≍≎≏ ≐≑≒≓≔≕≖≗≙≚≜≟ ≠≡≢≤≥≦≧≨≩≪≫≬≭≮≯ ≰≱≲≳≴≵≶≷≸≹≺≻≼≽≾≿ ⊀⊁⊂⊃⊄⊅⊆⊇⊈⊉⊊⊋⊍⊎⊏ ⊐⊑⊒⊓⊔⊕⊖⊗⊘⊙⊚⊛⊝⊞⊟ ⊠⊡⊢⊣⊤⊥⊧⊨⊩⊪⊫⊬⊭⊮⊯ ⊰⊲⊳⊴⊵⊶⊷⊸⊹⊺⊻⊽⊾⊿ ⋀⋁⋂⋃⋄⋅⋆⋇⋈⋉⋊⋋⋌⋍⋎⋏ ⋐⋑⋒⋓⋔⋕⋖⋗⋘⋙⋚⋛⋞⋟ ⋠⋡⋢⋣⋦⋧⋨⋩⋪⋫⋬⋭⋮⋯ ⋰⋱⋲⋳⋴⋵⋶⋷⋹⋺⋻⋼⋽⋾
Miscellaneous Technical (38) 2305-23E7	⌅⌆⌈⌉⌊⌋⌌⌍⌎⌏ ⌐⌒⌓⌕⌖⌜⌝⌞⌟ ⌢⌣⌭⌮ ⌶⌽⌿ ⍼ ⎰⎱⎴⎵⎶ ⏜⏝⏞⏟ ⏢⏧
Control Pictures (1) 2423-2423	␣
Enclosed Alphanumerics (1) 24C8-24C8	Ⓢ
Box Drawing (40) 2500-256C	─│┌ ┐└┘├ ┤┬ ┴┼ ═║╒╓╔╕╖╗╘╙╚╛╜╝╞╟ ╠╡╢╣╤╥╦╧╨╩╪╫╬
Block Elements (6) 2580-2593	▀▄█ ░▒▓
Geometric Shapes (25) 25A1-25FC	□▪▫▭▮ ▱△▴▵▸▹▽▾▿ ◂◃◊○ ◬◯ ◸◹◺◻◼
Miscellaneous Symbols (13) 2605-266F	★☆☎ ♀♂ ♠♣♥♦♪♭♮♯
Dingbats (7) 2713-2773	✓✗ ✠ ✶ ❘ ❲❳
Miscellaneous Mathematical Symbols-A (10) 27C8-27ED	⟈⟉ ⟦⟧⟨⟩⟪⟫⟬⟭
Supplemental Arrows-A (8) 27F5-27FF	⟵⟶⟷⟸⟹⟺⟼⟿
Supplemental Arrows-B (90) 2902-297F	⤂⤃⤄⤅⤌⤍⤎⤏ ⤐⤑⤒⤓⤖⤙⤚⤛⤜⤝⤞⤟ ⤠⤣⤤⤥⤦⤧⤨⤩⤪ ⤳⤵⤶⤷⤸⤹⤼⤽ ⥅⥈⥉⥊⥋⥎⥏ ⥐⥑⥒⥓⥔⥕⥖⥗⥘⥙⥚⥛⥜⥝⥞⥟ ⥠⥡⥢⥣⥤⥥⥦⥧⥨⥩⥪⥫⥬⥭⥮⥯ ⥰⥱⥲⥳⥴⥵⥶⥸⥹⥻⥼⥽⥾⥿
Miscellaneous Mathematical Symbols-B (62) 2985-29F6	⦅⦆⦋⦌⦍⦎⦏ ⦐⦑⦒⦓⦔⦕⦖⦚⦜⦝ ⦤⦥⦦⦧⦨⦩⦪⦫⦬⦭⦮⦯ ⦰⦱⦲⦳⦴⦵⦶⦷⦹⦻⦼⦾⦿ ⧀⧁⧂⧃⧄⧅⧉⧍⧎⧏ ⧐⧜⧝⧞ ⧣⧤⧥⧫ ⧴⧶
Supplemental Mathematical Operators (182) 2A00-2AFD	⨀⨁⨂⨄⨆⨌⨍ ⨐⨑⨒⨓⨔⨕⨖⨗ ⨢⨣⨤⨥⨦⨧⨩⨪⨭⨮⨯ ⨰⨱⨳⨴⨵⨶⨷⨸⨹⨺⨻⨼⨿ ⩀⩂⩃⩄⩅⩆⩇⩈⩉⩊⩋⩌⩍ ⩐⩓⩔⩕⩖⩗⩘⩚⩛⩜⩝⩟ ⩦⩪⩭⩮⩯ ⩰⩱⩲⩳⩴⩵⩷⩸⩹⩺⩻⩼⩽⩾⩿ ⪀⪁⪂⪃⪄⪅⪆⪇⪈⪉⪊⪋⪌⪍⪎⪏ ⪐⪑⪒⪓⪔⪕⪖⪗⪘⪙⪚⪝⪞⪟ ⪠⪡⪢⪤⪥⪦⪧⪨⪩⪪⪫⪬⪭⪮⪯ ⪰⪳⪴⪵⪶⪷⪸⪹⪺⪻⪼⪽⪾⪿ ⫀⫁⫂⫃⫄⫅⫆⫇⫈⫋⫌⫏ ⫐⫑⫒⫓⫔⫕⫖⫗⫘⫙⫚⫛ ⫤⫦⫧⫨⫩⫫⫬⫭⫮⫯ ⫰⫱⫲⫳⫽
Alphabetic Presentation Forms (5) FB00-FB04	fffiflffiffl
Mathematical Alphanumeric Symbols (133) 1D49C-1D56B	𝒜𝒞𝒟 𝒢𝒥𝒦𝒩𝒪𝒫𝒬𝒮𝒯 𝒰𝒱𝒲𝒳𝒴𝒵𝒶𝒷𝒸𝒹𝒻𝒽𝒾𝒿 𝓀𝓁𝓂𝓃𝓅𝓆𝓇𝓈𝓉𝓊𝓋𝓌𝓍𝓎𝓏 𝔄𝔅𝔇𝔈𝔉𝔊𝔍𝔎𝔏 𝔐𝔑𝔒𝔓𝔔𝔖𝔗𝔘𝔙𝔚𝔛𝔜𝔞𝔟 𝔠𝔡𝔢𝔣𝔤𝔥𝔦𝔧𝔨𝔩𝔪𝔫𝔬𝔭𝔮𝔯 𝔰𝔱𝔲𝔳𝔴𝔵𝔶𝔷𝔸𝔹𝔻𝔼𝔽𝔾 𝕀𝕁𝕂𝕃𝕄𝕆𝕊𝕋𝕌𝕍𝕎𝕏 𝕐𝕒𝕓𝕔𝕕𝕖𝕗𝕘𝕙𝕚𝕛𝕜𝕝𝕞𝕟 𝕠𝕡𝕢𝕣𝕤𝕥𝕦𝕧𝕨𝕩𝕪𝕫