Skip to content

Commit b67e358

Browse files
alexdowadalexdowad
committed
JIS7/JIS8 encoding: handle invalid 2nd byte for Kanji correctly
Previously, in ISO-2022-JP/JIS7/JIS8, if an escape sequence (starting with 0x1B) appeared where the 2nd byte of a multibyte character should have been, mbstring would forget all about the truncated multibyte character and happily accept the escape sequence. However, such sequences are not legal and should be flagged as errors. Also, any other illegal bytes appearing where the 2nd byte of a multibyte character was expected were just passed through quietly to the output. Fix that. Also add a test suite for both ISO-2022-JP and JIS7/JIS8. (These are extremely similar encodings; JIS7 and JIS8 are variants of ISO-2022-JP. mbstring's 'JIS' is actually a combination of JIS7 _and_ JIS8, since the extensions which each one adds to ISO-2022-JP are disjoint.)
1 parent d497c0e commit b67e358

File tree

4 files changed

+6527
-4
lines changed

4 files changed

+6527
-4
lines changed

‎ext/mbstring/libmbfl/filters/mbfilter_jis.c

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -170,10 +170,6 @@ mbfl_filt_conv_jis_wchar(int c, mbfl_convert_filter *filter)
170170
}
171171
}
172172
CK((*filter->output_function)(w, filter->data));
173-
} else if (c == 0x1b) {
174-
filter->status += 2;
175-
} else if ((c >= 0 && c < 0x21) || c == 0x7f) { /* CTLs */
176-
CK((*filter->output_function)(c, filter->data));
177173
} else {
178174
w = (c1 << 8) | c;
179175
w &= MBFL_WCSGROUP_MASK;

‎ext/mbstring/tests/data/JISX0201.txt

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
#
2+
# Name: JIS X 0201 (1976) to Unicode 1.1 Table
3+
# Unicode version: 1.1
4+
# Table version: 0.9
5+
# Table format: Format A
6+
# Date: 8 March 1994
7+
# Authors: Glenn Adams <[email protected]>
8+
# John H. Jenkins <[email protected]>
9+
#
10+
# Copyright (c) 1991-1994 Unicode, Inc. All Rights reserved.
11+
#
12+
# This file is provided as-is by Unicode, Inc. (The Unicode Consortium).
13+
# No claims are made as to fitness for any particular purpose. No
14+
# warranties of any kind are expressed or implied. The recipient
15+
# agrees to determine applicability of information provided. If this
16+
# file has been provided on magnetic media by Unicode, Inc., the sole
17+
# remedy for any claim will be exchange of defective media within 90
18+
# days of receipt.
19+
#
20+
# Recipient is granted the right to make copies in any form for
21+
# internal distribution and to freely use the information supplied
22+
# in the creation of products supporting Unicode. Unicode, Inc.
23+
# specifically excludes the right to re-distribute this file directly
24+
# to third parties or other organizations whether for profit or not.
25+
#
26+
# General notes:
27+
#
28+
# This table contains the data the Unicode Consortium has on how
29+
# single-byte JIS X 0201 characters map into Unicode 1.1
30+
# (ISO/IEC 10646:1-1993 UCS-2).
31+
#
32+
# Format: Three tab-separated columns
33+
# Column #1 is the shift JIS code (in hex as 0xXX)
34+
# Column #2 is the Unicode (in hex as 0xXXXX)
35+
# Column #3 the Unicode (ISO 10646) name (follows a comment sign)
36+
#
37+
# The entries are in JIS order
38+
#
39+
# These mappings are provisional, pending definition of
40+
# official mappings by Japanese standards bodies.
41+
#
42+
# Any comments or problems, contact <[email protected]>
43+
#
44+
#
45+
0x20 0x0020 # SPACE
46+
0x21 0x0021 # EXCLAMATION MARK
47+
0x22 0x0022 # QUOTATION MARK
48+
0x23 0x0023 # NUMBER SIGN
49+
0x24 0x0024 # DOLLAR SIGN
50+
0x25 0x0025 # PERCENT SIGN
51+
0x26 0x0026 # AMPERSAND
52+
0x27 0x0027 # APOSTROPHE
53+
0x28 0x0028 # LEFT PARENTHESIS
54+
0x29 0x0029 # RIGHT PARENTHESIS
55+
0x2A 0x002A # ASTERISK
56+
0x2B 0x002B # PLUS SIGN
57+
0x2C 0x002C # COMMA
58+
0x2D 0x002D # HYPHEN-MINUS
59+
0x2E 0x002E # FULL STOP
60+
0x2F 0x002F # SOLIDUS
61+
0x30 0x0030 # DIGIT ZERO
62+
0x31 0x0031 # DIGIT ONE
63+
0x32 0x0032 # DIGIT TWO
64+
0x33 0x0033 # DIGIT THREE
65+
0x34 0x0034 # DIGIT FOUR
66+
0x35 0x0035 # DIGIT FIVE
67+
0x36 0x0036 # DIGIT SIX
68+
0x37 0x0037 # DIGIT SEVEN
69+
0x38 0x0038 # DIGIT EIGHT
70+
0x39 0x0039 # DIGIT NINE
71+
0x3A 0x003A # COLON
72+
0x3B 0x003B # SEMICOLON
73+
0x3C 0x003C # LESS-THAN SIGN
74+
0x3D 0x003D # EQUALS SIGN
75+
0x3E 0x003E # GREATER-THAN SIGN
76+
0x3F 0x003F # QUESTION MARK
77+
0x40 0x0040 # COMMERCIAL AT
78+
0x41 0x0041 # LATIN CAPITAL LETTER A
79+
0x42 0x0042 # LATIN CAPITAL LETTER B
80+
0x43 0x0043 # LATIN CAPITAL LETTER C
81+
0x44 0x0044 # LATIN CAPITAL LETTER D
82+
0x45 0x0045 # LATIN CAPITAL LETTER E
83+
0x46 0x0046 # LATIN CAPITAL LETTER F
84+
0x47 0x0047 # LATIN CAPITAL LETTER G
85+
0x48 0x0048 # LATIN CAPITAL LETTER H
86+
0x49 0x0049 # LATIN CAPITAL LETTER I
87+
0x4A 0x004A # LATIN CAPITAL LETTER J
88+
0x4B 0x004B # LATIN CAPITAL LETTER K
89+
0x4C 0x004C # LATIN CAPITAL LETTER L
90+
0x4D 0x004D # LATIN CAPITAL LETTER M
91+
0x4E 0x004E # LATIN CAPITAL LETTER N
92+
0x4F 0x004F # LATIN CAPITAL LETTER O
93+
0x50 0x0050 # LATIN CAPITAL LETTER P
94+
0x51 0x0051 # LATIN CAPITAL LETTER Q
95+
0x52 0x0052 # LATIN CAPITAL LETTER R
96+
0x53 0x0053 # LATIN CAPITAL LETTER S
97+
0x54 0x0054 # LATIN CAPITAL LETTER T
98+
0x55 0x0055 # LATIN CAPITAL LETTER U
99+
0x56 0x0056 # LATIN CAPITAL LETTER V
100+
0x57 0x0057 # LATIN CAPITAL LETTER W
101+
0x58 0x0058 # LATIN CAPITAL LETTER X
102+
0x59 0x0059 # LATIN CAPITAL LETTER Y
103+
0x5A 0x005A # LATIN CAPITAL LETTER Z
104+
0x5B 0x005B # LEFT SQUARE BRACKET
105+
0x5C 0x00A5 # YEN SIGN
106+
0x5D 0x005D # RIGHT SQUARE BRACKET
107+
0x5E 0x005E # CIRCUMFLEX ACCENT
108+
0x5F 0x005F # LOW LINE
109+
0x60 0x0060 # GRAVE ACCENT
110+
0x61 0x0061 # LATIN SMALL LETTER A
111+
0x62 0x0062 # LATIN SMALL LETTER B
112+
0x63 0x0063 # LATIN SMALL LETTER C
113+
0x64 0x0064 # LATIN SMALL LETTER D
114+
0x65 0x0065 # LATIN SMALL LETTER E
115+
0x66 0x0066 # LATIN SMALL LETTER F
116+
0x67 0x0067 # LATIN SMALL LETTER G
117+
0x68 0x0068 # LATIN SMALL LETTER H
118+
0x69 0x0069 # LATIN SMALL LETTER I
119+
0x6A 0x006A # LATIN SMALL LETTER J
120+
0x6B 0x006B # LATIN SMALL LETTER K
121+
0x6C 0x006C # LATIN SMALL LETTER L
122+
0x6D 0x006D # LATIN SMALL LETTER M
123+
0x6E 0x006E # LATIN SMALL LETTER N
124+
0x6F 0x006F # LATIN SMALL LETTER O
125+
0x70 0x0070 # LATIN SMALL LETTER P
126+
0x71 0x0071 # LATIN SMALL LETTER Q
127+
0x72 0x0072 # LATIN SMALL LETTER R
128+
0x73 0x0073 # LATIN SMALL LETTER S
129+
0x74 0x0074 # LATIN SMALL LETTER T
130+
0x75 0x0075 # LATIN SMALL LETTER U
131+
0x76 0x0076 # LATIN SMALL LETTER V
132+
0x77 0x0077 # LATIN SMALL LETTER W
133+
0x78 0x0078 # LATIN SMALL LETTER X
134+
0x79 0x0079 # LATIN SMALL LETTER Y
135+
0x7A 0x007A # LATIN SMALL LETTER Z
136+
0x7B 0x007B # LEFT CURLY BRACKET
137+
0x7C 0x007C # VERTICAL LINE
138+
0x7D 0x007D # RIGHT CURLY BRACKET
139+
0x7E 0x203E # OVERLINE
140+
0xA1 0xFF61 # HALFWIDTH IDEOGRAPHIC FULL STOP
141+
0xA2 0xFF62 # HALFWIDTH LEFT CORNER BRACKET
142+
0xA3 0xFF63 # HALFWIDTH RIGHT CORNER BRACKET
143+
0xA4 0xFF64 # HALFWIDTH IDEOGRAPHIC COMMA
144+
0xA5 0xFF65 # HALFWIDTH KATAKANA MIDDLE DOT
145+
0xA6 0xFF66 # HALFWIDTH KATAKANA LETTER WO
146+
0xA7 0xFF67 # HALFWIDTH KATAKANA LETTER SMALL A
147+
0xA8 0xFF68 # HALFWIDTH KATAKANA LETTER SMALL I
148+
0xA9 0xFF69 # HALFWIDTH KATAKANA LETTER SMALL U
149+
0xAA 0xFF6A # HALFWIDTH KATAKANA LETTER SMALL E
150+
0xAB 0xFF6B # HALFWIDTH KATAKANA LETTER SMALL O
151+
0xAC 0xFF6C # HALFWIDTH KATAKANA LETTER SMALL YA
152+
0xAD 0xFF6D # HALFWIDTH KATAKANA LETTER SMALL YU
153+
0xAE 0xFF6E # HALFWIDTH KATAKANA LETTER SMALL YO
154+
0xAF 0xFF6F # HALFWIDTH KATAKANA LETTER SMALL TU
155+
0xB0 0xFF70 # HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK
156+
0xB1 0xFF71 # HALFWIDTH KATAKANA LETTER A
157+
0xB2 0xFF72 # HALFWIDTH KATAKANA LETTER I
158+
0xB3 0xFF73 # HALFWIDTH KATAKANA LETTER U
159+
0xB4 0xFF74 # HALFWIDTH KATAKANA LETTER E
160+
0xB5 0xFF75 # HALFWIDTH KATAKANA LETTER O
161+
0xB6 0xFF76 # HALFWIDTH KATAKANA LETTER KA
162+
0xB7 0xFF77 # HALFWIDTH KATAKANA LETTER KI
163+
0xB8 0xFF78 # HALFWIDTH KATAKANA LETTER KU
164+
0xB9 0xFF79 # HALFWIDTH KATAKANA LETTER KE
165+
0xBA 0xFF7A # HALFWIDTH KATAKANA LETTER KO
166+
0xBB 0xFF7B # HALFWIDTH KATAKANA LETTER SA
167+
0xBC 0xFF7C # HALFWIDTH KATAKANA LETTER SI
168+
0xBD 0xFF7D # HALFWIDTH KATAKANA LETTER SU
169+
0xBE 0xFF7E # HALFWIDTH KATAKANA LETTER SE
170+
0xBF 0xFF7F # HALFWIDTH KATAKANA LETTER SO
171+
0xC0 0xFF80 # HALFWIDTH KATAKANA LETTER TA
172+
0xC1 0xFF81 # HALFWIDTH KATAKANA LETTER TI
173+
0xC2 0xFF82 # HALFWIDTH KATAKANA LETTER TU
174+
0xC3 0xFF83 # HALFWIDTH KATAKANA LETTER TE
175+
0xC4 0xFF84 # HALFWIDTH KATAKANA LETTER TO
176+
0xC5 0xFF85 # HALFWIDTH KATAKANA LETTER NA
177+
0xC6 0xFF86 # HALFWIDTH KATAKANA LETTER NI
178+
0xC7 0xFF87 # HALFWIDTH KATAKANA LETTER NU
179+
0xC8 0xFF88 # HALFWIDTH KATAKANA LETTER NE
180+
0xC9 0xFF89 # HALFWIDTH KATAKANA LETTER NO
181+
0xCA 0xFF8A # HALFWIDTH KATAKANA LETTER HA
182+
0xCB 0xFF8B # HALFWIDTH KATAKANA LETTER HI
183+
0xCC 0xFF8C # HALFWIDTH KATAKANA LETTER HU
184+
0xCD 0xFF8D # HALFWIDTH KATAKANA LETTER HE
185+
0xCE 0xFF8E # HALFWIDTH KATAKANA LETTER HO
186+
0xCF 0xFF8F # HALFWIDTH KATAKANA LETTER MA
187+
0xD0 0xFF90 # HALFWIDTH KATAKANA LETTER MI
188+
0xD1 0xFF91 # HALFWIDTH KATAKANA LETTER MU
189+
0xD2 0xFF92 # HALFWIDTH KATAKANA LETTER ME
190+
0xD3 0xFF93 # HALFWIDTH KATAKANA LETTER MO
191+
0xD4 0xFF94 # HALFWIDTH KATAKANA LETTER YA
192+
0xD5 0xFF95 # HALFWIDTH KATAKANA LETTER YU
193+
0xD6 0xFF96 # HALFWIDTH KATAKANA LETTER YO
194+
0xD7 0xFF97 # HALFWIDTH KATAKANA LETTER RA
195+
0xD8 0xFF98 # HALFWIDTH KATAKANA LETTER RI
196+
0xD9 0xFF99 # HALFWIDTH KATAKANA LETTER RU
197+
0xDA 0xFF9A # HALFWIDTH KATAKANA LETTER RE
198+
0xDB 0xFF9B # HALFWIDTH KATAKANA LETTER RO
199+
0xDC 0xFF9C # HALFWIDTH KATAKANA LETTER WA
200+
0xDD 0xFF9D # HALFWIDTH KATAKANA LETTER N
201+
0xDE 0xFF9E # HALFWIDTH KATAKANA VOICED SOUND MARK
202+
0xDF 0xFF9F # HALFWIDTH KATAKANA SEMI-VOICED SOUND MARK

0 commit comments

Comments
 (0)