Skip to content

Commit c2884ae

Browse files
authored
Update PlainFullTokenizer.lex
include \p{Letter} in the {Identifier} initial character match
1 parent c01728f commit c2884ae

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/org/opensolaris/opengrok/analysis/plain/PlainFullTokenizer.lex

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ super(in);
4141
%char
4242

4343
//WhiteSpace = [ \t\f\r]+|\n
44-
Identifier = [a-zA-Z_] [a-zA-Z0-9_]*
44+
Identifier = [a-zA-Z\p{Letter}_] [a-zA-Z\p{Letter}0-9\p{Number}_]*
4545
Number = [0-9]+|[0-9]+\.[0-9]+| "0[xX]" [0-9a-fA-F]+
4646
Printable = [\@\$\%\^\&\-+=\?\.\:]
4747

0 commit comments

Comments
 (0)