Skip to content

gh-124130: Fix a bug in matching regular expression \B in empty string #127007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 2, 2025
Merged

gh-124130: Fix a bug in matching regular expression \B in empty string #127007

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
592a769
gh-124130: Fix a bug in matching regular expression \B in empty string
serhiy-storchaka Nov 19, 2024
bbace33
Update docs.
serhiy-storchaka Nov 22, 2024
4c79573
Merge branch 'main' into re-non-boundary
serhiy-storchaka Nov 22, 2024
ef0a6fc
Merge branch 'main' into re-non-boundary
serhiy-storchaka Jan 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions Doc/library/re.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -572,11 +572,8 @@ character ``'$'``.
Word boundaries are determined by the current locale
if the :py:const:`~re.LOCALE` flag is used.

.. note::

Note that ``\B`` does not match an empty string, which differs from
RE implementations in other programming languages such as Perl.
This behavior is kept for compatibility reasons.
.. versionchanged:: 3.14
``\B`` now matches the whole empty string.

Copy link

@wjssz wjssz Nov 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is LGTM.

Some users may not be sure what "the whole empty string" is.
I suggest explaining in more detail, and no need to use "whole" for "empty string".

\B used to be unable to match empty string. Now it can match, this behavior is consistent with mainstream languages.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know why you use "whole", the \B description:

\B Matches the empty string, but only when it ...

You want to distinguish from "empty string" here.
Let me think about how to describe it more clearly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is still difficult to avoid contradiction with "\B Matches the empty string, but only when it ...".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, "the empty string" means two different things there. The docs you're removing don't do a very good job either. Should be disambiguated somehow, for example

\B now matches if the input string is empty.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about this:

\B used to be unable to match 0-length string ""/b"". Now it can match, this behavior is consistent with mainstream languages.

Copy link

@wjssz wjssz Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\B now matches if the input string is empty.

@Alcaro uses the word "input", it looks fine. Candidate:

\B now matches empty input string ""/b"", this behavior is consistent with RE implementations in mainstream programming languages.

The previous test: #124130 (comment)
After that, I tested ASCII mode for "a", Unicode mode for "ю".
The results are same except in Perl v5.40.0.

Tested with: regex module, openjdk, .net, rust, ruby, php, pcre2.
javascript/golang only support ASCII word (for \w\W\b\B), even in Unicode mode.

It seems Perl v5.40.0 has bug in (\B + Unicode_mode).
Given this, it's not recommended to mention Perl in doc.

Save it to file perl.pl, and run: perl perl.pl

Click to see Perl script 
print "Perl version: $]\n\n";

# test \b in ASCII mode
my $n = () = ( "" =~ /\b/g );
print "\\b \"\"  matches: $n\n";

my $n = () = ( "a" =~ /\b/g );
print "\\b \"a\" matches: $n\n";

my $n = () = ( "=" =~ /\b/g );
print "\\b \"=\" matches: $n\n";

print "~~~~~~~~~~~~\n";

# test \B in ASCII mode
my $n = () = ( "" =~ /\B/g );
print "\\B \"\"  matches: $n\n";

my $n = () = ( "a" =~ /\B/g );
print "\\B \"a\" matches: $n\n";

my $n = () = ( "=" =~ /\B/g );
print "\\B \"=\" matches: $n\n";

print "\nxxxx  ASCII mode above / Unicode mode below xxxx\n\n";

# test \b in Unicode mode
my $n = () = ( "" =~ /\b/gu );
print "\\b \"\"   matches: $n\n";

my $n = () = ( "ю" =~ /\b/gu );
print "\\b \"ю\" matches: $n\n";

my $n = () = ( "=" =~ /\b/gu );
print "\\b \"=\"  matches: $n\n";

print "~~~~~~~~~~~~\n";

# test \B in Unicode mode
my $n = () = ( "" =~ /\B/gu );
print "\\B \"\"   matches: $n\n";

my $n = () = ( "ю" =~ /\B/gu );
print "\\B \"ю\" matches: $n  <- other REs get 0, it seems a bug.\n";

my $n = () = ( "=" =~ /\B/gu );
print "\\B \"=\"  matches: $n\n";

# explanation
print "\n(\\B + Unicode_mode) behaves inconsistently at the begin/end of a string:\n";

print "\n/^\\B/gu NOT IN \"ю\":\n";
my $n = () = ( "ю" =~ /^\B/gu );
print "matches: $n\n";

print "\n/\\B\$/gu IN \"ю\":\n";
my $n = () = ( "ю" =~ /\B$/gu );
print "matches: $n\n";

Script output:

Perl version: 5.040000

\b ""  matches: 0
\b "a" matches: 2
\b "=" matches: 0
~~~~~~~~~~~~
\B ""  matches: 1
\B "a" matches: 0
\B "=" matches: 2

xxxx  ASCII mode above / Unicode mode below xxxx

\b ""   matches: 0
\b "ю"  matches: 2
\b "="  matches: 0
~~~~~~~~~~~~
\B ""   matches: 1
\B "ю"  matches: 1  <- other REs get 0, it seems a bug.
\B "="  matches: 2

(\B + Unicode_mode) behaves inconsistently at the begin/end of a string:

/^\B/gu NOT IN "ю":
matches: 0

/\B$/gu IN "ю":
matches: 1

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, Perl doesn't have (\B+Unicode_mode) bug.
If add a line use utf8; at the beginning of Perl script, it works as expected.
ref: https://perldoc.perl.org/utf8

.. index:: single: \d; in regular expressions

Expand Down
13 changes: 5 additions & 8 deletions Lib/test/test_re.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -978,18 +978,15 @@ def test_word_boundaries(self):
self.assertIsNone(re.fullmatch(br".+\B", b"abc", re.LOCALE))
self.assertIsNone(re.fullmatch(r".+\B", "ьюя"))
self.assertTrue(re.fullmatch(r".+\B", "ьюя", re.ASCII))
# However, an empty string contains no word boundaries, and also no
# non-boundaries.
# However, an empty string contains no word boundaries.
self.assertIsNone(re.search(r"\b", ""))
self.assertIsNone(re.search(r"\b", "", re.ASCII))
self.assertIsNone(re.search(br"\b", b""))
self.assertIsNone(re.search(br"\b", b"", re.LOCALE))
# This one is questionable and different from the perlre behaviour,
# but describes current behavior.
self.assertIsNone(re.search(r"\B", ""))
self.assertIsNone(re.search(r"\B", "", re.ASCII))
self.assertIsNone(re.search(br"\B", b""))
self.assertIsNone(re.search(br"\B", b"", re.LOCALE))
self.assertTrue(re.search(r"\B", ""))
self.assertTrue(re.search(r"\B", "", re.ASCII))
self.assertTrue(re.search(br"\B", b""))
self.assertTrue(re.search(br"\B", b"", re.LOCALE))
# A single word-character string has two boundaries, but no
# non-boundary gaps.
self.assertEqual(len(re.findall(r"\b", "a")), 2)
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2024-11-19-10-46-57.gh-issue-124130.OZ_vR5.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Fix a bug in matching regular expression ``\B`` in empty string. Now it
is always the opposite of ``\b``.
12 changes: 0 additions & 12 deletions Modules/_sre/sre_lib.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,53 +42,41 @@ SRE(at)(SRE_STATE* state, const SRE_CHAR* ptr, SRE_CODE at)
return ((void*) ptr == state->end);

case SRE_AT_BOUNDARY:
if (state->beginning == state->end)
return 0;
thatp = ((void*) ptr > state->beginning) ?
SRE_IS_WORD((int) ptr[-1]) : 0;
thisp = ((void*) ptr < state->end) ?
SRE_IS_WORD((int) ptr[0]) : 0;
return thisp != thatp;

case SRE_AT_NON_BOUNDARY:
if (state->beginning == state->end)
return 0;
thatp = ((void*) ptr > state->beginning) ?
SRE_IS_WORD((int) ptr[-1]) : 0;
thisp = ((void*) ptr < state->end) ?
SRE_IS_WORD((int) ptr[0]) : 0;
return thisp == thatp;

case SRE_AT_LOC_BOUNDARY:
if (state->beginning == state->end)
return 0;
thatp = ((void*) ptr > state->beginning) ?
SRE_LOC_IS_WORD((int) ptr[-1]) : 0;
thisp = ((void*) ptr < state->end) ?
SRE_LOC_IS_WORD((int) ptr[0]) : 0;
return thisp != thatp;

case SRE_AT_LOC_NON_BOUNDARY:
if (state->beginning == state->end)
return 0;
thatp = ((void*) ptr > state->beginning) ?
SRE_LOC_IS_WORD((int) ptr[-1]) : 0;
thisp = ((void*) ptr < state->end) ?
SRE_LOC_IS_WORD((int) ptr[0]) : 0;
return thisp == thatp;

case SRE_AT_UNI_BOUNDARY:
if (state->beginning == state->end)
return 0;
thatp = ((void*) ptr > state->beginning) ?
SRE_UNI_IS_WORD((int) ptr[-1]) : 0;
thisp = ((void*) ptr < state->end) ?
SRE_UNI_IS_WORD((int) ptr[0]) : 0;
return thisp != thatp;

case SRE_AT_UNI_NON_BOUNDARY:
if (state->beginning == state->end)
return 0;
thatp = ((void*) ptr > state->beginning) ?
SRE_UNI_IS_WORD((int) ptr[-1]) : 0;
thisp = ((void*) ptr < state->end) ?
Expand Down
Loading