Assertion failure in Perl_yylex

[p5pRT]

Migrated from rt.perl.org#133872 (status was 'open')

Searchable as RT133872$

[p5pRT]

From @dur-randir

Created by @dur-randir

While fuzzing perl v5.29.8-21-gde59f38ed9 built with afl and run
under libdislocator, I found the following program (text also attached
to the ticket)

00000000 66 6f 72 6d 61 74 3d 0a 40 0a 2e 2f 2f 0a 2e |format=.@​..//..|

to cause an assertion failure​:

perl​: toke.c​:6368​: Perl_yylex​: Assertion
`((char*)PL_scopestack_name[PL_scopestack_ix-1] ==
(char*)"lex_format") || strEQ(PL_scopestack_name[PL_scopestack_ix-1],
"lex_format")' failed.

https://rt.perl.org/Public/Bug/Display.html?id=132583 is a similar
ticket, but this test case still fails in blead, and had started with
at least 5.8.8. GDB stack trace is following​:

Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@​entry=6) at ../sysdeps/unix/sysv/linux/raise.c​:50
50 ../sysdeps/unix/sysv/linux/raise.c​: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@​entry=6) at ../sysdeps/unix/sysv/linux/raise.c​:50
#1 0x00007f02db563535 in __GI_abort () at abort.c​:79
#2 0x00007f02db56340f in __assert_fail_base (fmt=0x7f02db6c5ee0
"%s%s%s​:%u​: %s%sAssertion `%s' failed.\n%n",
  assertion=0x55ea92b99268
"((char*)PL_scopestack_name[PL_scopestack_ix-1] ==
(char*)\"lex_format\") ||
strEQ(PL_scopestack_name[PL_scopestack_ix-1], \"lex_format\")",
file=0x55ea92b8a5e5 "toke.c", line=6368, function=<optimized out>) at
assert.c​:92
#3 0x00007f02db5710f2 in __GI___assert_fail (
  assertion=0x55ea92b99268
"((char*)PL_scopestack_name[PL_scopestack_ix-1] ==
(char*)\"lex_format\") ||
strEQ(PL_scopestack_name[PL_scopestack_ix-1], \"lex_format\")",
file=0x55ea92b8a5e5 "toke.c", line=6368, function=0x55ea92bb0338
<__PRETTY_FUNCTION__.19414> "Perl_yylex") at assert.c​:101
#4 0x000055ea928972c9 in Perl_yylex () at toke.c​:6368
#5 0x000055ea928c20dc in Perl_yyparse (gramtype=258) at perly.c​:340
#6 0x000055ea928422ea in S_parse_body (env=0x0, xsinit=0x55ea927f71d8
<xs_init>) at perl.c​:2507
#7 0x000055ea928405bc in perl_parse (my_perl=0x55ea9405d260,
xsinit=0x55ea927f71d8 <xs_init>, argc=2, argv=0x7ffd848363f8, env=0x0)
at perl.c​:1798
#8 0x000055ea927f711b in main (argc=2, argv=0x7ffd848363f8,
env=0x7ffd84836410) at perlmain.c​:126
(gdb) q

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.29.9:

Configured by dur-randir at Wed Feb 27 14:51:01 MSK 2019.

Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
  Commit id: c1e47bad34ce1d9c84ed57c9b8978bcbd5a02e98
  Platform:
    osname=darwin
    osvers=13.4.0
    archname=darwin-thread-multi-2level
    uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0:
mon jan 11 18:17:34 pst 2016; root:xnu-2422.115.15~1release_x86_64
x86_64 '
    config_args='-de -Dusedevel -DDEBUGGING -Dusethreads'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3 -g'
    cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=
    so=dylib
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined
dynamic_lookup -L/usr/local/lib -fstack-protector'



@INC for perl 5.29.9:
    lib
    /usr/local/lib/perl5/site_perl/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.29.9
    /usr/local/lib/perl5/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/5.29.9


Environment for perl 5.29.9:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/dur-randir
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin
    PERLBREW_HOME=/Users/dur-randir/.perlbrew
    PERLBREW_MANPATH=/Users/dur-randir/perlbrew/perls/perl-5.22.1/man
    PERLBREW_PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin
    PERLBREW_PERL=perl-5.22.1
    PERLBREW_ROOT=/Users/dur-randir/perlbrew
    PERLBREW_SHELLRC_VERSION=0.84
    PERLBREW_VERSION=0.84
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

[p5pRT]

From @dur-randir

0090

[p5pRT]

From @wolfsage

On Fri, Mar 1, 2019 at 11​:09 AM Sergey Aleynikov (via RT)
<perlbug-followup@​perl.org> wrote​:

While fuzzing perl v5.29.8-21-gde59f38ed9 built with afl and run
under libdislocator, I found the following program (text also attached
to the ticket)

00000000 66 6f 72 6d 61 74 3d 0a 40 0a 2e 2f 2f 0a 2e |format=.@​..//..|

to cause an assertion failure​:

perl​: toke.c​:6368​: Perl_yylex​: Assertion
`((char*)PL_scopestack_name[PL_scopestack_ix-1] ==
(char*)"lex_format") || strEQ(PL_scopestack_name[PL_scopestack_ix-1],
"lex_format")' failed.

This bisects to​:

  commit b1f87de (HEAD, refs/bisect/bad)
  Author​: Zefram <zefram@​fysh.org>
  Date​: Sun Nov 5 04​:45​:04 2017 +0000

  add assertions about format lexing

Prior to that, it was a syntax error.

-- Matthew Horsfall (alh)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[jkeenan]

This bisects to​:

commit b1f87de (HEAD, refs/bisect/bad) Author​: Zefram <zefram@​fysh.org> Date​: Sun Nov 5 04​:45​:04 2017 +0000

add assertions about format lexing

For the record, that commit was v5.27.5-200-gb1f87deab8:

commit b1f87deab83933d92fc290fdecf45641100ff81d (HEAD)
Author:     Zefram <zefram@fysh.org>
AuthorDate: Sun Nov 5 04:45:04 2017 +0000
Commit:     Zefram <zefram@fysh.org>
CommitDate: Sun Nov 5 04:45:04 2017 +0000

    add assertions about format lexing
    
    [perl #132158] describes a situation in which a weird kind of unterminated
    format caused the parser to crash.  Add some assertions that catch the
    messed up parser state before it actually crashes.

[richardleach]

This is another fuzzing ticket that can be closed thanks to:

commit eb54d46f7264ff7af62c409d8a6ab984a5a34f57 (HEAD)
Author: Yves Orton <demerphq@gmail.com>
Date:   Fri Aug 26 18:26:14 2022 +0200

    Stop parsing on first syntax error.

    ...

I'll do a PR for the test script.