Calling `keys` on an undefined hash key defines it(?)

[AdamWill]

Description
See test script below.

Steps to Reproduce

#!/bin/perl

use Data::Dump 'dd';

my %options;
dd $options{group};
my %hash = %{$options{group}};
keys %hash;
dd $options{group};
keys %{$options{group}};
dd $options{group};

This prints:

undef
undef
{}

i.e. given an empty hash %options, doing keys %{$options{group}}; somehow causes $options{group} to become defined as a ref to an empty hash. But splitting that operation into one step that assigns %{$options{group}} to a hash variable and another step that calls keys on it does not have the same effect.

With use strict, perl dies on my %hash = %{$options{group}};, with the error Can't use an undefined value as a HASH reference at /tmp/test.pm line 9. But if I drop that part and reduce the script to:

#!/bin/perl

use Data::Dump 'dd';
use strict;
use warnings;

my %options;
dd $options{group};
keys %{$options{group}};
dd $options{group};

then it executes with no errors or warnings, and has the same effect.

Expected behavior
I would have expected that $options{group} would remain undefined, and possibly there would be an error or warning trying to do keys %{$options{group}};. This report is based on real code whose author clearly had the same expectation.

Perl configuration

[adamw@toolbx anime]$ perl -V
Summary of my perl5 (revision 5 version 40 subversion 2) configuration:
   
  Platform:
    osname=linux
    osvers=6.11.0
    archname=x86_64-linux-thread-multi
    uname='linux localhost 6.11.0 #1 smp preempt_dynamic 6.11.0 x86_64 gnulinux '
    config_args='-des -Doptimize=none -Dccflags=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Dldflags=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1   -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1   -Dlddlflags=-shared -Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1   -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.40.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5/5.40 -Dsitearch=/usr/local/lib64/perl5/5.40 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize -Duse64bitint'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='gcc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='  -g'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fwrapv -fno-strict-aliasing -I/usr/local/include'
    ccversion=''
    gccversion='15.0.1 20250410 (Red Hat 15.0.1-0)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='gcc'
    ldflags ='-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1  -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib
    libs=-lpthread -lresolv -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lresolv -ldl -lm -lcrypt -lutil -lc
    libc=/lib/../lib64/libc.so.6
    so=so
    useshrplib=true
    libperl=libperl.so
    gnulibc_version='2.41.9000'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 '
    cccdlflags='-fPIC'
    lddlflags='-lpthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1  -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_LONG_DOUBLE
    HAS_STRTOLD
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_HASH_FUNC_SIPHASH13
    PERL_HASH_USE_SBOX32
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_SITECUSTOMIZE
    USE_THREAD_SAFE_LOCALE
  Locally applied patches:
    Fedora Patch1: Removes date check, Fedora/RHEL specific
    Fedora Patch2: support for libdir64
    Fedora Patch3: use libresolv instead of libbind
    Fedora Patch4: USE_MM_LD_RUN_PATH
    Fedora Patch5: Provide MM::maybe_command independently (bug #1129443)
    Fedora Patch6: Dont run one io test due to random builder failures
    Fedora Patch8: Define SONAME for libperl.so
    Fedora Patch9: Install libperl.so to -Dshrpdir value
    Fedora Patch10: Make *DBM_File desctructors thread-safe (RT#61912)
    Fedora Patch11: Replace EU::MakeMaker dependency with EU::MM::Utils in IPC::Cmd (bug #1129443)
    Fedora Patch12: Link XS modules to pthread library to fix linking with -z defs
    Fedora Patch13: Pass the correct CFLAGS to dtrace
    Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux
    Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux
    Fedora Patch202: Add definition of OPTIMIZE to .ph files
  Built under linux
  Compiled at Apr 14 2025 00:00:00
  @INC:
    /usr/local/lib64/perl5/5.40
    /usr/local/share/perl5/5.40
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5

[Grinnz]

The technical explanation is that keys() applies an lvalue context, making it legal to use undef and autovivify it to a hash. You can get a similar thing to occur by using it as the list for a foreach or map operation.

[AdamWill]

ah, okay. So this should have been expected behavior, if the author was aware that keys did that?

[Grinnz]

Yeah, autovivification in lvalue context is expected (though I would argue, probably underdocumented itself), but when lvalue context happens is also not obvious so this can have various surprising ramifications like this.

[AdamWill]

@jkeenan up to you whether to close it or take it as an opportunity to add more docs, I guess? I don't mind.

[jkeenan]

@jkeenan up to you whether to close it or take it as an opportunity to add more docs, I guess? I don't mind.

If @Grinnz or anyone else wants to add more docs, they can do so in a new ticket which simply provides a URL to this one. Closing. Thanks.

[AdamWill]

FWIW, I had a lot of fun trying to get Google Gemini to figure out this problem. After a lot of hints, it nearly got there, but still wasn't convinced:

"Autovivification Anomaly? Could keys %{undef} somehow autovivify $options{group} into an empty hash reference {} before crashing or evaluating? Standard autovivification works on assignment to non-existent nested keys, not usually on reads like keys. It's not the standard behavior."

so however the magical AI machine is sucking up the wisdom of the internet about this, apparently it doesn't quite have it right yet. :P