`ioctl_set_msg failed:-1` from `rpi-eeprom-config` after reboot

[srcshelton]

Describe the bug

I used EDITOR=vim rpi-eeprom-config -e to create a ~2248-byte firmware configuration file on a series of CM4 devices with the latest firmware (1733575168/Sat 7th Dec) - this change was accepted and an update written to apply on reboot.

After rebooting, however, I can't recall the current configuration:

$ sudo rpi-eeprom-config
ioctl_set_msg failed:-1
bootloader_config

So there appear to be two issues here:

• rpi-eeprom-config has allowed a configuration to be deployed which it can't later read-back;

• If there is an error, then rpi-eeprom-config -e will open an editor with the error message, rather than erroring-out - potentially allowing a valid configuration to be overwritten with the error message if the user isn't careful to erase the content and save before exiting their editor (which is likely non-obvious to most users, even those making low-level changes such as firmware configuration edits).

Steps to reproduce the behaviour

Upload the following firmware configuration file with EDITOR=vim rpi-eeprom-config -e:

[all]
#BOOT_UART=0
BOOT_UART=1

# N.B. Not relevant on [pi5], [rpi500], [cm5].
# Set to 2 to use an external GPIO power button.
#
#WAKE_ON_GPIO=1
WAKE_ON_GPIO=0

# N.B. Defaults to 1 on [pi400], [pi500], [cm5].
#
#POWER_OFF_ON_HALT=0
POWER_OFF_ON_HALT=1

# Boot Order Codes, from [1].
#
# Try SD first (1), followed by NVMe PCIe ([cm4] & [pi5] only), USB
# mass-storage, then loop.
#
#  0x0 - Deprecated
#  0x1 - SD card (or eMMC)
#  0x2 - Network
#  0x3 - RPiBOOT ([cm4])
#  0x4 - USB-MSD
#  0x5 - BCM-USB-MSD (USB2.0 boot from USB-C socket or CM4IO USB-A), not [pi5]
#  0x6 - NVME ([cm4], [pi5])
#  0x7 - HTTP
#  0xe - STOP (display error pattern, reboot needed to continue)
#  0xf - RESTART
#
# [1]: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#BOOT_ORDER
#
#BOOT_ORDER=0xf41

# Set to -1 for infinite retries.
#
#MAX_RESTARTS=-1
#SD_BOOT_MAX_RETRIES=0
#NET_BOOT_MAX_RETRIES=0
#DHCP_TIMEOUT=45000
#DHCP_REQ_TIMEOUT=4000
#TFTP_FILE_TIMEOUT=30000
#TFTP_IP=192.168.1.99
#TFTP_PREFIX=0
#TFTP_PREFIX_STR=
#PXE_OPTION43=Raspberry Pi Boot
#DHCP_OPTION97=0x34695052
#MAC_ADDRESS=
#MAC_ADDRESS_OTP=

# Static IP address
#
#CLIENT_IP=
#SUBNET=
#GATEWAY=

#DISABLE_HDMI=0
#HDMI_DELAY=0

# Set to 0 to prevent bootloader updates from USB/Network boot
# For remote units EEPROM hardware write protection should be used.
#
#ENABLE_SELF_UPDATE=1

#FREEZE_VERSION=0

#HTTP_HOST=fw-download-alias1.raspberrypi.com
#HTTP_PORT=443
#HTTP_PATH=net_install
#IMAGER_REPO_URL=http://downloads.raspberrypi.org/os_list_imagingutility_v3.json
#NET_INSTALL_ENABLED=0
#NET_INSTALL_AT_POWER_ON=0
#NET_INSTALL_KEYBOARD_WAIT=900

# Up to 32 characters, e.g. '6665@169.254.1.1/,6666@/'
# Recommended to activate only by GPIO filter, e.g. '[gpio7=0]'
#
#NETCONSOLE=

# Could be used to boot a recovery partition if the user presses a button via
# GPIO filter, e.g. '[gpio7=0]'
#
#PARTITION=0

#USB_MSD_EXCLUDE_VID_PID=
#USB_MSD_DISCOVER_TIMEOUT=20000
#USB_MSD_LUN_TIMEOUT=2000
#USB_MSD_STARTUP_DELAY=0

#XHCI_DEBUG=0x0

[pi4]
#NET_INSTALL_ENABLED=1
#USB_MSD_PWR_OFF_TIME=1000
SDRAM_BANKLOW=3
[cm4]
#NET_INSTALL_ENABLED=0
#VL805=0
[cm4s]
#NET_INSTALL_ENABLED=0
[pi5]
#UART_BAUD=115200
#NET_INSTALL_ENABLED=1
#PSU_MAX_CURRENT=
SDRAM_BANKLOW=1
[cm5]
#NET_INSTALL_ENABLED=0
[none]

# vi: set syntax=cfg:

Device (s)

Raspberry Pi CM4

Bootloader configuration.

Previous configuration (from backup directory) read:

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf461

System

Gentoo userland

$ sudo vcgencmd bootloader_version
2024/12/07 12:39:28
version 3858f977ab6d689a226ad24d26749266762b7160 (release)
timestamp 1733575168
update-time 1736077005
capabilities 0x0000007f

$ sudo vcgencmd version
Nov 26 2024 12:52:41 
Copyright (c) 2012 Broadcom
version 2ae30f53898ae2f1ba77ff570a92991bedfb0398 (clean) (release) (start_cd)

$ uname -a
Linux rpi-cm4-4-32-xx.tld 6.6.62-v8+ #1817 SMP PREEMPT Wed Nov 20 13:34:18 GMT 2024 aarch64 GNU/Linux

Bootloader logs

[2660406.104633] reboot: Restarting system

  0.10 RPi: BOOTSYS release VERSION:951e1cc9 DATE: 2024/10/21 TIME: 15:24:54
  0.15 BOOTMODE: 0x06 partition 0 build-ts BUILDTIMESTAMP=1729520694 serial xxxx boardrev d03140 stc 515044
  0.32 PM_RSTS: 0x00000020
  0.32 part 00000000 reset_info 00000000
  0.35 uSD voltage 3.3V
  0.61 Initialising SDRAM rank 2 total-size: 64 Gbit 3200 (0x14 0x00)
  0.64 DDR 3200 1 0 64 152 BL:-1
  2.06 OTP boardrev d03140 bootrom 48b0 48b0
  2.08 Customer key hash 000000000000000000000000000000000000000000000000000000000000000
  2.15 VC-JTAG unlocked


  2.93 RPi: BOOTLOADER release VERSION:951e1cc9 DATE: 2024/10/21 TIME: 15:24:54
  2.98 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1729520694 serial xxxx boardrev d03140 stc 2498061
  2.33 Boot mode: SD (01) order f2564
  2.48 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  2.59SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  2.68 EMMC
  2.68 SD retry 1 oc 0
  2.92 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  2.04 OCR c0ff8080 [0]
CID: 00150100414a544434520600f4c05138
  2.23 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 25000000 Hz actual: 25000000 HZ div: 8 (4) status: 0x1fff0000 delay: 4
  2.33 S HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
  2.42 MBR: 0x00000001, 2103295 type: 0xee
  2.45 MBR: 0x00201800,  131072 type: 0xef
  2.49 MBR: 0x00000000,       0 type: 0x00
  2.52 MBR: 000221800,28301312 type: 0xee
  2.56 Trying partition: 0
  2.59 GPT 12a538013c707946ac3185bbd1549129 000000001 001d1efff num-partitions 128 entry-size 128 pentry 2 first 0
  2.69 GPT partition: 0 426f6f74-0000-11aa-aa11-00306543ecac 000000800..0000017ff
  2.77 GPT partition: 10 b921b045-1df0-41c3-af44-4c6f280d3fae 000001800..0002017ff
  2.83 GPT partition: 11 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 000201800..0002217ff
  2.91 Matched GUID bootable-part-idx 0 want partition 0
  2.96 type: 32 lba: 2103296 'BSD  4.4' ' BOOT_0     ' clusters 129024 (1)
  2.02 rsc 32 fat-sectors 1008 root dir cluster 2 sectors 0 entries 0
  2.09 FAT32 clusters 129024
  2.17 [sdcard] autoboot.txt not found
  2.18 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 0
  2.24 Trying partition: 0
  2.27 GPT 12a538013c707946ac3185bbd1549129 000000001 001d1efff num-partitions 128 entry-size 128 pentry 2 firs 0
  2.37 GPT partition: 0 426f6f74-0000-11aa-aa11-00306543ecac 000000800..0000017ff
  2.45 GPT partition: 10 b921b045-1df0-413-af44-4c6f280d3fae 000001800..0002017ff
  2.51 GPT partition: 11 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 000201800..0002217ff
  2.59 Matched GUID bootable-part-idx 0 want partition 0
  2.64 type: 32 lba: 2103296 'BSD  4.4' ' BOOT_0     ' clusters 129024 (1)
  2.70 rsc 32 fat-sectors 1008 root dir cluster 2 sectors 0 entries 0
  2.77 FAT32 clusters 129024
  2.83 Read config.txt bytes     7373 hnd 0x1ea53
  2.88 Read overclock.txt bytes     2296 hnd 0x1664f
  2.93 Read legacy.txt bytes     3563 hnd 0x16648
  2.96 pieeprom.sig
  2.96 hash: 0cef128231dd69ae90ae97f3fddd1215b94333bf17bf1ff0dcaae2e03351c901
  3.03 ts: 1735278939
  3.91 SELF-UPDATE timestamp current 1729691315 new 1735278939
  3.94 Updating bootloader EEPROM
  3.97 Reading EEPROM: 524288 bytes 0x00b60000
  3.59 358ms
  3.94 Writing EEPROM
+++++++++++++..++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++...................................................++  6.71
  6.71 2877ms

  6.71 Verify BOOT EEPROM
  6.73 Reading EEPROM: 524288 bytes 0x00b60000
  6.35 358ms
  6.70 BOOT-EEPROM: UPDATED
  6.70 EEPROMs updated. Rebooting
  6.72 RESET

  0.19 RPi: BOOTSYS release VERSION:3858f977 DATE: 2024/12/07 TIME: 12:39:28
  0.23 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1733575168 serial xxxx boardrev d03140 stc 523156
  0.48 PM_RSTS: 0x00000020
  0.48 part 00000000 reset_info 00000000
  0.51 uSD voltage 3.3V
  0.77 Initialising SDRAM rank 2 total-size: 64 Gbit 3200 (0x14 0x00)
  0.80 DDR 3200 1 0 64 152 BL:3
  2.22 OTP boardrev d03140 bootrom 48b0 48b0
  2.24 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000
  2.31 VC-JTAG unlocked


  2.09 RPi: BOOTLOADER release VERSION:3858f977 DATE: 2024/12/07 TIME: 12:39:28
  2.13 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1733575168 serial xxxx boardrev d03140 stc 2513868
  2.49 Boot mode: SD (01) order f64
  2.63 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  2.74 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  2.83 EMMC
  2.83 SD retry 1 oc 0 
  2.08 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  2.19 OCR c0ff8080 [0]
CID: 00150100414a544434520600f4c05138
  2.38 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 25000000 Hz actual: 25000000 HZ div: 8 (4) status: 0x1fff0000 delay: 4
  2.49 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
  2.58 MBR: 0x00000001, 2103295 type: 0xee
  2.60 MBR: 0x00201800,  131072 type: 0xef
  2.64 MBR: 0x00000000,       0 type: 0x00
  2.68 MBR: 0x00221800,28301312 type: 0xee
  2.72 Trying partition: 0
  2.75 GPT 12a538013c707946ac3185bbd1549129 000000001 001d1efff num-paritions 128 entry-size 128 pentry 2 first 0
  2.85 GPT partition: 0 426f6f74-0000-11aa-aa11-00306543ecac 000000800..0000017ff
 2.92 GPT partition: 10 b921b045-1df0-41c3-af44-4c6f280d3fae 000001800..0002017ff
  2.99 GPT partition: 11 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 000201800..0002217ff
  2.06 Matched GUID bootable-part-idx 0 want partition 0
  2.12 type: 32 lba: 2103296 'BSlusters 129024
  2.32 [sdcard] autoboot.txt not found
  2.33 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 0
  2.40 Tryig partition: 0
  2.42 GPT 12a538013c707946ac3185bbd1549129 000000001 001d1efff num-partitions 128 entry-size 128 pentry 2 first 0
  2.52 GPT partition: 0 426f6f74-0000-11aa-aa11-00306543ecac 000000800..0000017ff
  2.60 GPT partition: 10 b921b045-1df0-41c3-af44-4c6f280d3fae 000001800..0002017ff
  2.67 GPT partition: 11 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 000201800..0002217ff
  2.74 Matched GUID bootable-part-idx 0 want partition 0
  2.79 type: 32 lba: 2103296 'BSD  4.4' ' BOOT_0     ' clusters 129024 1)
  2.86 rsc 32 fat-sectors 1008 root dir cluster 2 sectors 0 entries 0
  2.92 FAT32 clusters 129024
  2.98 Read config.txt bytes     7373 hnd 0x1ea53
  3.03 Read overclock.txt bytes     2296 hnd 0x1664f
  3.08 Read legacy.txt bytes     3563 hnd 0x16648
  3.11 pieeprom.sig
  3.12 hash: 0cef128231dd69ae90ae97f3fddd1215b94333bf17bf1ff0dcaae2e03351c901
  3.18 ts: 1735278939  
  3.06 SELF-UPDATE timestamp current 1735278939 new 1735278939 skip
  3.12 [sdcard] recover4.elf not found
  3.15 [sdcard] recvery.elf not found
  3.64 Read start4cd.elf bytes   812636 hnd 0x16014
  3.67 Read fixup4cd.dat bytes     3230 hnd 0x1f5b7
  3.70 0x00d03140 0x0000000 0x00001fff
  3.74 MEM GPU: 16 ARM: 998 TOTAL: 1014
  3.00 Firmware: 2ae30f53898ae2f1ba77ff570a92991bedfb0398 Nov 26 2024 12:52:41
  3.17 Starting start4cd.elf @ 0xff000200 partition 0
  3.19 PCI0 reset
  3.30 +

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

No response

[timg236]

What does "vcgencmd bootloader_config" report in this configuration?

That's a rather larger EEPROM config but should be legal

[srcshelton]

What does "vcgencmd bootloader_config" report in this configuration?

Same output:

$ sudo vcgencmd bootloader_config
ioctl_set_msg failed:-1
bootloader_config

The configuration does appear to have applied, as /proc/cmdline contains:

coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_headphones=0 cgroup_disable=memory numa_policy=interleave snd_bcm2835.enable_headphones=1 bcm2708_fb.fbwidth=0 bcm2708_fb.fbheight=0 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 numa=fake=2 system_heap.max_order=0 smsc95xx.macaddr=E4:5F:01:A8:3B:24 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000  console=tty0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 root=PARTUUID=xxxx rootfstype=ext4 fsck.repair=yes rootwait cpufreq.default_governor=performance cgroup_enable=memory cgroup_memory=1 zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=20 zswap.zpool=z3fold

… which includes numa_policy=interleave […] numa=fake=2 (which doesn't appear in cmdline.txt), and I believe that is only inserted if the EEPROM configuration contains SDRAM_BANKLOW=3 (unless that's now default?) which would suggest that the configuration can be applied but is not readable 😯

[pelwell]

Error -1 is -EPERM. Running under strace, i.e. sudo strace rpi-eeprom-config, while very verbose, should enable you to work out which system call is failing.

[srcshelton]

Additionally, almost the same configuration file (with the WAKE_ON_GPIO and POWER_OFF_ON_HALT overrides removed) works as expected on an RPi5 and RPi500 (albeit both running Debian).

[pelwell]

Running on the Pi 400 I have to hand eventually gets to:

openat(AT_FDCWD, "/sys/bus/nvmem/devices/rmem3/nvmem", O_RDONLY|O_CLOEXEC) = 4
newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=173, ...}, AT_EMPTY_PATH) = 0
ioctl(4, TCGETS, 0x7feb3da070)          = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR)                   = 0
lseek(4, 0, SEEK_CUR)                   = 0
newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=173, ...}, AT_EMPTY_PATH) = 0
read(4, "[all]\nBOOT_UART=1\nWAKE_ON_GPIO=1"..., 174) = 173
read(4, "", 1)                          = 0
close(4)                                = 0
close(3)                                = 0
write(1, "[all]\nBOOT_UART=1\nWAKE_ON_GPIO=1"..., 173[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
ENABLE_SELF_UPDATE=1
DISABLE_HDMI=0
BOOT_ORDER=0xf421

) = 173
...

[srcshelton]

$ sudo strace -vvffs 128 -- vcgencmd bootloader_config 2>&1 | grep -FC 5 -e EINVAL -e ioctl_set_msg
mprotect(0x558e35f000, 4096, PROT_READ) = 0
mprotect(0x7fa8802000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7fa87f9000, 14850)             = 0
openat(AT_FDCWD, "/dev/vcio", O_RDONLY) = 3
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x64, 0, 0x8), 0x7fdf236108) = -1 EINVAL (Invalid argument)
fstat(1, {st_dev=makedev(0, 0xd), st_ino=275688, st_mode=S_IFIFO|0600, st_nlink=1, st_uid=1000, st_gid=100, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1736178913 /* 2025-01-06T15:55:13.114061971+0000 */, st_atime_nsec=114061971, st_mtime=1736178913 /* 2025-01-06T15:55:13.114061971+0000 */, st_mtime_nsec=114061971, st_ctime=1736178913 /* 2025-01-06T15:55:13.114061971+0000 */, st_ctime_nsec=114061971}) = 0
getrandom("\xc8\xab\x1e\xa5\x25\xf3\x39\xb1", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x559ff75000
brk(0x559ff96000)                       = 0x559ff96000
close(3)                                = 0
write(1, "ioctl_set_msg failed:-1\nbootloader_config\n", 42) = 42
exit_group(0)                           = ?
+++ exited with 0 +++

… so it seems to be the VCIO driver which is unhappy, even though other vcgencmd invocations succeed.

And also note that the only system change between this command working and now when it doesn't was uploading the new configuration and then rebooting… although that could be due to a formerly loaded module now not being automatically loaded?

Update: I've loaded every module for the current kernel, and I'm still getting the same error.

[timg236]

Does it still fail if you make the configuration file smaller e.g. delete the comments to get it under 2K ?

[srcshelton]

Hmm - it seems not: I've reduced the length to 1677 characters and I'm still seeing the same behaviour. I guess that either means that this occurs well below 2k in size, that once the problem is triggered it keeps happening, or that it's related to the latest firmware or kernel releases (as both were updated between when this last worked as anticipated and now).

Is there any way (… short of recompiling the kernel with an updated driver with debug enabled?) to get more information about what's causing the ioctl syscall on the vcio device to return -1?

[pelwell]

I think the limit is probably in vcgencmd in our utils repo. You can try increasing this constant: https://github.com/raspberrypi/utils/blob/master/vcgencmd/vcgencmd.c#L44

[srcshelton]

I think the limit is probably in vcgencmd in our utils repo. You can try increasing this constant: https://github.com/raspberrypi/utils/blob/master/vcgencmd/vcgencmd.c#L44

Happy to give that a try! Are you able to tell me the physical limit/size of the EEPROM area reserved for bootloader configuration that sets an upper-limit for this constant?

[pelwell]

For now, just make it 10 times larger to see if it fixes the problem.

[srcshelton]

That's got it - I increased MAX_STRING to 4096 (and moved from commit raspberrypi/utils@aaeac09 to commit raspberrypi/utils@33aa4da), and things now work as expected.

This post - https://forums.raspberrypi.com/viewtopic.php?t=243549 seems to suggest that RPi4/CM4 models have 512KiB EEPROM capacity… should this therefore be the limit in https://github.com/raspberrypi/utils/blob/master/vcgencmd/vcgencmd.c#L44?

[pelwell]

Were the legitimate, useful content to expand to the point that an existing EEPROM configuration no longer fitted, the tools would fail and users would get upset. It's much better to set a sensible limit that we will always support and that cannot be exceeded - 4kB feels about right, but I wouldn't object to 2kB.

[timg236]

4K is a safe bet, it certainly won't exceed 8KB because there isn't enough free space in the Pi4/CM4 SPI flash to have a larger config file!