Fail to boot after bootloader config update

[holzgeist]

Describe the bug

I have a script on my raspberry pi 5 to power it off during night and auto-power it on in the morning via RTC.
Since the last update (rpi-update) I cannot enable POWER_OFF_ON_HALT anymore without breaking boot.
The green LED blinks 8 times, indicating SDCard issues. Resetting the bootloader with Raspberry Pi Imager gets the system to boot again, but of course without the power changes

It does not matter if I manually edit the config with rpi-eeprom-config -e or do the change automatically with raspi-config

Here's a log of trying the change with raspi-config:

Updating bootloader EEPROM
 image: /usr/lib/firmware/raspberrypi/bootloader-2712//default/pieeprom-2025-02-12.bin
config_src: /tmp/tmp.XN6W9e1Wsy
config: /tmp/tmp.XN6W9e1Wsy
################################################################################
[all]
BOOT_UART=1
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1
POWER_OFF_ON_HALT=1
WAKE_ON_GPIO=0

################################################################################
*** CREATED UPDATE /tmp/tmpyydizbmh/pieeprom.upd  ***

   CURRENT: Wed 12 Feb 10:51:52 UTC 2025 (1739357512)
    UPDATE: Wed 12 Feb 10:51:52 UTC 2025 (1739357512)
    BOOTFS: /boot/firmware
'/tmp/tmp.wdYlRegB4u' -> '/boot/firmware/pieeprom.upd'

UPDATING bootloader. This could take up to a minute. Please wait

*** Do not disconnect the power until the update is complete ***

If a problem occurs then the Raspberry Pi Imager may be used to create
a bootloader rescue SD card image which restores the default bootloader image.

flashrom -p linux_spi:dev=/dev/spidev10.0,spispeed=16000 -w /boot/firmware/pieeprom.upd
Verifying update
VERIFY: SUCCESS
UPDATE SUCCESSFUL

As mentioned, I can get the SD card to boot by resetting the booloader with a different SD card. But every time I try touch the eeprom, it goes in to 8-blink mode again.

UPDATE: Even running rpi-eeprom-config to print the current configuration breaks the bootloader

Steps to reproduce the behaviour

either running raspi-config -> Advanced Options -> Shutdown Behaviour -> Full power off ...
Or rpi-eeprom-config -e and enter following config:

[all]
BOOT_UART=1
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1
POWER_OFF_ON_HALT=1
WAKE_ON_GPIO=0

Device (s)

Raspberry Pi 5

Bootloader configuration.

This is the config that's in a working environment:

[all]
BOOT_UART=1
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1

That's the config I'd like to use:

[all]
BOOT_UART=1
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1
POWER_OFF_ON_HALT=1
WAKE_ON_GPIO=0

System

cat /etc/rpi-issue:

Raspberry Pi reference 2024-07-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 48efb5fc5485fafdc9de8ad481eb5c09e1182656, stage4

vcgencmd bootloader_version:

2025/02/12 10:51:52
version f788aab60121515fe59c37c899186831e6c19585 (release)
timestamp 1739357512
update-time 1739883498
capabilities 0x0000007f

vcgencmd version:

2025/02/12 10:51:52 
Copyright (c) 2012 Broadcom
version f788aab6 (release) (embedded)

uname -a: Linux raspberrypi5 6.6.78-v8-16k+ #1854 SMP PREEMPT Mon Feb 17 16:37:33 GMT 2025 aarch64 GNU/Linux

Bootloader logs

No response

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

No response

[timg236]

Is the failure mode specific to those lines or does it fail if you add some dummy values i.e. is it the flashrom update that's failing even though it was able to read it and verify it ok.
The settings worked for me so there must be something more subtle going on than just these settings being bad

e.g. Please could you try using rpi-eeprom-config --edit

[all]
BOOT_UART=1
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1
DUMMY1=FOO
DUMMY2=BAR

The other thing to try is setting changing
RPI_EEPROM_USE_FLASHROM=1 to 0 in /etc/default/rpi-eeprom-update

[lurch]

@holzgeist What Revision does cat /proc/cpuinfo display?

[timg236]

Does this work if the board is woken by pressing the power-button / is the failure specific to RTC wakeup?

[holzgeist]

@lurch cat /proc/cpuinfo:

processor	: 0
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

processor	: 1
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

processor	: 2
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

processor	: 3
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

Revision	: d04170
Serial		: c2c5056e1cd3be1b
Model		: Raspberry Pi 5 Model B Rev 1.0

@timg236 I'll try the options you gave me shortly

[holzgeist]

It's not related to RTC wakeup.
After resetting the bootloader and disabling my RTC script, the raspberry booted fine for at least 5 times.

Then I ran sudo rpi-eeprom-config --edit:

Updating bootloader EEPROM
 image: /usr/lib/firmware/raspberrypi/bootloader-2712/default/pieeprom-2025-02-12.bin
config_src: blconfig device
config: /tmp/tmp9g6wd_qc/boot.conf
################################################################################
[all]
BOOT_UART=1
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1
DUMMY1=FOO
DUMMY2=BAR

################################################################################
*** CREATED UPDATE /tmp/tmp9g6wd_qc/pieeprom.upd  ***

   CURRENT: Wed 12 Feb 10:51:52 UTC 2025 (1739357512)
    UPDATE: Wed 12 Feb 10:51:52 UTC 2025 (1739357512)
    BOOTFS: /boot/firmware
'/tmp/tmp.3FMeX6Ywn7' -> '/boot/firmware/pieeprom.upd'

UPDATING bootloader. This could take up to a minute. Please wait

*** Do not disconnect the power until the update is complete ***

If a problem occurs then the Raspberry Pi Imager may be used to create
a bootloader rescue SD card image which restores the default bootloader image.

flashrom -p linux_spi:dev=/dev/spidev10.0,spispeed=16000 -w /boot/firmware/pieeprom.upd
Verifying update
VERIFY: SUCCESS
UPDATE SUCCESSFUL

It rebooted fine. Running the command (with edit) again, the dummy options were still there. I quit without changing and the script showed the exact same output as above (with other tmp files of course).
But then reboot got stuck with 8-blink LED again

As mentioned above, the service that writes RTC is disabled currently

[holzgeist]

Also, sometimes I need to run the booloader reset twice to get it up and running again

[holzgeist]

ok, next update: It seems to be completely random. One time I can reboot 5 times in a row without reflashing. Then (without doing anything different) it doesn't work anymore. Another time the first reboot fails..
Everytime with the LED blinking 8 times

[timg236]

Are you able to try an older version of the bootloader e.g. pieeprom-2025-01-22?

pi@pi5:~ $ cp /lib/firmware/raspberrypi/bootloader-2712/default/pieeprom-2025-01-22.bin .
pi@pi5:~ $ sudo rpi-eeprom-config --edit pieeprom-2025-01-22.bin

8-blinks is SDRAM error

[holzgeist]

It rebooted fine at first. Adding DUMMY1=FOO to the config causes the SDRAM error (I used sudo rpi-eeprom-config --edit pieeprom-2025-01-22.bin too, so I didn't update the bootloader by accident)

[holzgeist]

I'll check if I have yet another SD Card to test with a completely new install

[holzgeist]

I could not get a new install to boot consistently either.. I'm giving up for today and will retry after the weekend