There are few things I dislike more in this world than getting questions on my setup guides that I don’t know the answer to, but thanks to an investigation by Bill Schatzow we can strike one of those issues off the list!
We’ve had a few comments of people who have encountered this issue over the years. Given that at best only 1% of people who visit the site leave a comment I think it’s safe to say that this issue has plagued thousands of people over the past 10-12 months.
Let’s take a look!
The Raspberry Pi 4 is available in different memory configurations all the way up to 8 GB. It’s about the size of a credit card and uses an extremely low amount of power making it ideal for all sorts of projects and ideas!
The Issue and Symptoms
The issue manifests in HomeAssistant versions above 5.4 when having a certain version of the Raspberry Pi firmware installed.
When using Raspberry Pi 4 with firmware
1.20201022the system occasionally (typically around 1-5 days) to hard freeze. The freeze leaves no traces: There is no stack trace or anything on the serial console, kernel dmesg seems to be quite (e.g. re-plugging a USB keyboard does not lead to re-enumeration or anything. Also magic sysrq via serial console (break signal) seems not to work at this point.
In my testing I useOfficial Raspberry Pi Firmware GitHub – Issue 1646
consoleblank=0to avoid the display output going into suspend mode. Home Assistant OS uses no graphical output, so the default screen shown is just the kernel built in virtual terminal. When the freeze happens, the HDMI screen is stuck on the last screen (e.g. Home Assistant welcome console). The console cursor stops blinking.
The problem was a change recommended to be made by Broadcom to the firmware that affected a small number of users. Here is the update that Bill gave me on the issue:
Just to keep you updated the issue is finally fixed. The fix had nothing to do with the different hardware that was attached to the Pi4 ( Power Supplies, controllers, SSD, etc). Back in Nov. 2020 RPI developers made a change in the firmware (recommended by Broadcom) that effected a small number of users. It has since been corrected and a release was issued in early Jan 2022. If interested you can follow the troubleshooting and fixes here: GitHub Issue #1646 – Raspberry Pi FirmwareLegendary Technology Blog – Comment 12688
It sounds like USB booting was not necessarily a requirement to experience this issue. If you’ve experienced it on a SD card it’s worth upgrading to the latest as well.
How to Fix
To fix this issue you just need to update to the new HAOS “stable” release which has an updated firmware inside that no longer has this issue. Here is one of the developer’s comments:
HAOS 7.1 with firmwareGitHub Issues 1646 comments
oldstable_20211201has now been on our beta channel for several days and got promoted to stable today. Multiple people confirmed that the new firmware/the new OS release fixed the stability issues. So we can consider this issue fixed! Thanks for your help to get this resolved, very much appreciated!
Instructions for how to flash the latest HAOS on Raspberry Pi are available here: HAOS Raspberry Pi Installation Instructions
Hopefully this is able to reach some of the people that may have been impacted by this issue. I know I’ve personally seen it through the comments here on the site over the years on my various guides involving SSD booting with the Pi.
I want to thank Bill Schatzow for doing nearly an 11 month investigation on this issue and since we’ve had a bunch of comments on the site about it I figured I’d post it here. Thanks Bill!