There are few things I dislike more in this world than getting questions on my setup guides that I don’t know the answer to, but thanks to an investigation by Bill Schatzow we can strike one of those issues off the list!
We’ve had a few comments of people who have encountered this issue over the years. Given that at best only 1% of people who visit the site leave a comment I think it’s safe to say that this issue has plagued thousands of people over the past 10-12 months.
Let’s take a look!
Hardware Involved
Raspberry Pi 4
The Raspberry Pi 4 is available in different memory configurations all the way up to 8 GB. It’s about the size of a credit card and uses an extremely low amount of power making it ideal for all sorts of projects and ideas!
The issue manifests in HomeAssistant versions above 5.4 when having a certain version of the Raspberry Pi firmware installed.
When using Raspberry Pi 4 with firmware 1.20201022 the system occasionally (typically around 1-5 days) to hard freeze. The freeze leaves no traces: There is no stack trace or anything on the serial console, kernel dmesg seems to be quite (e.g. re-plugging a USB keyboard does not lead to re-enumeration or anything. Also magic sysrq via serial console (break signal) seems not to work at this point.
In my testing I use consoleblank=0 to avoid the display output going into suspend mode. Home Assistant OS uses no graphical output, so the default screen shown is just the kernel built in virtual terminal. When the freeze happens, the HDMI screen is stuck on the last screen (e.g. Home Assistant welcome console). The console cursor stops blinking.
The problem was a change recommended to be made by Broadcom to the firmware that affected a small number of users. Here is the update that Bill gave me on the issue:
Just to keep you updated the issue is finally fixed. The fix had nothing to do with the different hardware that was attached to the Pi4 ( Power Supplies, controllers, SSD, etc). Back in Nov. 2020 RPI developers made a change in the firmware (recommended by Broadcom) that effected a small number of users. It has since been corrected and a release was issued in early Jan 2022. If interested you can follow the troubleshooting and fixes here: GitHub Issue #1646 – Raspberry Pi Firmware
It sounds like USB booting was not necessarily a requirement to experience this issue. If you’ve experienced it on a SD card it’s worth upgrading to the latest as well.
How to Fix
To fix this issue you just need to update to the new HAOS “stable” release which has an updated firmware inside that no longer has this issue. Here is one of the developer’s comments:
HAOS 7.1 with firmware oldstable_20211201 has now been on our beta channel for several days and got promoted to stable today. Multiple people confirmed that the new firmware/the new OS release fixed the stability issues. So we can consider this issue fixed! Thanks for your help to get this resolved, very much appreciated!
Hopefully this is able to reach some of the people that may have been impacted by this issue. I know I’ve personally seen it through the comments here on the site over the years on my various guides involving SSD booting with the Pi.
I want to thank Bill Schatzow for doing nearly an 11 month investigation on this issue and since we’ve had a bunch of comments on the site about it I figured I’d post it here. Thanks Bill!
I wish I could take credit for the fix. If you follow the thread in the link there were several more knowledgeable then me HA users that tracked down the cause and credit to them as well as Stefan ( the HA developer in charge of the OS) and several RPI people that came up with the fix. I am just the dog with the bone that would not drop the bone.
That is fair and very gracious! I definitely think you at least deserve credit for being the dog with the bone that would not drop the bone. I’ll take this chance to thank Stafan and the others as well for their work on this!
I share your enthusiasm and give credit where it is due at every opportunity I can here on the site. It would be a lie to pretend that the site was not essentially a community effort and that a lot of the valuable comment on here is other people’s comments or ideas/things I’ve covered from other people’s comments.
That’s the world I want to live in at least and my goals for this site are to try to facilitate that however I can (through open source projects, these types of conversations/dialog, etc) and keep it all available. Honestly I thought that was the world we would be living in when I grew up in the 90s and took it almost for granted that it would turn out that way but these days I’m really going against the grain even having a functioning web site anymore and not just submitting to a megaplatform trying to collect/sell everyone’s data. That internet isn’t gone yet but it’s getting smaller and not bigger it seems like.
I wish I could take credit for the fix. If you follow the thread in the link there were several more knowledgeable then me HA users that tracked down the cause and credit to them as well as Stefan ( the HA developer in charge of the OS) and several RPI people that came up with the fix. I am just the dog with the bone that would not drop the bone.
Hey Bill,
That is fair and very gracious! I definitely think you at least deserve credit for being the dog with the bone that would not drop the bone. I’ll take this chance to thank Stafan and the others as well for their work on this!
I share your enthusiasm and give credit where it is due at every opportunity I can here on the site. It would be a lie to pretend that the site was not essentially a community effort and that a lot of the valuable comment on here is other people’s comments or ideas/things I’ve covered from other people’s comments.
That’s the world I want to live in at least and my goals for this site are to try to facilitate that however I can (through open source projects, these types of conversations/dialog, etc) and keep it all available. Honestly I thought that was the world we would be living in when I grew up in the 90s and took it almost for granted that it would turn out that way but these days I’m really going against the grain even having a functioning web site anymore and not just submitting to a megaplatform trying to collect/sell everyone’s data. That internet isn’t gone yet but it’s getting smaller and not bigger it seems like.
Cheers and thanks for stopping by Bill!