Fix Home Assistant / HAOS Raspberry Pi USB/SSD Boot Freeze

Home Assistant / HAOS
Home Assistant / HAOS

There are few things I dislike more in this world than getting questions on my setup guides that I don’t know the answer to, but thanks to an investigation by Bill Schatzow we can strike one of those issues off the list!

We’ve had a few comments of people who have encountered this issue over the years. Given that at best only 1% of people who visit the site leave a comment I think it’s safe to say that this issue has plagued thousands of people over the past 10-12 months.

Let’s take a look!

Hardware Involved

Raspberry Pi 4
Raspberry Pi 4

The Raspberry Pi 4 is available in different memory configurations all the way up to 8 GB. It’s about the size of a credit card and uses an extremely low amount of power making it ideal for all sorts of projects and ideas!

Links: Amazon.com*, AliExpress*, Amazon.ca*, Amazon.com.au*, Amazon*.co.jp*, Amazon.co.uk*, Amazon.de*, Amazon.es*, Amazon.fr*, Amazon.it*, Amazon.nl*, Amazon.pl*, Amazon.se*, Amazon.sg*

The Issue and Symptoms

The issue manifests in HomeAssistant versions above 5.4 when having a certain version of the Raspberry Pi firmware installed.

When using Raspberry Pi 4 with firmware 1.20201022 the system occasionally (typically around 1-5 days) to hard freeze. The freeze leaves no traces: There is no stack trace or anything on the serial console, kernel dmesg seems to be quite (e.g. re-plugging a USB keyboard does not lead to re-enumeration or anything. Also magic sysrq via serial console (break signal) seems not to work at this point.

In my testing I use consoleblank=0 to avoid the display output going into suspend mode. Home Assistant OS uses no graphical output, so the default screen shown is just the kernel built in virtual terminal. When the freeze happens, the HDMI screen is stuck on the last screen (e.g. Home Assistant welcome console). The console cursor stops blinking.

Official Raspberry Pi Firmware GitHub – Issue 1646

The problem was a change recommended to be made by Broadcom to the firmware that affected a small number of users. Here is the update that Bill gave me on the issue:

Just to keep you updated the issue is finally fixed. The fix had nothing to do with the different hardware that was attached to the Pi4 ( Power Supplies, controllers, SSD, etc). Back in Nov. 2020 RPI developers made a change in the firmware (recommended by Broadcom) that effected a small number of users. It has since been corrected and a release was issued in early Jan 2022. If interested you can follow the troubleshooting and fixes here: GitHub Issue #1646 – Raspberry Pi Firmware

Legendary Technology Blog – Comment 12688

It sounds like USB booting was not necessarily a requirement to experience this issue. If you’ve experienced it on a SD card it’s worth upgrading to the latest as well.

How to Fix

To fix this issue you just need to update to the new HAOS “stable” release which has an updated firmware inside that no longer has this issue. Here is one of the developer’s comments:

HAOS 7.1 with firmware oldstable_20211201 has now been on our beta channel for several days and got promoted to stable today. Multiple people confirmed that the new firmware/the new OS release fixed the stability issues. So we can consider this issue fixed! Thanks for your help to get this resolved, very much appreciated!

GitHub Issues 1646 comments

Instructions for how to flash the latest HAOS on Raspberry Pi are available here: HAOS Raspberry Pi Installation Instructions

Conclusion

Hopefully this is able to reach some of the people that may have been impacted by this issue. I know I’ve personally seen it through the comments here on the site over the years on my various guides involving SSD booting with the Pi.

I want to thank Bill Schatzow for doing nearly an 11 month investigation on this issue and since we’ve had a bunch of comments on the site about it I figured I’d post it here. Thanks Bill!

Other Resources

Best Working SSD / Storage Adapters for Raspberry Pi 4 / 400

Fixing Storage Adapters for Raspberry Pi with Firmware Updates

guest

2 Comments
Inline Feedbacks
View all comments

Bill Schatzow
Bill Schatzow
10 months ago

I wish I could take credit for the fix. If you follow the thread in the link there were several more knowledgeable then me HA users that tracked down the cause and credit to them as well as Stefan ( the HA developer in charge of the OS) and several RPI people that came up with the fix. I am just the dog with the bone that would not drop the bone.