Native support for USB mass storage device booting has made it’s way to all of the Raspberry Pi 4’s firmware release channels! With that exciting development an old issue from the Pi 3 and earlier USB booting days has manifested itself once again in the form of a constant 1-2% CPU draw even with nothing running.
This guide will show you an easy config.txt parameter to eliminate this problem and optimize your mass storage boot setups!
Equipment Used
StarTech 2.5″ SATA to USB 3.1 Adapter
The USB 3.1 variant of the StarTech 2.5″ SATA adapter works well with the Pi 4. The USB 3.0 variant doesn’t have firmware updates available and is not recommended.
The Kingston A400 has been a great drive to use with the Pi for years. It’s reliable, widely available around the world, has low power requirements and performs very well. It’s also very affordable. This drive has been benchmarked over 1000 times at pibenchmarks.com and is the #1 most popular SSD among the Pi community!
The Raspberry Pi polls endlessly for a SD card to be inserted if there is not a SD card in the slot. It polls so quickly that it actually registers as CPU usage on your system. You can observe this if you write an image to a SD card and look at the idle CPU usage and then put the same image on a SSD. On there you will notice a constant ~1% CPU drain:
Top command running on Ubuntu booted on a USB mass storage device (SSD)
Notice the mystery rcu_preempt process using 0.7% CPU? That is the SD card polling. The exact name of the process can vary depending on which OS and version you are running but it will never be an application and will always be a generic system background process. There is absolutely nothing running on this machine other than my “top” command, which actually came in second in CPU usage to this process. This is wasting your CPU cycles!
You can actually test that this is indeed the SD card slot polling yourself. Look what happens once we put a SD card in the slot:
Top command after inserting SD card into SD card slot
Now we have a completely clean “top” where the only thing on the system consuming CPU cycles is the top application itself!
It doesn’t matter if the SD card is blank or not or what is on there, just that you put it in there. This will cause the polling to stop and free your CPU again. If you take it out the CPU usage will immediately return!
You may see the rcu_preempt still fire up and consume CPU cycles from time to time. That’s okay, it is taking care of other tasks than just SD polling, but the constant 0.7-1% CPU usage drain will disappear!
Fixing SD card polling with config.txt
Fortunately there is a more permanent / better way to fix this issue without having to put a SD card in the slot to do nothing! The solution is to add a parameter to your config.txt file (/boot/config.txt on Raspberry Pi OS, /boot/firmware/config.txt on Ubuntu) to instruct the Pi not to continuously check for a SD card to be inserted and instead to poll just once (upon startup). The parameter to add to your /boot/config.txt file is:
dtparam=sd_poll_once
Now restart your Pi. Even without a SD card inserted your CPU usage will be back to normal!
Old method
There was an older parameter for changing this that used to be used on the Raspberry Pi 3 and older. This was:
dtoverlay=sdtweak,poll_once=on
This syntax has been obsoleted and no longer works on the Pi 4. I figured I would put it here though to eliminate confusion and in case anyone is running a really old OS and needs it or is trying to search for this looking for the updated syntax.
Benchmarking / Testing Storage
If you want to verify your drive’s performance you may want to run my storage benchmark with:
I added a line dtparam=sd_poll_once under the [pi4] section in /boot/firmware/config.txt (Ubuntu 20.04), updated system and rebooted. But if I do: top -p pidof rcu_preempt the rcu_preempt process is still sitting there, with 0.0% CPU which briefly goes up to 0.3% every 5 seconds or so. Is this the expected result?
Ubuntu can be a little tricky as I believe they have a couple of different config files. The easiest way to tell is to do the SD card test for sure.
If you put a SD card in the slot does this behavior stop? If it does that means the change is for sure *not* working. It might not stop since rcu_preempt can be other things (it’s the realtime scheduler) and usually the usage is constant and doesn’t really go away.
My guess is that is the case and that putting in the SD card won’t change this behavior. If you take out the line from your config.txt file the original behavior should return (which has a fairly constant usage, it doesn’t go away and kind of stays at 1-2%ish because it’s polling the SD card slot hundreds of times per second).
If you find out that this config.txt change isn’t working you may need to use the file:
/boot/firmware/usercfg.txt
in Ubuntu 20.04. I believe making changes in the main config.txt get ignored on that version (or get replaced often by updates) but I would say try using usercfg.txt if you determine it’s not working. This might not be having an impact right now and the change might be working but I think next time the boot files are updated the change will be lost so I thought I’d add this either way. Let us know what you find!
So I made another test: I made a new fresh Ubuntu 20.04 setup on my SSD (thanks for the guide btw!) and I took a look again at the rcu_preempt process without changing anything. The process was there but 0% CPU, didn’t even go up to 0.3% as before. I put an SD card while running top, and no behavior change..
Thanks for the followup, that’s very interesting and thanks for your kind words about the USB booting guide! It’s possible the explanation is that they may have finally patched this on newer kernels to not check so often (at least for Ubuntu). I’ve been waiting literally *years* for this across multiple Pi generations! This was even a problem while USB booting all the way back on Pi 3B models (and others that supported native USB booting before the Pi 4 came out which didn’t have it at launch but does now).
Everything sounds okay with the process still being present as it’s used for other low-level hardware interrupts. Activity in that process is often the Pi talking to your Linux OS at a very low level behind the scenes via the scheduler asking “was a SD card inserted yet”? And then about 1 ms later (or maybe even more often than this is my understanding) it again asks “was a SD card inserted yet” over and over and that is what is generating the activity this guide addresses but other low level activity that uses the scheduler can create activity as well. That is why putting in the SD card stops it because the Pi finally says “YES IT’S IN HERE SHUT UP!” and it does!
I believe this issue would be a kernel level patch to address so it may just be in the newer LTS version of Ubuntu. Last time I checked the fix hasn’t made it’s way to Raspberry Pi OS yet but maybe this is a fairly recent development so I’ll have to check for it the next time I’m USB booting one.
Thanks for the heads up! I think people on Pi OS (for now) and 18.04 are probably likely to still encounter this as if the fix is ever backported that far it will take quite a while to get there. I’m glad you were able to get yours resolved all the way down to a very negligible amount!
I added a line dtparam=sd_poll_once under the [pi4] section in /boot/firmware/config.txt (Ubuntu 20.04), updated system and rebooted. But if I do: top -p
pidof rcu_preemptthe rcu_preempt process is still sitting there, with 0.0% CPU which briefly goes up to 0.3% every 5 seconds or so. Is this the expected result?Hey Panos,
Ubuntu can be a little tricky as I believe they have a couple of different config files. The easiest way to tell is to do the SD card test for sure.
If you put a SD card in the slot does this behavior stop? If it does that means the change is for sure *not* working. It might not stop since rcu_preempt can be other things (it’s the realtime scheduler) and usually the usage is constant and doesn’t really go away.
My guess is that is the case and that putting in the SD card won’t change this behavior. If you take out the line from your config.txt file the original behavior should return (which has a fairly constant usage, it doesn’t go away and kind of stays at 1-2%ish because it’s polling the SD card slot hundreds of times per second).
If you find out that this config.txt change isn’t working you may need to use the file:
/boot/firmware/usercfg.txtin Ubuntu 20.04. I believe making changes in the main config.txt get ignored on that version (or get replaced often by updates) but I would say try using usercfg.txt if you determine it’s not working. This might not be having an impact right now and the change might be working but I think next time the boot files are updated the change will be lost so I thought I’d add this either way. Let us know what you find!
So I made another test: I made a new fresh Ubuntu 20.04 setup on my SSD (thanks for the guide btw!) and I took a look again at the rcu_preempt process without changing anything. The process was there but 0% CPU, didn’t even go up to 0.3% as before. I put an SD card while running top, and no behavior change..
Hey Panos,
Thanks for the followup, that’s very interesting and thanks for your kind words about the USB booting guide! It’s possible the explanation is that they may have finally patched this on newer kernels to not check so often (at least for Ubuntu). I’ve been waiting literally *years* for this across multiple Pi generations! This was even a problem while USB booting all the way back on Pi 3B models (and others that supported native USB booting before the Pi 4 came out which didn’t have it at launch but does now).
Everything sounds okay with the process still being present as it’s used for other low-level hardware interrupts. Activity in that process is often the Pi talking to your Linux OS at a very low level behind the scenes via the scheduler asking “was a SD card inserted yet”? And then about 1 ms later (or maybe even more often than this is my understanding) it again asks “was a SD card inserted yet” over and over and that is what is generating the activity this guide addresses but other low level activity that uses the scheduler can create activity as well. That is why putting in the SD card stops it because the Pi finally says “YES IT’S IN HERE SHUT UP!” and it does!
I believe this issue would be a kernel level patch to address so it may just be in the newer LTS version of Ubuntu. Last time I checked the fix hasn’t made it’s way to Raspberry Pi OS yet but maybe this is a fairly recent development so I’ll have to check for it the next time I’m USB booting one.
Thanks for the heads up! I think people on Pi OS (for now) and 18.04 are probably likely to still encounter this as if the fix is ever backported that far it will take quite a while to get there. I’m glad you were able to get yours resolved all the way down to a very negligible amount!