Raspberry Pi 4 USB Boot Config Guide for SSD / Flash Drives

Hey wyuenho,

That is fantastic information! If it says it’s going to take 20 hours don’t bother. That is running *extremely* slow to put it mildly! The score is going to be like < 200-300 points I'm certain if it's taking that long. The power hungry and ICY Box also strikes me as correct. I've always had to use a powered USB hub with my ICY Box pretty much. The most power efficient ones I've found are the ones that look like little USB sticks and are basically just a barebones PCB with a M.2 slot and a USB port on the end of it. It makes sense these would use the least as they often don't even have status indicator lights and all the circuitry is minimal. An even worse one I covered was the ASUS Rog Strix RGB enclosure which uses absurd levels of power but is actually really nice for thermals/cooling. I actually really love this enclosure. I originally bought it because of how stupid I thought it was putting user-controlled RGB in an NVMe enclosure and wanted to test how ridiculous the power requirements would be (exactly what you’d expect) but I know it will always need a hub when I use this one.

Do you have access to a powered USB hub? If it’s showing up as UAS but running this slow then it is very likely power related. I’ve seen this behavior on NVMe running in lower power modes. Even doing a cd will have a several second (or longer) delay sometimes when it’s in this state. I think you’ve probably got the adapters in fine shape and they may just need more power. If you plug them into a PC I bet it’s lightning fast as well.

You have enough hardware coming in that for sure something like the ~$20ish Sabrent powered USB hub (2.5A of dedicated power) would be worth it to test a bunch of scenarios with these. I have personally used and powered every drive I’ve thrown in it off of that. It’s also very physically small, or a lot smaller than what you would think when you hear “powered USB hub”. If you have any of these around it would be great to try for a test on all of these. I am fairly certain your firmware is in tip-top shape and that some of these are bumping up against power (like you’ve said with the ICY Box).

There are different power failure levels if that makes sense. You have won’t even boot but there’s a bunch inbetween that. In fact it sounds like a lot of low power states are working on yours that I wouldn’t even necessarily expect to work on the Pi. It’s definitely not hitting the high performance states here though it doesn’t sound like by the time length of that benchmark!

Oh wow, that looks much better and tells us a lot! It actually isn’t bad at all. I looked through your benchmark and it turns out your adapter does report power levels which is pretty cool:

Device-1: 2-2.1:4 info: Realtek type: Mass Storage driver: uas interfaces: 1 rev: 3.2
speed: 5 Gb/s power: 896mA chip-ID: 0bda:9210 class-ID: 0806 Hub-6: 3-0:1 info:

Here we can see that during the benchmark your drive was pulling 896mA of power. This I would say is definitely not P0 (the highest performance power level) but is actually still a really respectable score! If you click on the 980 Pro brand you can see that 9000-9500ish is about as good as it gets on a Pi 4 without jumping to a CM4 and using real NVMe through a PCIe 1x adapter or something like that.

I checked several other adapters and they all had the same power reading so maybe this isn’t actually a reading (or it’s not very accurate, I’d never noticed this reading so I’ll have to look into that more as I checked 3-4 of them and they all said the same thing so that seems like it may be an estimate or a static value perhaps).

The btrfs is really cool. I wonder if that has any performance overhead. I know it has a ton of incredible features. I can actually see that here:

fs: btrfs block-size: 4096 B dev: /dev/sda maj-min: 8:0 label: MobileCat
uuid: 8fc20653-6244-X-X-XXX

I’d be really curious to see a comparison if you do end up trying a powered hub on any of these enclosures. For ones like the ICY Box I think it would definitely be worth it. I am betting that the 980 Pro can do more in there as well but honestly depending on what you’re doing you may be totally happy with it. It’s still probably 7-8x faster than even the fastest SD card benchmarks even without hitting P0 state.

The only indication that anything could be improved is just that the score is a little on the low end (if you go to the “latest” page there were some 980 regulars a few benchmarks behind yours that did 7800 and 8800 and your drive is far superior) but that could be due to btrfs overhead (or maybe it’s faster and with more power it will be 10k+, I’m not sure in this case).

It would be interesting to see some comparisons with your other enclosures using the same drive as I know you said you had a bunch of them you recently purchased. You should be able to just swap the module to a different enclosure and just boot (or at least I’ve always been able to). I wonder if the more power efficient enclosures will give higher performance scores (or lower). Everything looks good overall!

Hey wyuenho,

Great testing! You actually answered my burning question which is what effects does btrfs has on performance. This also led me to fixing any benchmarks that were submitted when using BerryBoot (the drive was showing up as none for some btrfs submissions and after investigating I was able to correct the parsing of that) which solved about 112 mystery submissions for me and fixed them!

Your power draw numbers sound right to me for sure. The 980 Pro has a fantastic power controller and has many additional power states that other drives don’t have at all. I think this is a great choice.

The Sabrent hub has come in handy for me for tons of things for sure. There are other situations on the Pi where it really comes in handy to have such a small unit (not much bigger than the Pi itself) that can independently power accessories on the Pi. I’ve been able to test countless things I wouldn’t have been able to try and probably would have forgot about if I didn’t keep one of these around for sure.

I definitely think the ICY BOX firmware fixes would be worth a try. The main reason is what I was talking about earlier with the 980 Pro having a bunch of new technology and power states onboard. These fixes likely either fix issues with those power states or enable them to work in the first place which will definitely only help for sure. I will make sure to install these on mine as I had not seen these ones before.

Everything is looking great (other than the few lingering issues you’re chasing down with the various firmware versions)! Your latest submission scored 8,488 which is right on the money for USB 3.0. The only way to break into those really high scores of 10k+ is to jump to true NVMe or a higher USB revision than currently comes on the Pi 4 but comes on some CM4 IO boards.

Thanks a ton for sharing your findings. I’m sure they will help others and I personally learned several things!

Hey wyuenho,

Great additional findings! I just wanted to chime in with some pibenchmarks data to illustrate some of your points for sure.

Check out all the ICY BOX submissions with this search: Search IB-

Most people put their full model numbers which really helps get an idea of performance gaps / averages / etc.

For the RTL9210BN we can use this search. You can already see a bunch of high scores pop out on this adapter in the 8000-9000 range. It’s honestly almost about 50% are over 8000.

If we compare that to the ICY BOX there are a lot more ICY BOX submissions but only 2-3 of them ever broke 8,000 points.

For the VB series we had less data but it looks like someone did 4 benchmarks on it all at right around 8300 (very good). This isn’t really enough to draw many conclusions off yet though for the VB series as we’d like to see some other drives / adapters using that chipset submitted for that.

Basically the data does seem to support your conclusions. It definitely comes in handy to check some of this stuff as you can pretty much see it over all the benchmarks.

The only thing I would say is that the variations here are minimal between most of these. The reason for that is that the bottleneck is USB 3.0. You’re basically completely maxed out on the USB 3.0 bus with even a mid-range drive which is generally why I personally don’t worry too much about the minor variations as long as there isn’t a major compatibility issue.

What do I mean? Observe Pi Benchmark #48648 by myself using the CM4 with a PCIe 1x adapter (not using USB):

This was done by me in August of last year for my PCIe 1x NVMe on Raspberry Pi?! Compute Module 4 Guide.

Notice that the score is 20,641. That is using a several generation old Samsung 960 1TB (the EVO version even) and I am absolutely curb stomping 980 Pros all day by a factor of 2-2.5x. That is the bottlenecking for sure and is basically a “wall” for people over USB 3.0 as you are hitting the limitations of USB 3.0 themselves.

This is to illustrate / put in context how bottlenecked your drives actually are on the USB 3.0 bus on the Pi 4. Essentially any performance differences between the adapters are due to really strange firmware and compatibility issues as well as better power management and support.

If it is functioning correctly it doesn’t matter whether it’s a 6G/sec enclosure or a 10G or any of that because the Pi 4 only uses USB 3.0 and doesn’t use USB 3.1 or any of those features (and can’t, not on the Pi 4 at least, probably the next Pi will I think we all expect).

As far as actual raw performance goes you are completely bottlenecked and when you bypass this with the CM4 you can see how bad it is. I’d say any adapter scoring in the 7000-8000 range is doing just fine taking into account the bottlenecking. If you go to the “Top Scores” page you can see people who curb stomp anything I’ve ever done on the CM4 by putting things like the 980 Pro in it and absolutely 3-5X’ing the fastest USB 3.0 tests ever done if that makes sense.

Basically if extremely fast storage is a requirement the Compute Module 4 is what you should be using right now because the numbers that the Pi 4 can put up are a joke compared to a PCIe 1x passthrough like I am using in my article I linked. It’s annoying to have to get set up for it the first time but the numbers to make it worth it are more than there for the people who need that level of IO performance and I have some guides available for it to help get started at least.

It would be far more cost effective (and higher performing) to invest in a CM4 setup than going too crazy chasing the various enclosures as by going that route you are limiting yourself to speeds that a mid range M.2 SATA can hit over USB 3.0. Just today someone used a M.2 SATA Western Digital Blue drive and hit 8200 which is pretty much right at the USB 3.0 “wall” and he’s not even in the same league as NVMe with that drive. The enclosures are expensive and it will only take 2-3 of the spendier ones to make up an entire Compute Module setup (saying this from experience).

Note that this is information is for people chasing extreme performance. I know a lot of Google searches lead people here and for people looking for that I wanted to put into context that it actually doesn’t make sense to use any of these enclosures or the Pi 4 at all for extreme high performance scenarios and that the CM4 is multiple times faster for that and costs about the same as a Pi 4 setup from scratch but gives multiple times the performance.

To explain briefly what a Compute Module 4 is it’s basically a Raspberry Pi 4 that you get to choose your own motherboard for (called the IO board). That means we aren’t limited to what capabilities the Pi Foundation puts on the current/next Pi. Whoever makes the IO board decides what capabilities to put on it and since the Pi is running Linux it easily supports a very surprisingly wide range of hardware (it seems like almost anything I or other people try).

These can and do look like regular Raspberry Pis sometimes and lots of companies make all shapes and sizes of boards with varying capabilities for different purposes. The compute module just “snaps” in and out of those boards (there’s no soldering, it’s easy, you gently press and it clicks into place).

You do need a power adapter (12V recommended for NVMe and is an option for Compute Modules but 5V works too) and an IO board to place the Compute Module into (there’s a few starter pieces you need basically) but the overall cost is pretty comparable to a full Pi 4 setup (and cheaper after you buy things like the development IO board which you would presumably only purchase once). There are IO boards with higher version USB ports like 3.1 or 3.2 out there as well and that would solve a lot of these problems and greatly enhance your existing adapters as well.

I spent ~$120 or so for everything including the Compute Module itself (which like the Pi is available with multiple memory configurations or with the CM4 even eMMC and optional WiFi modules). Not too far off from starting fresh with a Pi 4 4GB-8GB and needing the power adapter, SD card and other accessories. I also got one of the larger memory sizes like I usually do with my Pi 4s as well as a built in eMMC and WiFi module so you can lower the cost substantially by opting out of some options I chose.

When you’re putting a 980 Pro in it then that drive is worth 2.5-3X an entire combined Compute Module 4 setup and like I outlined you would go from 6500-8500ish to undoubtedly higher than my 960 EVO score of 20,000 at least. She’s worth it I’d say and you can actually leverage this performance with a real official Raspberry Pi (just not the Pi 4) in the same or even better form factor (depending on what kind of IO board you want). I wouldn’t blame you for being happy enough at this point though but I know what that drive can do (it’s my main rig drive I’m using as I type this) and this type of setup is holding it back a lot because of the USB 3.0 bottleneck issue. It’s a bit like having a Ferrari and never exceeding the speed limit and running it on unleaded gas instead of premium when the Ferrari really just wants to go fast to be happy.

The leap would completely swamp any btrfs performance penalties as well. That took you from 8500 to 6500 very roughly or a 24% drop. My official on-the-record guess is that if you do try this (don’t feel pressured to if you’re happy with the setup, that is what counts) you’d get roughly the 23k of the benchmark I linked to with the CM4 980 Pro subtracted by 24% gives us… 17588. That is with btrfs enabled. Obviously this is just an educated guess for fun as it could end up that you’ve already “paid” the performance penalty for btrfs and all of the gains are just gains and the overhead won’t scale with the increased throughput. I divided that number by 6500 to get the performance increase and it would be a 2.70x increase.

Thanks a ton for the additional info / links and documenting this somewhere on the internet!

Hey wyuenho,

Absolutely. I deal with this issue constantly. If it shows up on my storage benchmark web site then I’m detecting it (not always possible). That is a fully automated parsing engine. It’s usually pretty easy but the more generic and cheap the adapter or storage controller is directly related to whether it supports hdparm, smartctl, lshw, smartctl, nvme-cli and all the other methods to identify drives and adapters. Take a look at my MySQL backend here:

4

As you can see here the Realtek adapters are identifiable by nearly every test. Including the CG part. You have a lot of options and depending on what your drive manufacturer implemented all of them or none of them may work. I’ll often detect one with inxi and then another adapter will show up fine on udev or hdparm. Blocking hdparm is especially annoying as it makes it much less likely to identify the actual drive.

The Realtek ones are annoying to deal with sometimes because they are *too* gung-ho about reporting what they are. They often report themselves instead of their drive like you can see my hdparm tests are not finding the drive and are just finding the Realtek adapter again. That’s why they’re showing in this view. I actually can’t identify these outside of the chipset which is why they’re stuck.

Most of them are fully compliant with HDParm and will allow me to identify not only the adapter but the drive itself. The above screenshot is basically what I see when I’m working on the site. It’s pretty easy to spot new models and adapters but the worst offending most generic adapters will sometimes literally have that manufacturer field as “EnterManufacturer” or some placeholder template.

It’s basically directly related to how crappy your adapter is whether the manufacturers filled out these fields a lot of the time. If they didn’t then you shouldn’t expect them to have taken much care making sure the adapter is widely compatible. What are the chances they took every step to ensure TRIM and UASP would work properly across all devices? Very low. These are the most common adapters to have major issues.

If someone tries to manipulate these fields it basically wouldn’t be parsed and would be “stuck” until I update the parsing engine to parse that entry. It’s extremely rare people try this but a few spammers have tried spamming it with fake entries before and I have plenty of ways to deal with it but it’s quite rare.

Fascinating stuff, I wanted to show kind of how it worked on the backend. It’s probably smarter than people think. If people think I’m guessing or sitting here all day putting storage benchmarks on there one at a time I’d be ruined doing that for about a day (nor would I ever agree to it). It’s real basically. It’s literally all automated and parsed.

I don’t think there’s another site like it out there as I coded it from scratch. The benchmarks go all the way back years to something like 2019 and this has been a years long project. There’s only a few similar sites for PC and they are pretty crappy looking compared to mine, almost never have pictures, etc. Cheers!

Hey wyuenho,

Great questions, I’m glad you got to take a look at the script as well! There’s no logic in the script to detect it for sure. It’s all on the web site backend to detect the drives.

I’m able to detect and identify *most* drives over the Realtek adapters with about a 12% failure rate. Here’s what it looks like:

Here I just did a query for ‘data_udev_model like “RTL%”‘. If the “Brand” field is blank that is an unidentified benchmark. I was only unable to identify 43 out of 319 submissions. This is sometimes due to really old / bad firmware on the device or probably more likely they are using a drive that doesn’t implement hdparm (as in the drive itself doesn’t have anything to answer the identify command with or returns garbage which I see a lot) but if it did it could be passed through. That’s not to say it’s not a problem. That’s a 12% failure rate which is among the highest for any adapters on the site so these ones definitely do cause me some headaches.

hdparm is the only test that seems to really pick up anything on the Realtek. This is basically a hdparm -Ii /dev/sda2. This communicates directly with the drive to retrieve the information in the drive. Honestly it’s pretty similar to “undocumented magic” for sure and not all drive manufacturers implement it / support it. It does have nvme-cli parsing in there as well (if it’s available) although not many people have implemented true NVMe on the Pi that can be picked up with this tool unless they are using the CM4 so I only have a few benchmarks that use it.

A lot of the PC and other device benchmarks people have submitted have valid nvme-cli tests though and eventually as I keep improving it I would like to open it up to other devices. It has been a pretty overwhelming project though just to parse something like 32,000 identified tests. I cannot believe how many SSD manufacturers there are that I have never heard of out there. I’ve apparently individually identified (and classified so they’ll parse on the site) 1159 individual models of SSDs, SD cards, etc.

To give an idea of how it’s working here’s some higher level code:

function updateSubmission(&$pdo, &$stmt, &$value)
{
saveOldVariables($pdo, $stmt, $value);

updateChanges($pdo, $stmt, $value);
}

The juicy stuff looks more like this:

function parseBrand(&$pdo, &$stmt, &$value)
{
$value['model'] = str_replace(' ', '', $value['model']);

if (!empty($value['manufacturer'])) {
switch (mb_strtoupper($value['manufacturer'])) {
case 'SAMSUNG':
case 'CORPORATIONSAMSUNG':
case 'CORPORATION SAMSUNG':
parseBrandSamsung($pdo, $stmt, $value);
break;
case 'SANDISK':
parseBrandSanDisk($pdo, $stmt, $value);
break;
case 'KINGSTON':
parseBrandKingston($pdo, $stmt, $value);
...


function parseBrandKingston(&$pdo, &$stmt, &$value)
{
$value['manufacturer'] = 'Kingston';

if (startsWith($value['model'], 'SA400S37')) {
$value['brand'] = 'Kingston A400S Series';
$value['approved'] = 'true';
$value['class'] = 'SSD (2.5" SATA)';
$value['image'] = 'Kingston_SSD_A400S37.jpg';
} elseif (startsWith($value['model'], 'SA400M8')) {
$value['brand'] = 'Kingston A400M Series';
$value['approved'] = 'true';
$value['class'] = 'SSD (M.2 NVMe)';
$value['image'] = 'Kingston_SSD_A400M8.jpg';
} elseif (startsWith($value['model'], 'SQ500S')) {
$value['brand'] = 'Kingston Q500 Series';
$value['approved'] = 'true';
$value['class'] = 'SSD (2.5" SATA)';
$value['image'] = 'Kingston_SSD_A400S37.jpg';
...


That’s pretty much how the parsing logic works only extended by tens of thousands of lines. As far as the main parsing logic it looks more like this (to try to parse the drive info in as many ways as possible like we were discussing):

function parseSSDInfo(&$pdo, &$stmt, &$value)
{
if (isVendor($value["data_hdparm_vendor"]) && isModel($value["data_hdparm_model"])) {
$value['manufacturer'] = $value["data_hdparm_vendor"];
$value['model'] = $value["data_hdparm_model"];
} elseif (isVendor($value["data_hdparm_model"]) && isModel($value["data_hdparm_vendor"])) {
$value['manufacturer'] = $value["data_hdparm_model"];
$value['model'] = $value["data_hdparm_vendor"];
} elseif (isVendor($value["data_lshw_vendor"]) && isModel($value["data_lshw_model"])) {
$value['manufacturer'] = $value["data_lshw_vendor"];
$value['model'] = $value["data_lshw_model"];
} elseif (isVendor($value["data_udev_vendor"]) && isModel($value["data_udev_model"])) {
$value['manufacturer'] = $value["data_udev_vendor"];
$value['model'] = $value["data_udev_model"];
} elseif (isVendor($value["data_nvme_vendor"]) && isModel($value["data_nvme_model"])) {
$value['manufacturer'] = $value["data_nvme_vendor"];
$value['model'] = $value["data_nvme_model"];
} elseif (isVendor($value["inxi_vendor"]) && isModel($value["inxi_model"])) {
$value['manufacturer'] = $value["inxi_vendor"];
$value['model'] = $value["inxi_model"];
...


This is basically why there’s nothing in the script as it’s all done on the backend pretty much. Every test I basically try to parse every individual identification method available. You pointed out that lshw and udev basically have the same “sources” for the drive identification which is right. The reason I run all the tests anyways is that not every test succeeds on every system. There’s a lot of systems that will have udev and then not have lshw and fail to fetch the dependencies for it. When this happens I do everything I can to still identify it even without all the potential tests if that makes sense.

Another test I run but don’t parse yet is smartctl. I’ve seen submissions that I wasn’t able to identify through hdparm, lshw, or any other regular means that did have smartctl populated. This seems to talk directly to the SMART interface of the drive but again is definitely not implemented on all drives and some adapters will not pass/communicate this information. That’s on my todo list.

The reason I do it this way (as in parsing on the backend and not within the script) is it’s very easy for me to fix models if I make a mistake or figure out a way to identify more information I wasn’t able to parse before. I can regenerate the entire web site and the 30,000+ benchmarks in a couple of minutes from scratch by reparsing the tests with the new “parsing engine”. People have let me know I’ve miscategorized things before and I can retroactively fix it very easily which if it was all parsed by the script would be a very big problem if that makes sense. The way it is now I just tell it to update *all* submissions again and it will basically update everything with my latest changes.

You’re completely right though about the adapter IDs and everything you’ve said for sure. I think it’s really hdparm that has saved my bacon. This still seems to work through most NVMe to USB adapters or my site would have virtually no NVMe benchmarks on it I’d imagine (or the closest I’d be able to get is just identifying the adapter or even worse just the chipset). The individual drive magic is interesting. I almost wonder if I should try to implement some of the ones for the bigger manufacturers and see if it’s possible to scrape that data. Very interesting ideas!