I’ve covered how to get the right type of storage adapter for your Raspberry Pi for years on this site and cataloged storage adapters that both work and don’t work with the Raspberry Pi. Over the years we’ve learned that many of these adapters can be “fixed” with a firmware update to work with the Raspberry Pi.
In this article I’ll put together an evolving list of firmware adapters that can be fixed with these updates from my own experience as well as comments people have left over the years!
Overview
The preferred and safest way to identify your device is by brand name. This will work if you have a “popular” or “name brand” storage adapter.
If you have a generic / unbranded adapter then the next best way is by chipset. We can identify your chipset by using the following command:
sudo lsusb
This yields the following result:
pi@pi:~ $ lsusb
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 04d9:0007 Holtek Semiconductor, Inc.
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Yours will look similar to mine. I’ve bolded the important line which is our storage adapter. The above result is for the StarTech 2.5″ SATA to USB 3.1 (USB312SAT3CB) adapter. This example is a name brand adapter that will be on the list but if it wasn’t the generic chipset would be the ASMedia ASM1153E chipset for this adapter. Other common chipsets include JMS-578, etc.
If you’re confused about which is which use this version of the command to get a lot more detail (including device properties that often make it much easier to identify):
sudo lsusb -v
This will yield something like this:
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.10
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 9
idVendor 0x174c ASMedia Technology Inc.
idProduct 0x55aa Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
bcdDevice 1.00
iManufacturer 2 asmedia
iProduct 3 ASMT1051
iSerial 1 123456799FA6
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0079
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xc0
Self Powered
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 8 Mass Storage
bInterfaceSubClass 6 SCSI
bInterfaceProtocol 80 Bulk-Only
iInterface 0
The above example didn’t give us the name “StarTech” anywhere but it did give us some clues. This is identified as a “Mass Storage” interface class device which definitely narrows things down. Your other peripherals will show as the category they are from like mouse, keyboard, etc. You should be able to narrow things down by unplugging everything else from your Pi if you are still having trouble identifying which is which.
Many of these updates need to be applied using a Windows machine as that will be the only platform these updates will be offered on from their web site. Some manufacturers have update utilities for multiple platforms available but from what I’ve found if you’re lucky enough they offer them at all it will usually be for Windows.
Warning / Disclaimer
This is not an entirely risk free procedure. If something goes wrong during a firmware update it is possible to brick it. This doesn’t happen very often but understand it’s possible. If you lose power at the moment you are updating the firmware for example that could definitely do it.
There is less risk for the “branded” adapters as these are the manufacturer’s tools intended for the manufacturer’s devices. It’s as safe as it gets but even in these cases things can go wrong (like the examples I mentioned above). There is also some risk that even chipsets identifying as the same chip may have slight variations in how they are actually implemented or which revision they are.
Make sure you understand this is not a completely risk free procedure (and carries the same risk as firmware updates on any other device, and a little bit extra risk even for the generics since they may not have been intended for that exact device) before proceeding!
StarTech Adapters
StarTech 2.5″ SATA to USB 3.1 Adapter
StarTech 2.5″ SATA to USB 3.1 Adapter
The USB 3.1 variant of the StarTech 2.5″ SATA adapter works well with the Pi 4. The USB 3.0 variant doesn’t have firmware updates available and is not recommended.
usb3s2sat3cb – USB 3.0 Version – No updates available
Click the “downloads” tab and choose “firmware.zip”. This must be ran on a Windows machine. It will update the firmware almost instantly when you launch the program with the adapter plugged in and say “SUCCESS – UNPLUG AND REPLUG THE DEVICE”.
Sabrent Adapters
Sabrent and Orico both have the worst track records for working storage adapters for the Pi. I don’t recommend them at all but they can sometimes be fixed.
The following Sabrent JMicron adapters can be updated with their official tool:
Note with generic adapters there is some risk. These may not necessarily be by the same manufacturer of your device. It usually doesn’t matter as these all have the same storage controller but due to slight variations in the way some manufacturers implement this technology it’s possible it could cause an issue / brick it. Make sure you understand that there is some risk before proceeding!
JMicron JMS578 Firmware
This is a copy of the JMS578 firmware that has fixed this issue for many people (but not everyone) on the Raspberry Pi. You may still need to enable “quirks mode” (see quirks mode section) even with the updated firmware in some cases.
It’s a little bit trickier to use than some of the other ones but not too extraordinarily difficult. You will need the updater utility and the .bin file.
This thread is worth a read as well to see the different types of adapters/chipsets people tried with this and their different results: Raspberry Pi Forums – Topic 245931
Watch Out For Power Issues
If you are using a drive that has high power demands a common solution I’ve been recommending for years is to use a Sabrent powered USB hub to power the drive. This eliminates your Pi from having to use it’s own power to power the drive at all. This is often required for higher performance NVMe drives.
Sabrent Powered USB 3.0 Hub
The Sabrent powered USB hub delivers a whopping 2.5A of dedicated power for your USB attached devices. This is almost as much as the Pi adapter itself is rated for (3.0A). It will easily power the most thirsty of setups such as NVMe enclosures.
Note: Make sure Amazon doesn’t try to take you to the non-powered version and that it’s the one with the AC adapter that plugs in to provide extra power
Verify Drive Performance
You can make sure everything is running correctly (and as fast as it should be) by running my quick storage benchmark. You can run the benchmark with the following one-liner:
This will give you a score you can compare to the other Raspberry Pi Storage Benchmark results and make sure that you are getting an equivalent speed to your peers with the same device!
Benchmarking / Testing Storage
If you want to verify your drive’s performance you may want to run my storage benchmark with:
If you search for the model of your drive on Pi Benchmarks you can compare your score with others and make sure the drive is performing correctly!
Fix (some) USB Adapter Problems Using Quirks
Some adapters can be made to work by using USB quirks to disable UAS mode on the drive. This lowers performance, but it’s still much faster than a SD card and your adapter won’t go to waste. Some adapters also require it even with the updated firmware!
To find out the quirks we need to find the device ID string for your adapter and then add an entry to cmdline.txt telling the kernel to apply them on boot.
Find Your Adapter
To apply the quirks we first need to get the adapter id. We will use the sudo lsusb command:
$ sudo lsusb
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
On line 2 we can see my ASM1051E SATA 6Gb/s bridge adapter (it’s the known working StarTech.com 2.5″ SATA to USB 3.0* adapter). You will see something very similar to mine when you run the command and it shouldn’t be too hard to figure out which device it is. If you need more information add a -v switch to make the command sudo lsusb -v. This can sometimes add some additional details to make it easier to figure out which one is your adapter.
If you’re still not sure, we have another command that between the two that can narrow things down. Type / paste the following:
sudo dmesg | grep usb
[0.828535] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
[0.828568] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[0.828597] usb usb3: Product: DWC OTG Controller
[0.828620] usb usb3: Manufacturer: Linux 4.19.75-v7l+ dwc_otg_hcd
[0.828644] usb usb3: SerialNumber: fe980000.usb
[0.830051] usbcore: registered new interface driver uas
[0.830182] usbcore: registered new interface driver usb-storage
[0.836488] usbcore: registered new interface driver usbhid
[0.836511] usbhid: USB HID core driver
[0.971598] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[1.154217] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.20
[1.154254] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[1.154281] usb 1-1: Product: USB2.0 Hub
[1.301989] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[1.332965] usb 2-1: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00
[1.332999] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[1.333026] usb 2-1: Product: ASM105x
[1.333048] usb 2-1: Manufacturer: ASMT
[1.333071] usb 2-1: SerialNumber: 123456789B79F
This is the dmesg log showing the hardware detection as hardware is activated on the Pi. If your log is really long you can generate fresh entries by just unplugging a device and plugging it back in and running the command again. Here we can clearly see that the ASM105x is what our StarTech adapter is being detected as.
Now we can go back to our first lsusb command and we want the 8 characters from the ID field that comes right after the Device:
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge
Our adapter’s ID is: 174c:55aa
Applying Quirks
To apply the quirks to our USB adapter we are going to edit /boot/cmdline.txt. Type:
sudo nano /boot/cmdline.txt
We are going to add the following entry into the very front of cmdline.txt:
usb-storage.quirks=XXXX:XXXX:u
In place of the X’s above you will put in your adapter’s ID that we got before. With the example commands I gave above mine would look like this: usb-storage.quirks=174c:55aa:u. After this my cmdline.txt looks like this (everything should be one continuous line, no line breaks!):
Now reboot the Pi. If the Pi fails to boot you can plug the SD card into the computer and go to /boot/cmdline.txt and undo the change we did so you can boot back in with your SD card.
Verifying Quirks
Once you have rebooted after changing cmdline.txt we can verify the quirks have been applied by doing another dmesg | grep usb command:
sudo dmesg | grep usb
[1.332924] usb 2-1: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00
[1.332957] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[1.332983] usb 2-1: Product: ASM105x
[1.333006] usb 2-1: Manufacturer: ASMT
[1.333028] usb 2-1: SerialNumber: 123456789B79F
[1.335967] usb 2-1: UAS is blacklisted for this device, using usb-storage instead
[1.336071] usb 2-1: UAS is blacklisted for this device, using usb-storage instead
[1.336103] usb-storage 2-1:1.0: USB Mass Storage device detected
[1.336479] usb-storage 2-1:1.0: Quirks match for vid 174c pid 55aa: c00000
[1.336611] scsi host0: usb-storage 2-1:1.0
This time we can see in dmesg that UAS was blacklisted for the device and it has loaded with the usb-storage driver instead. This driver tends to be more compatible with the “problematic adapters” but the performance is usually significantly lower. It’s definitely worth a try though as some adapters do better with the quirks performance-wise. The only way to know for sure is to run a benchmark (see “Verify Drive Performance” section).
I am testing several times INTEL 760p 128GB NVMe on AXAGON GTX with Pi4 4GB, Transcend 120GB mSATA in EVOLVEO USB box and old mSATA 64GB from the computer in mSATA 3GB speed USB box. Just to add, that all tests in variants with/without USB powered HUB. For some combination it does not work with USB HUB even just because of USB-C cable. Finally should be the difference for Pi4 new power version (related with Bullseye upgrade info). I am recomend to add some possibility to comment the results some time later – if the configuration works stable or not. For me – only old version of 3GB speed mSATA USB box works perfectly with mSATA disk, now testing Transcend in EVOLVEO looks OK for just short time now.
Great feedback! I actually really like the idea. It should be theoretically possible. The site would need to have some kind of account system created but each Pi and SSD/HDD has a serial # so I’m envisioning that I could let you “claim” old results under your account if you do a new benchmark with that Pi so it can read the serial #. Maybe something like that (all really rough tentative ideas).
The powered USB hubs can be tricky for sure. Which hub did you end up going with? One trick I often use is that the Pi won’t turn on when connected to these powered USB hubs sometimes. If you disconnect the USB cable to your powered USB hub and then connect the Pi it should power on then plug in the USB hub immediately as fast as you can before it starts the boot process. This works around a lot of weird USB hub issues (related to booting at least).
I had not heard of the AXAGON GTX adapter before but I do see it in Google. Would you be willing to post a lsusb from that adapter? I’m curious what storage chipset it’s using inside (could be a JMicron chipset such as a JMS567, or an ASMedia one as well). Have you checked to see if there’s any firmware updates available for it? It sounds like you may have things working stable here though for the most part but it may be worth checking. Thanks again for the feedback!
Hi James,
GTX is my mistake…AXAGON EEM2-GTR is the one I had been testing.
You can see I did a lot of tests 🙂 as I mentioned. I am really flustrating when realised today, that some of them significantly influenced results just because of USB 3 cable used to connect Pi4 with powered USB HUB. The topic is visible between Test #53,476 and Test #53,472. The difference just USBA-C cable. The result is great, but Pi4 stop working after some time even with good cable 🙁 anyway.
Most of the times I tested even Quirks solution, but with no serious difference in stability.
I changed my Raspberry Pi4 for new 4GB version with updated power part, therefore repeated tests if the solution exist.
And last but not least… I realised, that I have better results with powered USB HUB where both Pi4 & USB BOX are connected as “the equipment” in the box, means not used main USB connector… have no idea why. I am still looking for the final configuration where it is no needs for powerd USB HUB. Expect you do not need lsusb for AXAGON GTR :-).
Regards!
Ivo
Fascinating information, thanks for the followup! That is amazing that the cable made that much of an impact. Another really weird thing is that USB 3.0 literally causes RF interference for WiFi. I’ve actually encountered this before when we had someone who had their mouse adapter stop working due to the interference from USB 3.0. USB 2.0 has *no* interference strangely enough!
For the lsusb command it just spits out which devices are connected to your system. The compatibility with the Pi is actually impacted by the chipset that will show up in the lsusb tool. It’s not a driver or anything like that, it just gives you the real name of the storage adapter (AXAGON doesn’t make this piece of it, they buy it from someone else since only like 3-4 companies in the world actually make these types of storage controllers used in the adapters). It looks like it’s a JMicron storage controller in yours according to this result.
I wonder if your USB cable was one of the “charging” USB cables. You can sometimes get data over these at USB 2.0 speeds. At any rate, this is great information for people to double check as this gets overlooked a lot. Connecting everything to your powered USB hub without using the main USB connector is a really interesting trick to use as well, thanks again for posting it! Scores are looking really good as well!
I’ve found the point! As part of the optimization I upgraded my RPi from 2GB to 4GB. And I found out that my original RPi4-2GB has HW version 1.1. Maybe you know that this version 1.1 has a problem with the specification of the power part/USBC connector. That is, if I used the original USB3 cable, there was not enough power supply to work with the SSD.
Very interesting, great find! I do remember this actually at the time.
I had not read your source before and it actually paints a better picture of the issue I think so thanks for sharing it. I believe the old inventories of these are long gone but some of my early launch Pis are still this revision. This is a great thing to check so thanks again for posting it!
Danny
2 years ago
Thank you very much for posting this! The usb-storage.quirks addition worked well for my Sabrent EC-SSHD sata to usb adapter. Boot times went from ~ 1:30s to 0:18s using a WD 500Gb SATA III drive! Really appreciate your help!
Thanks so much for your kind words and a confirmation! I’ve been hearing mostly about problems (not the people’s fault, just the nature of comments a lot of the time) so I’m really glad to hear of a successful update / project!
Those gains aren’t just impressive, those are game changer numbers. It’s several times faster. Congratulations and enjoy!
Richard Hornby
2 years ago
Hi James
It worked 🙂 🙂 Hopefully this account will enable those who may have watched these posts with amusement / bemusement to take something useful away.
GOAL
1 To create a backup SD from an SSD which is over 32Gb in size, because it also contains an NTFS data partition.
2 To be able to update / upgrade astroberry without crashing the SSD.
METHOD
1 Turn this process on its head and create a base SD image which can be updated / amended and then deployed to SSD as required
2 Burn a new astroberry image to SD, apt-get update and upgrade, then install any extra components required (NTFS support is now included automatically). Install gparted.
3 Use gparted to shrink /root partition as required to create ~2Gb unallocated space after /root
4 Install and configure everything required on the SD eg timesync and WiFi
6 COPY THE SD TO USB using SD Card copier. Copier will report it could not mount one partition – ignore.
7 Shut down, remove SD card from pi, check it boots from USB.
8 Assuming all ok, shut down again, then plug in SSD, insert SD Card and copy SD card to SSD via SD card copier.
9 Shutdown again, remove SD card and boot from SSD.
10 The SSD should have /boot, /root and a large unallocated space. Use gparted to format the space as NTFS (or as you wish).
CONCLUSION
Now you have (i) a working backup SD in case of SSD crash, (ii) a backup / dev USB to update before deploying to SD/SSD, and (iii) a working SSD for day-to-day use.
One of my posts concerned updating the JMS 587 firmware driver. I was able to do this, but the drive seems still to require /boot/cmdline.txt to contain quirks.
Great work! I’m very appreciative to you for posting such detailed steps. It’s tough to tell exactly how many people are helped by these articles and comments but this article still gets around 300-400 hits per day and the SD bootstrapping one I linked to earlier still gets 1200 hits a day or so. I think it’s safe to say that the vast vast majority (99.99%+) don’t comment but that undoubtedly a lot of them do find useful information. Thanks for contributing!
I’d almost forgotten that we started with the JMS587 firmware updates. There are different revisions of this adapter out there and variations within the JMS587 family even so it’s not a 100% guaranteed fix (unless the manufacturer releases them). I still think this was a worthwhile journey for sure as I think that your end setup turned out really nice!
I wouldn’t sweat the performance difference too much as it is around maybe 700-2000 points or so of a difference depending on the drive. The fastest SD cards will barely break 1000 though and you benchmarked many times faster than that even with quirks enabled. This is still a very fast performance option even if it isn’t going to win the local storage benchmarking competition and is in an entirely different class of performance than the SD card!
I admire your patience and persistence throughout the adventure. Your willingness to fight with this for a week or so just to understand what is happening / knowing there is a solution reminds me a lot of myself. I personally do this all the time and it is definitely one of my “secrets” powering my IT background/career. You learn 50 different things you couldn’t have ever imagined learning before going down these rabbit holes. I imagine you have learned a ton from this, and I’ve actually learned a few new tricks from this conversation / reached a better understanding of some concepts as well.
Cheers and good luck with rest of the project that this is all powering, and thanks again for sharing your detailed solution!
Richard Hornby
2 years ago
Ok very useful post! The video essentially shows where I had got to with your help, but somewhere along the line I seem to have lost the PARTUUID references – or possibly the firmware,as I have all the correct references in the correct files 🙁
Is it possible to install the firmware to the SSD directly without booting from it first?
Here’s where I am:
1 Made a fresh astroberry image to SD. This boots. Got the PARTUUID’s with blkid and checked they are referenced in cmdline.txt and fstab. They are, and are different from the PARTUUIDs for the SSD partitions.
Changed the fstab and cmdline on the SSD to point to the SD for /boot and the SSD for root. No boot. Changed them back – ie left them pointing to the SSD PARTUUID.
Tried booting from SSD without SD – fails with a GRUB-type message ‘failed to find target’. I can’t log on with CTL-ALT-F2 so have left this for the moment. I can’t read the error messages.
There seems to be a full image on the SSD rootfs, but I’m not sure.
2 On the new astroberry image, I have updated the FW to latest stable version – I guess this should be OK. As I said, this boots fine.
3 I think my only remaining course of action, unless there is a way of installing the correct firmware to the SSD without booting from it (as above) is to image the new working SD to the SSD, which will remove the data partition, then install gparted, restore the data partition (and NTFS file support, and everything else I had!), and hope all works.
SO going forward … any advice on making backup images of only the boot and root partitions for the future? The video shows how to copy all the partitions on the OP’s installation, but he didn’t have a data-only partition.
Best
R
Richard Hornby
2 years ago
Thanks for the advice! There are reasons to stay with Astroberry, viz most importantly it comes with a built in VNC and INDI server and a bunch of bundled astro programs. I have added my own set of apps for my Pi and the whole set is saved on the SSD. One of the bundled apps is ‘aggressively’ 32bit so I need for the mo to stick with the 32 bit astroberry OS. Until the astroberry upgrade it was basically working.
I will download a fresh astroberry image to SD and see if I can at least use that to boot the SSD, but I am not holding my breath.
On another point .. I have found some sites on the web which advocate copying the SD image to the SSD. When I try to do that with the Pi4B it tells me I can’t as it doesn’t find the eeprom. Is it not the case that the Pi4B will still boot from SD even with the Raspberry-Pi OS? This is what the Pi.org site also suggests. If that is the case, then (1) aren’t all Pi4B users stuck with a 5 yr old process and (2) I’m not sure that loading the Pi OS will help all my issues, specifically how to take backups of a SSD which also includes a data partition. If there is a way to copy the SD to SSD and boot from that, I may have missed it.
I will take it in steps and get a new astroberry image and let you know what happens. I appreciate the help.
Great additional questions, I’m glad I wasn’t too off-putting with my last message and I think you’re on the right track here!
So in regards to SSD booting with AstroBerry you shouldn’t need to use the SD bootstrapping method. The Pi natively supports booting from USB now (if you have the latest firmware, which I doubt AstroBerry can update, you would need to update your firmware using the recovery image that the Pi Imager tool creates, more on that + a link in a moment). Here’s a bunch of people that have done it with AstroBerry.
They used BerryBoot as the bootstrapper if that makes sense but this is no longer necessary (according to people at the bottom of the post who confirmed this). BerryBoot is a distribution that supports booting multiple OS’s off a single storage device (perfect for SSDs which have room for many different installs). This may be of assistance for sure. People at the bottom of the post said that BerryBoot is no longer even necessary and you can image it directly to the SSD with Etcher now (using the latest Pi firmware of course) but this may destroy your other partitions so in that sense BerryBoot might still be easier for that setup. It’s specifically designed to “play nice” with multiple other partitions on the same storage device which is exactly the goal you’re working toward here.
Have you tried booting with the SD card taken out completely? This should work on a clean working install without the SD card inserted. The SSD should still have it’s own boot partition (but it may be damaged, maybe this is what the apt-upgrade messed up, it would explain several things). All of the magic happens in the /etc/fstab file. The catch is you need new enough firmware on your Pi to be able to boot (you should see a BIOS looking screen when it boots up, if it fails to boot it should try over and over, it should also try to boot off multiple devices like the SSD if the SD card isn’t inserted). I do have a guide on updating the firmware here: Raspberry Pi 4/400 Bootloader Firmware Update/Recovery Guide
There is an example of someone doing exactly what you mentioned and copying the SD card image to the SSD and they said it worked for them. This is definitely worth a try I think as well! I think any one of these methods should get it going. I would highly recommend creating the “Recovery Image” with the Raspberry Pi Imager tool to make sure to update your Pi to the latest bootloader firmware and make sure you are seeing the BIOS-looking-ish startup screen. You would only need to do this one time then you can go right back to AstroBerry permanently once it’s updated. Let us know how it turns out!
Richard Hornby
2 years ago
Hi James
I really appreciate how much you are helping me and I hope the discussion may be useful to others. I definitely want to get a strategy for being able to do n(1) a full system upgrade and (2) a backup of my SSD. I take your point about a compressed backup may be the way to go, but ATM am still stuck on no boot.
So I have concentrated on trying to sort out /etc/fstab – and /etc/.fstab which I also found.
Something odd seems to be happening. Somehow my PARTUUID for the SD card seems to have changed to match the PARTUUID for the SSD. If I just insert the SD and then the SSD into my Linux lappie and do a blkid on each, I’m getting the same PARTUUID. And /etc/fstab seems to have changed to point to that UUID for both /boot and root. Previous to what I can only consider as a change, the PI did boot, but to the wrong drive. Now it won’t boot at all.
Whatever, I think I may have to change the PARTUUID of the SD card to get anything to happen. I guess I would do that then alter fstab and .fstab accordingly on both SD and SSD, and then try again.
No problem at all. You’re basically dealing with how SSD booting on the Pi used to work 5 years ago so I’m familiar with exactly what you’re dealing with. In fact, I think one of my very, very old articles from before the Pi natively supported USB booting may be of assistance:
I actually cover how to change the PARTUUIDs, resizing partitions and everything in my very very old USB Booting for Raspberry Pi guide here. These instructions are completely obsolete for modern Pis. Nobody else is going through *anything* like what you’re fighting with using this AstroBerry distribution after whatever happened to it with apt. In other words, the Raspberry Pi supports NATIVE USB booting with the latest firmware and a compatible OS. If you are having to do it the way in the article I linked to above that is before the Raspberry Pi even supported USB booting that you had to worry about things like PARTUUIDs.
It’s not typical to have to do any of this. I would say your installation of AstroBerry has been broken ever since your apt upgrade and the whole thing should be thrown out. We know it has been broken since the apt-upgrade before which it was acting relatively normal (although not perfect). There’s nothing to save at this point, it’s just holding you back if you have a copy of the files you need. Even if you want to stick with AstroBerry I would install it clean and not run the apt upgrade and see if you can at least get it working as well as before.
You can literally just image a SSD with Raspberry Pi OS and it will just boot up. No SD card even inserted. Messing with things like the PARTUUID hasn’t been necessary in years.
I don’t know if this is an AstroBerry thing or if this is because of the apt dist-upgrade that got ran on it but this is a complete mess and none of this should be necessary. That’s why I suggested starting over as your boot files seem to have been modified by your apt dist-upgrade into something that isn’t compatible / booting correctly.
If you do start over I would use a different OS. It should take you no more than 15 minutes to completely download another OS and try it. I bet you it will boot right up. I’m not sure what else to tell you at this point though as literally the first thing I would have tried was reinstalling Raspberry Pi OS on it and after several days of back and forth I don’t think this drive has been reimaged with Raspberry Pi OS or even a fresh install of AstroBerry yet.
We can’t rule anything in / out at this point and you’re way, way down the rabbit hole at this point trying to track down partition IDs when frankly you just need to reimage the thing. Just the fact that it broke from an accidental apt dist-upgrade should have been an *automatic* reimage from scratch completely especially after several days of troubleshooting that is spinning it’s wheels. We have to start narrowing things down. You’re just guessing trying to avoid the one thing that makes sense to do in this situation: reinstall.
Your adapter might have compatibility issues with the Pi but we haven’t done the most basic tests to rule it out such as trying with Raspberry Pi OS or anything really to narrow things down. We’ve basically got nowhere for several days until we start getting serious with it and ruling things out / even trying different OS’s to figure out what we’re dealing with here. I can personally assure you we are not dealing with normal or typical behavior at all. The behavior before the apt-upgrade was a little bit more normal but I’ve never heard of AstroBerry and upon some Google searches I found lots of issues booting from SSD out there that implied to me that maybe the support is a couple of years behind and that it isn’t as easy/straightforward as something like Raspberry Pi OS, Ubuntu, etc.
It should literally take 2 minutes to image and boot up with a SSD if that makes sense! There’s a lot of other things to try (like taking out your SD card and just booting off the SSD) but it almost sounds like you may have old firmware for the Pi. This is typically updated with Raspberry Pi OS. Does your Pi have a “BIOS” looking screen when it starts up? If it doesn’t it also has old firmware, but I almost don’t feel like we even want to go there yet. Until we try with Raspberry Pi OS and narrow down whether we are dealing with an incompatible adapter I don’t even want to get to the later stages of the troubleshooting until we figure out the basics here like will this adapter work, is AstroBerry even compatible with this type of booting, and real basic things we need to figure out before modifying partition IDs makes a lot of sense!
Hopefully this message comes across the right way as my intention was to basically impress that we need a new strategy since we’ve had some back and forth for several days and have basically learned nothing. We aren’t even sure if this is a hardware or software issue even after all this troubleshooting. I’d have been fired at work if I worked on a ticket with this issue for more than a couple of hours without trying a clean image let alone a few days (wouldn’t have been invited back and they would have requested a new tech after the first day of fighting with it but getting no answers).
We need to know if this is an adapter issue, a bad OS install botched from an apt dist-upgrade, anything! There’s no way to do that without a reimage. If you absolutely can’t do that with this drive then you need to buy a spare drive we can test with otherwise we could have back and forth replies about this for another 2 weeks and still not even be sure if it’s an adapter issue or just a bad image basically. Starting to tweak the PARTUUIDs rather than starting with the basics is even further out in the weeds when we need to establish the most basic things first.
Reimaging this with either a fresh AstroBerry or even Raspberry Pi OS will tell us more than everything we’ve tried so far combined and much more. If it fires right up on Raspberry Pi OS we know we’re dealing with an AstroBerry specific issue. If it won’t even work with Raspberry Pi OS we know it’s a hardware/adapter issue or potentially even a firmware issue. If we knew that we could really zero in on the actual issue and would learn more from just that one fact than everything we’ve learned so far (which is basically nothing). Hopefully that helps!
Richard Hornby
2 years ago
Thanks James, but you are assuming a bit too much!
I haven’t done the update, because the version 173 of the firmware is not the version which is on the odroid site as latest. That is the 04.00.01.04 version. What I was wondering was whether my proposed method of installing that version – ie by running the same commands as in the odroid instructions, but using the .bin file from the odroid .zip download extracted to the same folder as the updater program – was the right way to do it.
However in the meantime I have managed to mess with the whole thing by doing a full APT upgrade on the astroberry. That has fritzed my root and home directories – though I can still see (but not open) the folders on the SSD partitions – and in the meantime I have to work out how to get back to booting to SSD. I think I am going to have to boot from SD and examine the files on the SSD without mounting the partitions. Then I may be able to see what has happened.
Ref the idea of taking a backup before attempting to dd the boot and root partitions of the SD, that is exactly what I am trying to do! How do I ‘bootstrap’ that process to get a backup before backing up? The ‘SD duplicator’ on the PI will only backup complete devices, and I don’t need or want the SSD data partition. You are correct that the pi sees the SSD partitions as sda1, 2 etc. Here are the relevant terminal outputs: do you see anything wrong here?
astroberry@astroberry:~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
|-sda1 8:1 0 256M 0 part /media/astroberry/boot1
|-sda2 8:2 0 31.7G 0 part /media/astroberry/rootfs1
-sda3 8:3 0 87.3G 0 part /media/astroberry/PI_SSD_DATA
mmcblk0 179:0 0 29.8G 0 disk
|-mmcblk0p1 179:1 0 256M 0 part /boot-mmcblk0p2 179:2 0 29.6G 0 part /
Most likely your /etc/fstab file is no longer pointing to the SSD. That is the file that controls which partitions are mounted at startup at least. Almost everything you are attempting to do will be overwritten by updates because you are doing this at a very, very low level manually.
Booting and running from HDD/SSD is not astroberry specific. If you configure your system to boot/run from external drive astroberry will be fine.
However raspberry does not boot directly from external drive. It always uses SD for it. So you need to keep at least SD card with boot partition and configure root parameter in /boot/cmdline.txt to point to your external drive partition e.g. /dev/sda1 (it must be ext4 formated and it should contain all the files from root partition). Also you would need to change the configuration of /etc/fstab to change / point to new partition (e.g. /dev/sda1).
I think this is likely all we are dealing with. The only other possibility is if there was a change made to the partition table but I don’t think this is the case. I think /etc/fstab just needs an update to point to the SSD partitions for the home directory.
I definitely agree I assumed too much as now I’m still not sure I understand your backup strategy. The reason the SD imager takes full drive backups is that you need the partition table for those images to be of any use for the most part. If you don’t need the raw disk images then it does not make sense to use dd for this.
What are you trying to back up exactly? Here is my full backup for jamesachambers.com that can be moved and deployed on a brand new AWS dedicated Linux instance in about 3 minutes:
# Backup all databases
mysqldump -u root -p --all-databases > all_databases.sql
sudo tar -cJf all_databases.tar.xz all_databases.sql
# Backup Files
cd /
sudo tar -cJf apachecnf.tar.xz /etc/apache2
sudo tar -cJf phpcnf.tar.xz /etc/php/8.0/fpm
sudo tar -cJf jamesachambers.com.web.tar.xz /var/www
sudo tar -cJf cert.tar.xz /etc/letsencrypt
sudo tar -cJf redis.tar.xz /etc/redis
sudo tar -cJf mesh.tar.xz /home/ubuntu/meshcentral
And to restore:
sudo apt install mysql, other dependencies
# Restore databases
tar -xf all_databases.tar.xz
mysql -u root -p < all_databases.sql
# Restore files
cd /
upload/restore backup files to root of drive
sudo tar -xf apachecnf.tar.xz
sudo tar -xf phpcnf.tar.xz
sudo tar -xf jamesachambers.com.web.tar.xz
etc.
All I need to do on the new machine is run sudo apt install for the dependencies and then extract those tar archives. They contain my complete apache, php and other services configurations as well as directory trees. The backups are small and compressed.
Taking a disk/partition image is the opposite of this strategy. If you had a full disk image you would actually be able to fix this even if your partition table was modified because you would have the full partition table backed up. This is one of the benefits of having a full image but without the full image things get real tricky. You've said you don't need the SSD partitions, but then what do you need to back up on the Linux partitions?
The fastest way to fix this (if it's more than fixing the /etc/fstab file, check both the SD card's /etc/fstab and the SSD's /etc/fstab, one of them must be wrong) would literally be to start over. I would never try to fix a broken server or VM like this because with a full or even targeted backup like I posted because it only takes minutes to restore a targeted backup like this (even over the top of a brand new system) unless there were no backups/system in place yet and had no choice but if you can access the files you need you do still have a choice.
Backups should be simple and fast/effective to restore for sure! If you don't have a rapid deploy modular backup system like this then it may be time to start one and from your situation I think you would be able to do this. Since you have access to the partitions you could easily get what you need in a form you can restore on a cleaned up image/install. If you save the commands to copy/restore the files back you now have a new modular backup routine you can save in a .sh file (this is what I do). I personally take the backups and copy them to an off-site location (I use it as a chance to look over everything in the server and make sure nothing looks unusual) every couple of weeks but this can be automated fairly easily as well.
Is this strategy possible for you? Almost anything should be able to be restored/backed up this way as it's pretty standard in Linux (with full DD backups being the other common alternative). I really recommend against the dd images of specific partitions vs. a strategy like this and I think you could still easily recover/restore the files on a completely reimaged/fixed partition system!
I am testing several times INTEL 760p 128GB NVMe on AXAGON GTX with Pi4 4GB, Transcend 120GB mSATA in EVOLVEO USB box and old mSATA 64GB from the computer in mSATA 3GB speed USB box. Just to add, that all tests in variants with/without USB powered HUB. For some combination it does not work with USB HUB even just because of USB-C cable. Finally should be the difference for Pi4 new power version (related with Bullseye upgrade info). I am recomend to add some possibility to comment the results some time later – if the configuration works stable or not. For me – only old version of 3GB speed mSATA USB box works perfectly with mSATA disk, now testing Transcend in EVOLVEO looks OK for just short time now.
Hey Ivo12s,
Great feedback! I actually really like the idea. It should be theoretically possible. The site would need to have some kind of account system created but each Pi and SSD/HDD has a serial # so I’m envisioning that I could let you “claim” old results under your account if you do a new benchmark with that Pi so it can read the serial #. Maybe something like that (all really rough tentative ideas).
The powered USB hubs can be tricky for sure. Which hub did you end up going with? One trick I often use is that the Pi won’t turn on when connected to these powered USB hubs sometimes. If you disconnect the USB cable to your powered USB hub and then connect the Pi it should power on then plug in the USB hub immediately as fast as you can before it starts the boot process. This works around a lot of weird USB hub issues (related to booting at least).
I had not heard of the AXAGON GTX adapter before but I do see it in Google. Would you be willing to post a lsusb from that adapter? I’m curious what storage chipset it’s using inside (could be a JMicron chipset such as a JMS567, or an ASMedia one as well). Have you checked to see if there’s any firmware updates available for it? It sounds like you may have things working stable here though for the most part but it may be worth checking. Thanks again for the feedback!
Hi James,
GTX is my mistake…AXAGON EEM2-GTR is the one I had been testing.
You can see I did a lot of tests 🙂 as I mentioned. I am really flustrating when realised today, that some of them significantly influenced results just because of USB 3 cable used to connect Pi4 with powered USB HUB. The topic is visible between Test #53,476 and Test #53,472. The difference just USBA-C cable. The result is great, but Pi4 stop working after some time even with good cable 🙁 anyway.
Most of the times I tested even Quirks solution, but with no serious difference in stability.
I changed my Raspberry Pi4 for new 4GB version with updated power part, therefore repeated tests if the solution exist.
And last but not least… I realised, that I have better results with powered USB HUB where both Pi4 & USB BOX are connected as “the equipment” in the box, means not used main USB connector… have no idea why. I am still looking for the final configuration where it is no needs for powerd USB HUB. Expect you do not need lsusb for AXAGON GTR :-).
Regards!
Ivo
Hey Ivo12s,
Fascinating information, thanks for the followup! That is amazing that the cable made that much of an impact. Another really weird thing is that USB 3.0 literally causes RF interference for WiFi. I’ve actually encountered this before when we had someone who had their mouse adapter stop working due to the interference from USB 3.0. USB 2.0 has *no* interference strangely enough!
For the lsusb command it just spits out which devices are connected to your system. The compatibility with the Pi is actually impacted by the chipset that will show up in the lsusb tool. It’s not a driver or anything like that, it just gives you the real name of the storage adapter (AXAGON doesn’t make this piece of it, they buy it from someone else since only like 3-4 companies in the world actually make these types of storage controllers used in the adapters). It looks like it’s a JMicron storage controller in yours according to this result.
I wonder if your USB cable was one of the “charging” USB cables. You can sometimes get data over these at USB 2.0 speeds. At any rate, this is great information for people to double check as this gets overlooked a lot. Connecting everything to your powered USB hub without using the main USB connector is a really interesting trick to use as well, thanks again for posting it! Scores are looking really good as well!
I’ve found the point! As part of the optimization I upgraded my RPi from 2GB to 4GB. And I found out that my original RPi4-2GB has HW version 1.1. Maybe you know that this version 1.1 has a problem with the specification of the power part/USBC connector. That is, if I used the original USB3 cable, there was not enough power supply to work with the SSD.
Hey Ivo,
Very interesting, great find! I do remember this actually at the time.
I had not read your source before and it actually paints a better picture of the issue I think so thanks for sharing it. I believe the old inventories of these are long gone but some of my early launch Pis are still this revision. This is a great thing to check so thanks again for posting it!
Thank you very much for posting this! The usb-storage.quirks addition worked well for my Sabrent EC-SSHD sata to usb adapter. Boot times went from ~ 1:30s to 0:18s using a WD 500Gb SATA III drive! Really appreciate your help!
Hey Danny,
Thanks so much for your kind words and a confirmation! I’ve been hearing mostly about problems (not the people’s fault, just the nature of comments a lot of the time) so I’m really glad to hear of a successful update / project!
Those gains aren’t just impressive, those are game changer numbers. It’s several times faster. Congratulations and enjoy!
Hi James
It worked 🙂 🙂 Hopefully this account will enable those who may have watched these posts with amusement / bemusement to take something useful away.
GOAL
1 To create a backup SD from an SSD which is over 32Gb in size, because it also contains an NTFS data partition.
2 To be able to update / upgrade astroberry without crashing the SSD.
METHOD
1 Turn this process on its head and create a base SD image which can be updated / amended and then deployed to SSD as required
2 Burn a new astroberry image to SD, apt-get update and upgrade, then install any extra components required (NTFS support is now included automatically). Install gparted.
3 Use gparted to shrink /root partition as required to create ~2Gb unallocated space after /root
4 Install and configure everything required on the SD eg timesync and WiFi
6 COPY THE SD TO USB using SD Card copier. Copier will report it could not mount one partition – ignore.
7 Shut down, remove SD card from pi, check it boots from USB.
8 Assuming all ok, shut down again, then plug in SSD, insert SD Card and copy SD card to SSD via SD card copier.
9 Shutdown again, remove SD card and boot from SSD.
10 The SSD should have /boot, /root and a large unallocated space. Use gparted to format the space as NTFS (or as you wish).
CONCLUSION
Now you have (i) a working backup SD in case of SSD crash, (ii) a backup / dev USB to update before deploying to SD/SSD, and (iii) a working SSD for day-to-day use.
One of my posts concerned updating the JMS 587 firmware driver. I was able to do this, but the drive seems still to require /boot/cmdline.txt to contain quirks.
Hope this may be of use.
Best
R
Hey Richard,
Great work! I’m very appreciative to you for posting such detailed steps. It’s tough to tell exactly how many people are helped by these articles and comments but this article still gets around 300-400 hits per day and the SD bootstrapping one I linked to earlier still gets 1200 hits a day or so. I think it’s safe to say that the vast vast majority (99.99%+) don’t comment but that undoubtedly a lot of them do find useful information. Thanks for contributing!
I’d almost forgotten that we started with the JMS587 firmware updates. There are different revisions of this adapter out there and variations within the JMS587 family even so it’s not a 100% guaranteed fix (unless the manufacturer releases them). I still think this was a worthwhile journey for sure as I think that your end setup turned out really nice!
I wouldn’t sweat the performance difference too much as it is around maybe 700-2000 points or so of a difference depending on the drive. The fastest SD cards will barely break 1000 though and you benchmarked many times faster than that even with quirks enabled. This is still a very fast performance option even if it isn’t going to win the local storage benchmarking competition and is in an entirely different class of performance than the SD card!
I admire your patience and persistence throughout the adventure. Your willingness to fight with this for a week or so just to understand what is happening / knowing there is a solution reminds me a lot of myself. I personally do this all the time and it is definitely one of my “secrets” powering my IT background/career. You learn 50 different things you couldn’t have ever imagined learning before going down these rabbit holes. I imagine you have learned a ton from this, and I’ve actually learned a few new tricks from this conversation / reached a better understanding of some concepts as well.
Cheers and good luck with rest of the project that this is all powering, and thanks again for sharing your detailed solution!
Ok very useful post! The video essentially shows where I had got to with your help, but somewhere along the line I seem to have lost the PARTUUID references – or possibly the firmware,as I have all the correct references in the correct files 🙁
Is it possible to install the firmware to the SSD directly without booting from it first?
Here’s where I am:
1 Made a fresh astroberry image to SD. This boots. Got the PARTUUID’s with blkid and checked they are referenced in cmdline.txt and fstab. They are, and are different from the PARTUUIDs for the SSD partitions.
Changed the fstab and cmdline on the SSD to point to the SD for /boot and the SSD for root. No boot. Changed them back – ie left them pointing to the SSD PARTUUID.
Tried booting from SSD without SD – fails with a GRUB-type message ‘failed to find target’. I can’t log on with CTL-ALT-F2 so have left this for the moment. I can’t read the error messages.
There seems to be a full image on the SSD rootfs, but I’m not sure.
2 On the new astroberry image, I have updated the FW to latest stable version – I guess this should be OK. As I said, this boots fine.
3 I think my only remaining course of action, unless there is a way of installing the correct firmware to the SSD without booting from it (as above) is to image the new working SD to the SSD, which will remove the data partition, then install gparted, restore the data partition (and NTFS file support, and everything else I had!), and hope all works.
SO going forward … any advice on making backup images of only the boot and root partitions for the future? The video shows how to copy all the partitions on the OP’s installation, but he didn’t have a data-only partition.
Best
R
Thanks for the advice! There are reasons to stay with Astroberry, viz most importantly it comes with a built in VNC and INDI server and a bunch of bundled astro programs. I have added my own set of apps for my Pi and the whole set is saved on the SSD. One of the bundled apps is ‘aggressively’ 32bit so I need for the mo to stick with the 32 bit astroberry OS. Until the astroberry upgrade it was basically working.
I will download a fresh astroberry image to SD and see if I can at least use that to boot the SSD, but I am not holding my breath.
On another point .. I have found some sites on the web which advocate copying the SD image to the SSD. When I try to do that with the Pi4B it tells me I can’t as it doesn’t find the eeprom. Is it not the case that the Pi4B will still boot from SD even with the Raspberry-Pi OS? This is what the Pi.org site also suggests. If that is the case, then (1) aren’t all Pi4B users stuck with a 5 yr old process and (2) I’m not sure that loading the Pi OS will help all my issues, specifically how to take backups of a SSD which also includes a data partition. If there is a way to copy the SD to SSD and boot from that, I may have missed it.
I will take it in steps and get a new astroberry image and let you know what happens. I appreciate the help.
Best R
Hey Richard,
Great additional questions, I’m glad I wasn’t too off-putting with my last message and I think you’re on the right track here!
So in regards to SSD booting with AstroBerry you shouldn’t need to use the SD bootstrapping method. The Pi natively supports booting from USB now (if you have the latest firmware, which I doubt AstroBerry can update, you would need to update your firmware using the recovery image that the Pi Imager tool creates, more on that + a link in a moment). Here’s a bunch of people that have done it with AstroBerry.
They used BerryBoot as the bootstrapper if that makes sense but this is no longer necessary (according to people at the bottom of the post who confirmed this). BerryBoot is a distribution that supports booting multiple OS’s off a single storage device (perfect for SSDs which have room for many different installs). This may be of assistance for sure. People at the bottom of the post said that BerryBoot is no longer even necessary and you can image it directly to the SSD with Etcher now (using the latest Pi firmware of course) but this may destroy your other partitions so in that sense BerryBoot might still be easier for that setup. It’s specifically designed to “play nice” with multiple other partitions on the same storage device which is exactly the goal you’re working toward here.
Have you tried booting with the SD card taken out completely? This should work on a clean working install without the SD card inserted. The SSD should still have it’s own boot partition (but it may be damaged, maybe this is what the apt-upgrade messed up, it would explain several things). All of the magic happens in the /etc/fstab file. The catch is you need new enough firmware on your Pi to be able to boot (you should see a BIOS looking screen when it boots up, if it fails to boot it should try over and over, it should also try to boot off multiple devices like the SSD if the SD card isn’t inserted). I do have a guide on updating the firmware here: Raspberry Pi 4/400 Bootloader Firmware Update/Recovery Guide
There is an example of someone doing exactly what you mentioned and copying the SD card image to the SSD and they said it worked for them. This is definitely worth a try I think as well! I think any one of these methods should get it going. I would highly recommend creating the “Recovery Image” with the Raspberry Pi Imager tool to make sure to update your Pi to the latest bootloader firmware and make sure you are seeing the BIOS-looking-ish startup screen. You would only need to do this one time then you can go right back to AstroBerry permanently once it’s updated. Let us know how it turns out!
Hi James
I really appreciate how much you are helping me and I hope the discussion may be useful to others. I definitely want to get a strategy for being able to do n(1) a full system upgrade and (2) a backup of my SSD. I take your point about a compressed backup may be the way to go, but ATM am still stuck on no boot.
So I have concentrated on trying to sort out /etc/fstab – and /etc/.fstab which I also found.
Something odd seems to be happening. Somehow my PARTUUID for the SD card seems to have changed to match the PARTUUID for the SSD. If I just insert the SD and then the SSD into my Linux lappie and do a blkid on each, I’m getting the same PARTUUID. And /etc/fstab seems to have changed to point to that UUID for both /boot and root. Previous to what I can only consider as a change, the PI did boot, but to the wrong drive. Now it won’t boot at all.
Whatever, I think I may have to change the PARTUUID of the SD card to get anything to happen. I guess I would do that then alter fstab and .fstab accordingly on both SD and SSD, and then try again.
Does this make any sense?
Tks
R
Hey Richard,
No problem at all. You’re basically dealing with how SSD booting on the Pi used to work 5 years ago so I’m familiar with exactly what you’re dealing with. In fact, I think one of my very, very old articles from before the Pi natively supported USB booting may be of assistance:
I actually cover how to change the PARTUUIDs, resizing partitions and everything in my very very old USB Booting for Raspberry Pi guide here. These instructions are completely obsolete for modern Pis. Nobody else is going through *anything* like what you’re fighting with using this AstroBerry distribution after whatever happened to it with apt. In other words, the Raspberry Pi supports NATIVE USB booting with the latest firmware and a compatible OS. If you are having to do it the way in the article I linked to above that is before the Raspberry Pi even supported USB booting that you had to worry about things like PARTUUIDs.
It’s not typical to have to do any of this. I would say your installation of AstroBerry has been broken ever since your apt upgrade and the whole thing should be thrown out. We know it has been broken since the apt-upgrade before which it was acting relatively normal (although not perfect). There’s nothing to save at this point, it’s just holding you back if you have a copy of the files you need. Even if you want to stick with AstroBerry I would install it clean and not run the apt upgrade and see if you can at least get it working as well as before.
You can literally just image a SSD with Raspberry Pi OS and it will just boot up. No SD card even inserted. Messing with things like the PARTUUID hasn’t been necessary in years.
I don’t know if this is an AstroBerry thing or if this is because of the apt dist-upgrade that got ran on it but this is a complete mess and none of this should be necessary. That’s why I suggested starting over as your boot files seem to have been modified by your apt dist-upgrade into something that isn’t compatible / booting correctly.
If you do start over I would use a different OS. It should take you no more than 15 minutes to completely download another OS and try it. I bet you it will boot right up. I’m not sure what else to tell you at this point though as literally the first thing I would have tried was reinstalling Raspberry Pi OS on it and after several days of back and forth I don’t think this drive has been reimaged with Raspberry Pi OS or even a fresh install of AstroBerry yet.
We can’t rule anything in / out at this point and you’re way, way down the rabbit hole at this point trying to track down partition IDs when frankly you just need to reimage the thing. Just the fact that it broke from an accidental apt dist-upgrade should have been an *automatic* reimage from scratch completely especially after several days of troubleshooting that is spinning it’s wheels. We have to start narrowing things down. You’re just guessing trying to avoid the one thing that makes sense to do in this situation: reinstall.
Your adapter might have compatibility issues with the Pi but we haven’t done the most basic tests to rule it out such as trying with Raspberry Pi OS or anything really to narrow things down. We’ve basically got nowhere for several days until we start getting serious with it and ruling things out / even trying different OS’s to figure out what we’re dealing with here. I can personally assure you we are not dealing with normal or typical behavior at all. The behavior before the apt-upgrade was a little bit more normal but I’ve never heard of AstroBerry and upon some Google searches I found lots of issues booting from SSD out there that implied to me that maybe the support is a couple of years behind and that it isn’t as easy/straightforward as something like Raspberry Pi OS, Ubuntu, etc.
It should literally take 2 minutes to image and boot up with a SSD if that makes sense! There’s a lot of other things to try (like taking out your SD card and just booting off the SSD) but it almost sounds like you may have old firmware for the Pi. This is typically updated with Raspberry Pi OS. Does your Pi have a “BIOS” looking screen when it starts up? If it doesn’t it also has old firmware, but I almost don’t feel like we even want to go there yet. Until we try with Raspberry Pi OS and narrow down whether we are dealing with an incompatible adapter I don’t even want to get to the later stages of the troubleshooting until we figure out the basics here like will this adapter work, is AstroBerry even compatible with this type of booting, and real basic things we need to figure out before modifying partition IDs makes a lot of sense!
Hopefully this message comes across the right way as my intention was to basically impress that we need a new strategy since we’ve had some back and forth for several days and have basically learned nothing. We aren’t even sure if this is a hardware or software issue even after all this troubleshooting. I’d have been fired at work if I worked on a ticket with this issue for more than a couple of hours without trying a clean image let alone a few days (wouldn’t have been invited back and they would have requested a new tech after the first day of fighting with it but getting no answers).
We need to know if this is an adapter issue, a bad OS install botched from an apt dist-upgrade, anything! There’s no way to do that without a reimage. If you absolutely can’t do that with this drive then you need to buy a spare drive we can test with otherwise we could have back and forth replies about this for another 2 weeks and still not even be sure if it’s an adapter issue or just a bad image basically. Starting to tweak the PARTUUIDs rather than starting with the basics is even further out in the weeds when we need to establish the most basic things first.
Reimaging this with either a fresh AstroBerry or even Raspberry Pi OS will tell us more than everything we’ve tried so far combined and much more. If it fires right up on Raspberry Pi OS we know we’re dealing with an AstroBerry specific issue. If it won’t even work with Raspberry Pi OS we know it’s a hardware/adapter issue or potentially even a firmware issue. If we knew that we could really zero in on the actual issue and would learn more from just that one fact than everything we’ve learned so far (which is basically nothing). Hopefully that helps!
Thanks James, but you are assuming a bit too much!
I haven’t done the update, because the version 173 of the firmware is not the version which is on the odroid site as latest. That is the 04.00.01.04 version. What I was wondering was whether my proposed method of installing that version – ie by running the same commands as in the odroid instructions, but using the .bin file from the odroid .zip download extracted to the same folder as the updater program – was the right way to do it.
However in the meantime I have managed to mess with the whole thing by doing a full APT upgrade on the astroberry. That has fritzed my root and home directories – though I can still see (but not open) the folders on the SSD partitions – and in the meantime I have to work out how to get back to booting to SSD. I think I am going to have to boot from SD and examine the files on the SSD without mounting the partitions. Then I may be able to see what has happened.
Ref the idea of taking a backup before attempting to dd the boot and root partitions of the SD, that is exactly what I am trying to do! How do I ‘bootstrap’ that process to get a backup before backing up? The ‘SD duplicator’ on the PI will only backup complete devices, and I don’t need or want the SSD data partition. You are correct that the pi sees the SSD partitions as sda1, 2 etc. Here are the relevant terminal outputs: do you see anything wrong here?
astroberry@astroberry:~ $ lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
|-sda1 8:1 0 256M 0 part /media/astroberry/boot1
|-sda2 8:2 0 31.7G 0 part /media/astroberry/rootfs1
-sda3 8:3 0 87.3G 0 part /media/astroberry/PI_SSD_DATA
mmcblk0 179:0 0 29.8G 0 disk
|-mmcblk0p1 179:1 0 256M 0 part /boot-mmcblk0p2 179:2 0 29.6G 0 part /
astroberry@astroberry:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 30G 6.0G 23G 22% /
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 9.5M 3.9G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mmcblk0p1 253M 55M 198M 22% /boot
tmpfs 783M 4.0K 783M 1% /run/user/1001
/dev/sda3 88G 24G 65G 27% /media/astroberry/PI_SSD_DATA
/dev/sda2 31G 16G 14G 53% /media/astroberry/rootfs1
/dev/sda1 253M 49M 204M 20% /media/astroberry/boot1
astroberry@astroberry:~ $
Tks for your continued advice
Best R
Hey Richard,
Most likely your /etc/fstab file is no longer pointing to the SSD. That is the file that controls which partitions are mounted at startup at least. Almost everything you are attempting to do will be overwritten by updates because you are doing this at a very, very low level manually.
From the Astroberry forums:
I think this is likely all we are dealing with. The only other possibility is if there was a change made to the partition table but I don’t think this is the case. I think /etc/fstab just needs an update to point to the SSD partitions for the home directory.
I definitely agree I assumed too much as now I’m still not sure I understand your backup strategy. The reason the SD imager takes full drive backups is that you need the partition table for those images to be of any use for the most part. If you don’t need the raw disk images then it does not make sense to use dd for this.
What are you trying to back up exactly? Here is my full backup for jamesachambers.com that can be moved and deployed on a brand new AWS dedicated Linux instance in about 3 minutes:
# Backup all databasesmysqldump -u root -p --all-databases > all_databases.sql
sudo tar -cJf all_databases.tar.xz all_databases.sql
# Backup Files
cd /
sudo tar -cJf apachecnf.tar.xz /etc/apache2
sudo tar -cJf phpcnf.tar.xz /etc/php/8.0/fpm
sudo tar -cJf jamesachambers.com.web.tar.xz /var/www
sudo tar -cJf cert.tar.xz /etc/letsencrypt
sudo tar -cJf redis.tar.xz /etc/redis
sudo tar -cJf mesh.tar.xz /home/ubuntu/meshcentral
And to restore:
sudo apt install mysql, other dependencies# Restore databases
tar -xf all_databases.tar.xz
mysql -u root -p < all_databases.sql # Restore files cd / upload/restore backup files to root of drive sudo tar -xf apachecnf.tar.xz sudo tar -xf phpcnf.tar.xz sudo tar -xf jamesachambers.com.web.tar.xz etc.
All I need to do on the new machine is run sudo apt install for the dependencies and then extract those tar archives. They contain my complete apache, php and other services configurations as well as directory trees. The backups are small and compressed.
Taking a disk/partition image is the opposite of this strategy. If you had a full disk image you would actually be able to fix this even if your partition table was modified because you would have the full partition table backed up. This is one of the benefits of having a full image but without the full image things get real tricky. You've said you don't need the SSD partitions, but then what do you need to back up on the Linux partitions?
The fastest way to fix this (if it's more than fixing the /etc/fstab file, check both the SD card's /etc/fstab and the SSD's /etc/fstab, one of them must be wrong) would literally be to start over. I would never try to fix a broken server or VM like this because with a full or even targeted backup like I posted because it only takes minutes to restore a targeted backup like this (even over the top of a brand new system) unless there were no backups/system in place yet and had no choice but if you can access the files you need you do still have a choice.
Backups should be simple and fast/effective to restore for sure! If you don't have a rapid deploy modular backup system like this then it may be time to start one and from your situation I think you would be able to do this. Since you have access to the partitions you could easily get what you need in a form you can restore on a cleaned up image/install. If you save the commands to copy/restore the files back you now have a new modular backup routine you can save in a .sh file (this is what I do). I personally take the backups and copy them to an off-site location (I use it as a chance to look over everything in the server and make sure nothing looks unusual) every couple of weeks but this can be automated fairly easily as well.
Is this strategy possible for you? Almost anything should be able to be restored/backed up this way as it's pretty standard in Linux (with full DD backups being the other common alternative). I really recommend against the dd images of specific partitions vs. a strategy like this and I think you could still easily recover/restore the files on a completely reimaged/fixed partition system!