PCIe 1x NVMe on Raspberry Pi?! Compute Module 4 Guide

CableCC Vertical Adapter in Compute Module 4 IO Board
NVMe PCIe 1x Vertical Adapter in CM4 IO Board

I recently covered all the pieces you need for a complete setup to work with the Raspberry Pi Compute Module 4. Today I want to cover the whole reason I wanted to investigate the Compute Module 4: The PCI express capabilities!

In this guide I’ll cover a couple of 1x PCIe to NVMe adapters I picked up to try doing this with as well as full configuration and setup instructions. Let us begin!

Prerequisites

The Compute Module 4 by the Raspberry Pi Foundation is a single-board computer (SBC) that is meant to be used in embedded devices. It has many new capabilities that have not been seen on other Pis before such as a built in eMMC module (optional) and has PCI express capabilities when used with the IO board.

Links: AliExpress.com*, Amazon.ae*, Amazon.ca*, Amazon.com*, Amazon.com.au*, Amazon.com.mx*, Amazon.co.uk*, Amazon.de*, Amazon.es*, Amazon.fr*, Amazon.in*, Amazon.it*, Amazon.nl*, Amazon.pl*, Amazon.sa*, Amazon.se*, Amazon.sg*

Make sure you’ve imaged your NVMe drive with an operating system of your choice such as Raspberry Pi OS, Ubuntu or your preferred distro. You may have to actually put the drive into a PC temporarily (you can use the 1x adapter in one) or if you have a USB to NVMe adapter (lots of Pi users should have one of these already) you’re good to go.

This guide will assume you’ve done this and the drive is prepped. The official Raspberry Pi Imager tool is a great choice to take care of imaging needs for you and is available for Windows / Mac OS X / Linux. Here’s a suggested drive if you don’t have one yet:

Samsung 970 EVO NVMe SSD
Samsung 970 EVO Plus NVMe SSD

The Samsung 970 EVO Plus is a fantastic drive and has fallen in price substantially. It’s widely available around the world. The smaller capacities (such as the 250GB version) of this drive are perfect for the Pi! This is the top performance option without going into the “Pro” series of the lineup which are much more expensive.

Links: AliExpress.com*, Amazon.ae*, Amazon.ca*, Amazon.com*, Amazon.com.au*, Amazon.com.mx*, Amazon.co.jp*, Amazon.co.uk*, Amazon.de*, Amazon.es*, Amazon.fr*, Amazon.in*, Amazon.it*, Amazon.nl*, Amazon.pl*, Amazon.sa*, Amazon.se*, Amazon.sg*

You will also need a IO board that has a PCI express slot:

This is essentially the “motherboard” of the compute module. This IO breakout board lets us work with the compute module and get the ports we need to interface with it. This includes a SD card slot for models that don’t have eMMC, a PCI express slot (very exciting), 2 USB ports and more.

Links: AliExpress.com*, Amazon.ae*, Amazon.ca*, Amazon.com*, Amazon.com.au*, Amazon.com.mx*, Amazon.co.jp*, Amazon.co.uk*, Amazon.de*, Amazon.es*, Amazon.fr*, Amazon.in*, Amazon.it*, Amazon.nl*, Amazon.pl*, Amazon.sa*, Amazon.se*, Amazon.sg*

Power requirements are also an issue. If you are using a powerful (or power hungry) NVMe drive you will for sure need something like this:

Delivers a full 12V DC to the IO board so that there’s enough power to run very powerful NVMe drives / accessories

Links: AliExpress.com*, Amazon.ae*, Amazon.ca*, Amazon.com*, Amazon.com.au*, Amazon.com.mx*, Amazon.co.jp*, Amazon.co.uk*, Amazon.de*, Amazon.es*, Amazon.fr*, Amazon.in*, Amazon.it*, Amazon.nl*, Amazon.pl*, Amazon.sa*, Amazon.se*, Amazon.sg*

If your NVMe drive is performing extremely slowly / booting into emergency mode / things like that it’s very likely that it’s not receiving enough power!

PCIe to NVMe 1x Adapters

I picked up two different style adapters from Amazon to test. The first one is a very simple vertical standing adapter. My full benchmark for this adapter is available here: https://pibenchmarks.com/benchmark/48606

The Cablecc 1x PCIe M.2 NVMe adapter sticks straight up out of the board. It doesn’t take up a lot of space and ends up being pretty compact (as long as you have vertical space!)

Links: AliExpress.com*, Amazon.ae*, Amazon.ca*, Amazon.com*, Amazon.com.au*, Amazon.com.mx*, Amazon.co.jp*, Amazon.co.uk*, Amazon.de*, Amazon.es*, Amazon.fr*, Amazon.in*, Amazon.it*, Amazon.nl*, Amazon.pl*, Amazon.sa*, Amazon.se*, Amazon.sg*

Next up we have an adapter that sits horizontally and comes with some optional mounting brackets. This adapter scored about 400 points higher (may be a slightly better chipset) and has a green LED indicator light (the cablecc has no light of any kind). Benchmark available here: https://pibenchmarks.com/benchmark/48607

The Xiwai PCI-E 3.0 x1 Lane to M.2 NGFF M-Key SSD NVMe AHCI adapter card sits horizontally and comes with a couple of different size shrouds meant for mounting (not required). This takes up less vertical space but more horizontal space.

Links: AliExpress.com*, Amazon.ae*, Amazon.ca*, Amazon.com*, Amazon.com.au*, Amazon.com.mx*, Amazon.co.jp*, Amazon.co.uk*, Amazon.de*, Amazon.es*, Amazon.fr*, Amazon.it*, Amazon.nl*, Amazon.pl*, Amazon.sa*, Amazon.se*, Amazon.sg*

Changing Boot Order

While the PCIe to NVMe adapters are pretty much plug and play in this setup switching the boot order of the Compute Module 4 is a little bit more in depth. It requires using the rpiboot tool on a different computer / Pi with the CM4’s “disable eMMC boot” jumper (j2) bridged with the pin directly below it so the Pi is able to boot into USB mass storage mode and be flashed with bootloader/firmware updates.

To do this we need to get a copy of the rpiboot/usbboot utility’s source tree. If you followed my CM4 setup guide you will already have the ‘usbboot’ folder and can skip cloning. If you are using Windows you will need to do this on your Pi and we will copy the final resulting folder over to Windows (for now just do the instructions on your Pi).

git clone --depth=1 https://github.com/raspberrypi/usbboot
cd usbboot
make

Now we’re going to enter the ‘nvme’ subdirectory and edit the bootloader configuration like this to change the boot priority:

cd nvme
sed -i -e '/^BOOT_ORDER=/ s/=.*$/=0xf25416/' boot.conf
./update-pieeprom.sh

You can also change boot.conf manually using nano followed by running ./update-pieeprom.sh. The boot type for PCI express is 6, so that should be your first type (which means it should be the number at the end of the string like I have in the above example) if you want to boot from PCI express.

(Optional) Choose / Update Firmware Version

The CM4 will have the exact firmware that is in your git source tree as the file “pieeprom.original.bin” inside the nvme folder at the end of this process.

I will use the latest beta firmware at time of writing but check for the latest firmware at: https://github.com/raspberrypi/rpi-eeprom/tree/master/firmware/beta (if there is a newer one replace the pieeprom-2021-07-06.bin in the curl command below with the newer version file name). The beta firmware is required to boot with PCIe/NVMe at time of writing.

Use these commands (from inside the usbboot/nvme directory):

rm -f pieeprom.original.bin
curl -L -o pieeprom.original.bin https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/pieeprom-2021-07-06.bin
./update-pieeprom.sh

(Optional) Further Bootloader Configuration Changes

If there are any other bootloader parameters you would like to change you can do so using the file boot.conf like this (from inside the usbboot/nvme directory):

nano boot.conf
Make changes and press Ctrl+X then type 'y' to save
./update-pieeprom.sh

Flashing New Boot Order / Firmware / Configuration

We are now ready to run the rpiboot utility and use it to flash the new bootloader configuration and firmware to our Pi. You should still be in the “nvme” directory from the previous steps. Go to the base of the “usbboot” directory with:

$ cd ..
$ ls
bin2c    debian      imager   main.c    msd   Readme.md  rpiboot
bin2c.c  fmemopen.c  LICENSE  Makefile  nvme  recovery   win32

You should see the “rpiboot” utility like the above example if you’re in the base of the usbboot directory. If your host PC you will be plugging the USB cable into for flashing runs Linux use the command:

sudo ./rpiboot -d nvme

Make sure your Compute Module 4 IO board’s “disable MMC boot” jumper J2 is bridged with the lower pin to disable eMMC boot. Now connect the power:

api@pi:~/usbboot $ sudo ./rpiboot -d nvme
Loading: nvme/bootcode4.bin
Waiting for BCM2835/6/7/2711...
Loading: nvme/bootcode4.bin
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711...
Loading: nvme/bootcode4.bin
Second stage boot server
Loading: nvme/config.txt
File read: config.txt
Loading: nvme/pieeprom.bin
Loading: nvme/pieeprom.bin
Loading: nvme/pieeprom.sig
File read: pieeprom.sig
Loading: nvme/pieeprom.bin
File read: pieeprom.bin
Second stage boot server done
pi@pi:~/usbboot $

(Optional) Windows Specific Instructions

If you are using Windows transfer the “nvme” folder you generated using the Pi over to Windows and run it in “Command Prompt” like this (make sure you’ve installed the rpiboot utility for Windows first, see my CM4 setup guide for links/instructions):

"C:\Program Files (x86)\Raspberry Pi\rpiboot.exe" -d nvme

or you can fully qualify your paths with:

"C:\Program Files (x86)\Raspberry Pi\rpiboot.exe" -d "C:\YourPathTo\nvme"

and connect the IO board (with the disable eMMC boot jumper set). You will see the same output as the Linux example above and the CM4 will be updated!

Preconfigure NVMe Device for CM4

If you attempt to boot now you will not have working USB ports, ssh, or any wireless configuration. We can preconfigure your NVMe device by plugging the device into a computer / Pi. You should see the “boot” volume where we can preconfigure our CM4 to be working right from the start.

Enable USB Ports

When mounted in the IO board the USB ports will not function without making a change to config.txt and adding an overlay.

Add the following line to config.txt:

dtoverlay=dwc2,dr_mode=host

After adding this line the two USB ports (as well as the “External USB” header) will function the next time the CM4 boots.

Enable SSH

It’s very useful to have SSH enabled when working with the Compute Module 4. It is enabled simply by creating an empty file named:

ssh

in the root folder of the “boot” volume (the top-most folder of the drive where start4x.elf, fixup.dat and those files are located).

Preconfigure WiFi / Wireless

You can also preconfigure the Compute Module 4 (and any Pi) by creating wpa-supplicant.conf at the base of the “boot” volume like we did with “ssh” file.

Create a new file named wpa-supplicant.conf that contains the following:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
country=US
update_config=1

network={
 ssid="YourNetwork"
 psk="YourPassword"
}

When this file is created in the base of the “boot” volume it will automatically be applied to the Pi upon the next startup. Make sure to adjust your country=US line to your own country as this can impact what channels and functionality are available from your WiFi card!

Updating to Beta Firmware (Required for NVMe Boot)

We need to upgrade the boot files on the NVMe drive using rpi-update. This needs to be done from a Pi (any Pi will do). You can even use the Compute Module 4 itself booted from the eMMC if you need to.

Depending on how you connect the drive to your Pi your device may show up in ls /dev as /dev/sdX1, 2 etc. or /dev/nvme0n1p1, p2 etc. Determine the correct set of partitions. Partition 1 such as /dev/sda1 or /dev/nvme0n1p1 will be the boot partition. Partition 2 such as /dev/nvme0n1p2 or /dev/sdb2 would be the “root” partition containing the system and all applications.

If your OS tries to automount the drives (usually the “boot” drive will be the one that does this) go ahead and eject/dismount them first. If the drives are already mounted / in use you will get errors using the next set of commands so check this first if any commands are giving you trouble.

First let’s resize our root partition so it will have enough room for the beta updates:

sudo parted /dev/sdX resizepart 2 100%

Now let’s resize the root partition’s filesystem to use all available space:

sudo resize2fs /dev/sdX2

The Pi normally does this step for us on first startup but we are going to do it early because the pre-sized partition (by default around ~3.3 GB no matter how large your drive is) won’t have enough free space for rpi-update to store the new beta firmware updates without this step.

We can now mount and update the partitions using the commands below:

cd /mnt
sudo mkdir boot
sudo mkdir root
sudo mount /dev/sdX1 /mnt/boot
sudo mount /dev/sdX2 /mnt/root
sudo ROOT_PATH=/mnt/root BOOT_PATH=/mnt/boot rpi-update

A successful update will look like this:

pi@pi:/mnt $ sudo ROOT_PATH=/mnt/root BOOT_PATH=/mnt/boot rpi-update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** Performing self-update
 *** Relaunching after update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** We're running for the first time
 *** Backing up files (this will take a few minutes)
 *** Backing up firmware
 *** Backing up modules 5.10.17-v7l+
#############################################################
WARNING: This update bumps to rpi-5.10.y linux tree
See: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=288234
'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.

DO NOT use 'rpi-update' as part of a regular update process.

##############################################################
Would you like to proceed? (y/N)
 *** Downloading specific firmware revision (this will take a few minutes)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   168  100   168    0     0     48      0  0:00:03  0:00:03 --:--:--    48
100  121M  100  121M    0     0  6133k      0  0:00:20  0:00:20 --:--:-- 8055k
 *** Updating firmware
 *** Updating kernel modules
 *** depmod 5.10.52-v7l+
 *** depmod 5.10.52+
 *** depmod 5.10.52-v7+
 *** depmod 5.10.52-v8+
 *** Updating VideoCore libraries
 *** Using SoftFP libraries
 *** Updating SDK
 *** Running ldconfig
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to d4c184e19b8b48ac358ae595d09db1d77bf0f150

First Boot

You are now ready to boot your Compute Module 4. Go ahead and remove the jumper now from J2 and the pin below it and connect your prepped NVMe drive. If you’re using a monitor connect it to the IO board first before turning on the power.

Now connect power to the IO board and the PCIe/NVMe boot process will start. Your OS should boot with your preconfigured settings! My first boot was a little sluggish (most likely first boot operations on your drive) so give it a little bit of extra time here before assuming it failed.

You should verify your boot device by running:

sudo mount | grep "/dev/"

Make sure that the first line (usually your root) is something like /dev/nvme0n1p2 (the NVMe drive) and not /dev/mmcblk0p2 (that would be the eMMC or a SD card)!

Benchmarking / Testing Storage

Once you’ve booted up you can verify your drive’s performance using my storage benchmark with:

sudo curl https://raw.githubusercontent.com/TheRemote/PiBenchmarks/master/Storage.sh | sudo bash

If you search for the model of your drive / eMMC / etc. on pibenchmarks.com you can compare your score with others and make sure the drive is performing correctly!

Verify Bootloader / Firmware

You can verify your firmware version once you boot the Compute Module 4 by running rpi-eeprom-update which should look like this:

sudo CM4_ENABLE_RPI_EEPROM_UPDATE=1 rpi-eeprom-update

BOOTLOADER: up to date
   CURRENT: Tue 06 Jul 2021 10:44:53 AM UTC (1625568293)
    LATEST: Tue 06 Jul 2021 10:44:53 AM UTC (1625568293)
   RELEASE: stable (/lib/firmware/raspberrypi/bootloader/stable)
            Use raspi-config to change the release.

  VL805_FW: Using bootloader EEPROM
     VL805: up to date
   CURRENT:
    LATEST:

The firmware version for “CURRENT” should match the exact version you used for pieeprom.original.bin in our curl command we used when we were preparing the “nvme” folder earlier regardless of what is available in the repository.

Conclusion

The configuration is definitely not as easy as things on previous Pis but this is the bleeding edge! It should get easier over time as NVMe support slowly makes it way into the “stable” branches.

I’m excited to experiment more with the PCI express capabilities and see what more the Compute Module 4 is capable of. I also have ordered a 8 GB CM4 that has eMMC as well as the built in WiFi (missing on my current model and it’s only a 2GB) which should let me experiment with some of the other smaller IO boards available and some of the other compute module specific capabilities that look really exciting as well.

I have 1x PCI risers that I use for cryptocurrency mining with GPUs that would theoretically allow me to connect some interesting things like a GPU, a Intel Optane 900p PCI express drive, and a few other wild ideas! This is just scratching the surface of what the PCIe capabilities could enable on the Pi.

These are by far the highest benchmarking scores I’ve ever got and at a score of nearly 20,000 it put me on the top scores page and was definitely well worth the setup and configuration!

guest
2 Comments
Inline Feedbacks
View all comments