Minecraft 1.17 (Caves and Cliffs Update) is here! This script and guide are written to help you get a great performing Raspberry Pi Minecraft server up and running in only a few minutes.
This is the standalone version. It runs on most flavors of Linux and should work on most architectures as well (arm, aarch64, etc.).
I highly recommend using Docker over the standalone version for most people. Installing Docker is as simple as sudo apt install docker.io. There are 3 Docker options available:
Sets up Minecraft as a system service with option to autostart at boot
Automatic backups to minecraft/backups when server restarts
Updates automatically to the latest version when server is started
Easy control of server with start.sh, stop.sh and restart.sh scripts
Optional scheduled daily restart of Pi using cron
Requirements
Raspberry Pi model with 1 GB of RAM or higher. Basically a Raspberry Pi 2B or higher. (No Zero unfortunately, 512MB is not enough RAM to do this, I’ve tried!)
Headless Linux distribution such as Raspbian Buster Lite, Ubuntu Server 18.04.2, or any Debian based distribution (GUI distros can be used at the expense of available RAM and server performance)
Solid state drive highly recommended but not required. You can get a SSD setup on a Pi for less than most Micro SD cards cost. See my article here for details
The Raspberry Pi 4 is available in different memory configurations all the way up to 8 GB. It’s about the size of a credit card and uses an extremely low amount of power making it ideal for all sorts of projects and ideas!
The Raspberry Pi 400 kit includes everything you need for a full Pi 400 desktop build. The Pi 400 is the fastest Raspberry Pi ever released and comes in the form factor of a keyboard!
The Kingston A400 has been a great drive to use with the Pi for years. It’s reliable, widely available around the world, has low power requirements and performs very well. It’s also very affordable. This drive has been benchmarked over 1000 times at pibenchmarks.com and is the #1 most popular SSD among the Pi community!
The USB 3.1 variant of the StarTech 2.5″ SATA adapter works well with the Pi 4. The USB 3.0 variant doesn’t have firmware updates available and is not recommended.
The SanDisk Extreme A1-A2 SD card has the best scoring SD card on Pi Benchmarks for years and is second in popularity only to the SanDisk Ultra (often included in combo kits). The application class (A1) means random I/O speeds (very important when running an OS) have to meet a higher standard. There’s no benefit on the Pi for A2 right now so get whichever is cheaper/available.
The most important consideration when choosing which flavor of Linux to run the server on is simple: available RAM. Headless Linux distributions such as Raspbian Lite that don’t have a built in GUI have
Our biggest obstacle when running a Minecraft server on the Pi is available RAM since 1 GB is extremely low for this type of server. To have a playable experience you should not be running anything else on the Pi so all memory is available to be used.
After testing on many different distros I am finding Raspbian Lite and Ubuntu Server 18.04.4 32-bit to be the best choices. These distributions come with very few background processes and have rock solid support and performance.
64-bit vs 32-bit
There’s a lot of discussion in the Pi world about the up and coming aarch64 64-bit distributions vs. armhf 32-bit distributions. They have been and continue to improve dramatically. There are already use cases where 64-bit is far superior such as video encoding, advanced compression, etc.
So how about for running a Minecraft server? I have been testing extensively with Ubuntu Server 18.04 64-bit and the Debian Buster 64-bit. I have consistently had worse performance and stability than on 32-bit versions of the exact same distros.
But how can that be? It’s certainly true that Minecraft servers benefit in CPU performance from 64-bit versions of Java. The answer is actually incredibly simple: memory. The server running on a 64-bit Java Virtual Machine uses a minimum of about 100 MB more memory. This makes perfect sense because 64 bits > 32 bits by definition!
The Raspberry Pi’s 1 GB of memory has been the biggest obstacle for this project since the very beginning. Back when I first went into the Paper Minecraft developer IRC room and told them what I was trying to do I was practically laughed out of the chat room for even thinking of trying this. Most Minecraft server branches including vanilla can’t even start on the Pi because of the limited memory.
For a dedicated Minecraft server on the Pi I very highly recommend staying 32-bit. You will have more available memory which means it will be much faster and more stable. Since memory is our bottleneck the increased CPU throughput does not help us and losing *any* of our memory is disastrous!
If the Raspberry Pi 4 has more memory like we all expect it to this recommendation will change completely. Even 2 GB of memory would make the extra memory that 64-bit uses a non-issue and the CPU throughput performance gains very desirable. For now though stay 32-bit for a Minecraft server!
Tested Distributions
Raspberry Pi OS – It’s Raspbian. It has very low memory usage and is the official distribution of the Raspberry Pi. The server runs very well on this. It’s overall the best choice. The Buster release has made OpenJDK 11 available on it so it’s no longer behind the rest of the distros.
Ubuntu Server 18.04 / 20.04 – Ubuntu Server is my favorite Linux distro. I use it for nearly all of my projects. The performance of the 32-bit armhf version is on par with Raspbian. It’s a great choice! Click here for my Ubuntu setup guide for Raspberry Pi. The 64-bit version is not a fantastic choice and not recommended because of the higher memory usage. Stick with 32-bit and you’ll be a happy camper with Ubuntu Server.
Debian Buster 64-bit – Debian is the distribution Raspbian is based on. This version is a preview of Debian “Buster” which is the successor to Stretch and will be the next version of Raspbian when it is released. I like this distribution but it is currently still unofficial and unsupported. Performance and stability was less than Ubuntu and Raspbian.
Minecraft Server Installation
SSH into your Raspberry Pi and paste the following commands:
The script will setup the Minecraft sever and ask you some questions on how to configure it. I’ll explain here what they mean.
“Enter amount of memory in megabytes to dedicate to the Minecraft server” – The amount of memory that will be dedicated to the Minecraft server. The more the better, but you must leave some room for the operating system background processes.
If you exceed the total available memory either the server will crash or the Pi will get incredibly slow to the point where your SSH session will start timing out. The setup script will make a recommendation to you which is your available memory – 10% for headroom. If you aren’t sure what to put just go with the recommended amount.
Note for Raspberry Pi 4: Currently on 32-bit Raspbian 2700 MB is the maximum that Linux will let us allocate in a 32 bit environment. The script has been updated to check for this as the server will not start if it is set over 2700M on a 32 bit server. 64 bit operating systems will be able to allocate all available memory as Pi 4 support rolls out for them.
“Start Minecraft server at startup automatically (y/n)?” – This will set the Minecraft service to start automatically when your Pi boots. This is great because whenever you want to play you can just plug it in and go without having to SSH in.
“Automatically reboot Pi and update server at 4am daily (y/n)?” – This will add a cron job to the server that reboots the Pi every day at 4am. This is great because every time the server restarts it backs up the server and updates to the latest version. See the “Scheduled Daily Reboots” section below for information on how to customize the time or remove the reboot.
That is it for the setup script. The server will finish configuring and start!
Check Java Version
Sometimes if you have multiple versions of Java installed the wrong version of Java will be selected as the default. If the server didn’t start check that the right version of Java is selected with this command:
sudo update-alternatives --config java
If you get the message “update-alternatives: error: no alternatives for java” then you only have one version of Java installed and can skip to the next section.
If you are presented with a list of choices then your machine has multiple versions of Java installed. It will look like this:
update-alternatives: warning: /etc/alternatives/java has been changed (manually or by a script); switching to manual updates only
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 manual mode
2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
You will usually want to just select the newest version of OpenJDK that is listed so you would type 0 and press enter. In some cases on some platforms you may want to switch to the official Oracle JDK although I strongly recommend sticking with OpenJDK!
First Run
The first time you run the server it will take a little longer to start since it is generating all the server data. If you try to log in before it fully starts you will get a connection timeout error. Watch for the line: “Timings Reset”. This is the last line that prints when the server is ready to rock and roll. At this point you will be able to connect successfully.
The very first time you log into the server it will be slow for about 1-2 minutes. This is because since nobody has logged in before the server has to scramble to generate all the chunks within your view distance (10 by default) and send them to you/store them. During this time you may not be able to see very far and if you try to destroy blocks there will be noticeable lag from when they break to when they actually disappear.
Don’t panic! This will go away within a couple of minutes as the Pi catches up with all the first time login stuff it needs to do. Performance stabilizes and it will feel very much like the offline experience after that.
If you are hosting for a few friends I’d recommend logging in for the first time right after you set up the server instead of having several people nail a blank server at first startup. This gets it out of the way and when everyone is ready to log in the starting area chunks will be fully fleshed out and the Pi just has to read them. It’s an order of magnitude faster for the Pi to read chunks than to generate and store chunks.
In my experience after the initial login exploring new parts of the server doesn’t cause any lag even though new chunks are being generated. The reason for this is that when you’re walking it’s really only having to generate a new chunk as you get close to the border instead of a huge square area of chunks in all directions and all at the same time like during the first login.
Benchmarking / Testing Storage
If you’re getting poor performance or just want to verify everything is working correctly you may want to run my storage benchmark with:
Now press Ctrl+X to exit nano and answer “y” to save. Now let’s run the script:
chmod +x SetupMinecraft.sh ./SetupMinecraft.sh
And the setup will run and install the version of Minecraft you set at the top of the file!
Changing Minecraft Client Version
If you are wisely running the “stable” branch instead of the “development” branch there will be times where you need to select the version of Minecraft to run otherwise you will get an error message that your client is outdated when you try to log in.
Fortunately this is very easy. Open up the Minecraft launcher and instead of hitting “Play” choose “Launch Options” in the menu at the top of the window. It will look like this:
Minecraft Launcher “Launch Options” Tab
Click the “Add new” button and pick which version you want to add. You can optionally gave it a name or just click save.
Now when you go back to the “News” tab you will see a dropdown arrow where you can select which version of Minecraft you want to play!
Start, Stop and Restart Server
The server can be started, stopped and restarted two different ways. You can use the provided scripts in the Minecraft folder or you can use systemctl. Here are the commands:
The server backs up each time it starts. This helps you recover easily if something goes wrong. This system works best if you configured the server to restart daily since it means you will have a backup every day.
To access these backups type:
cd ~/minecraft/backups ls
When a backup is made the filename will be the date and time the backup was taken. If you need to restore a backup it’s very easy. Substitute the timestamp in my example to the backup you want to roll back to. Type:
cd ~/minecraft
./stop.sh
rm -rf world world_nether world_the_end
tar -xf backups/2019.02.15.22.06.30.tar.gz
./start.sh
Your world has now been restored! It’s a good idea to download these backups off the Pi periodically just in case the Pi’s storage fails.
Scheduled Daily Reboots
The daily reboots are scheduled using cron. It’s very easy to customize the time your server restarts.
To change the time that the server restarts type: crontab -e
This will open a window that will ask you to select a text editor (I find nano to be the easiest) and will show the cronjobs scheduled on the Pi. The Minecraft one will look like the following:
0 4 * * * /home/ubuntu/minecraft/restart.sh
There are 5 fields here. The default restart time is set to reboot at 0 minutes of the 4th hour of the day (4 AM). The other 3 fields are left as * to represent every day of every month. Make any desired changes here and press Ctrl+X to exit nano and update the cronjob.
To remove the daily reboot simply delete the line and save.
Installing Mods / Plugins
The server supports plugins that are compatible with Bukkit / Spigot / Paper. A popular place that you can get plugins is at dev.bukkit.org where there are thousands of them!
To install a plugin you simply download the .jar to the minecraft/plugins folder and restart the server. For example, WorldGuard is a very popular plugin that lets you add protection to different areas of your server.
To install this plugin on our Minecraft server we would use the following commands:
cd ~/minecraft/plugins
curl -H "Accept-Encoding: identity" -H "Accept-Language: en" -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.12.212 Safari/537.36" -o worldguard.jarhttps://dev.bukkit.org/projects/worldguard/files/latest
sudo systemctl restart minecraft
The reason the middle line is so long is that “robots” (roughly anything that isn’t a web browser being used by a user in this context) including scripts and utilities are blocked by the Bukkit server. The extra parameters we’re including in this line will add the location (-L) flag as well as a user agent and an identity / language header which will allow us to fetch the files without getting a 403 forbidden error.
Make sure to change “-o worldguard.jar” (second to last parameter) and the URL (very last parameter) to match the project you want to download.
The server will restart and the plugin will be installed. It’s that simple! To use the plugin refer to the documentation on the plugin download page to find out which commands you use to configure/interact with it.
Warning: be advised that plugins are the #1 issue for performance degradation on Minecraft servers. This isn’t because all plugins are bad. Some plugins are coded very inefficiently or perform features that require a lot of hooks in the code.
You should be careful about what plugins you install on the server and if you start having bad performance disable your plugins one by one until you find the culprit!
Reconfigure / Update Scripts
The scripts can always be reconfigured and updated by downloading the latest SetupMinecraft.sh and running the installer again. It will update all of the scripts in the Minecraft directory and reinstall the startup service for you.
Running SetupMinecraft.sh again will also give you a chance to reconfigure options such as the memory dedicated to the server, daily reboots, starting the server on boot, etc.
This will not overwrite your world or any other data so it is safe to run!
Port Forwarding
If everyone on your server is on the same LAN or WiFi network as you then you don’t need to do this. If you want people to connect from outside your local network then you need to set up port forwarding on your router.
The process for this is different for every router so the best thing to do is just look at your router and find the model # and put that in google with port forwarding for easy instructions on how to do it for your specific router.
You want to forward port 25565. The type of connection is TCP if your router asks. Once you do this people will be able to connect to your Minecraft server through your public IP address. This is different than your local IP which is usually a 192.x.x.x or 10.x.x.x. If you don’t know what that is just go to google and type “what’s my ip” and Google will kindly tell you!
Wired vs. Wireless
Going with an ethernet (wired) connection is going to be faster and more reliable. There’s so much wireless traffic and other interference in the air that running your server on WiFi is not recommended.
Even if it is working great 99% of the time it can ruin your experience very quickly if the WiFi drops for a couple of seconds and you get blown up by a creeper!
All that being said, the server works fine on wireless. The script will work fine as is with a wireless connection.
Upgrading
PLEASE BACK UP YOUR SERVER FIRST! The server makes automated backups by default for you in the backups folder but I recommend you back up the entire server folder yourself (basically the entire minecraft folder) any time you attempt to upgrade or downgrade. If you need to roll back to older versions it won’t work without a backup from that version or older!
The easiest way to upgrade an installation is to download the latest SetupMinecraft.sh and run it. This will automatically upgrade you to the latest version.
Upgrading and downgrading to versions that aren’t the default the script chooses is pretty simple. Simply change the Version line at the top in the SetupMinecraft.sh script:
#!/bin/bash # Minecraft Server Installation Script - James A. Chambers - https://jamesachambers.com # More information at https://jamesachambers.com/raspberry-pi-minecraft-server-script-with-startup-service/ # GitHub Repository: https://github.com/TheRemote/RaspberryPiMinecraft
# Minecraft server version Version="1.16.1"
Edit this file in your favorite text editor (you can use nano or vi on the Pi like nano SetupMinecraft.sh) and change the Minecraft version to what you want.
Downgrading
PLEASE RESTORE USING A BACK UP FROM THE VERSION YOU ARE DOWNGRADING TO
If you are having problems on a newer version of Minecraft and want to downgrade you can do so using a complete backup of your server before you ran it on a newer version.
The reason you can’t take server data that has been touched by a version such as 1.17 and go back to 1.16 is that the new version adds all sorts of new data types/structures for the new content into your server data files. If you try to roll back the old versions of the Minecraft server will not understand these data types since they didn’t exist in that version and will crash.
As long as you use a backup for your server files from that version (or older) it’s as simple as changing the version in SetupMinecraft.sh just like I show in the “Upgrading” section.
You can upgrade any old version of Minecraft to any version, but again make sure you have a backup first as it is a one way street and you will need that backup if you want to roll back!
Troubleshooting Note – Oracle Virtual Machines
A very common problem people have with the Oracle Virtual Machine tutorials out there that typically show you how to use a free VM is that the VM is much more difficult to configure than just about any other product / offering out there.
It is because there are several steps you need to take to open the ports on the Oracle VM. You need to both:
Set the ingress ports (TCP/UDP) in the Virtual Cloud Network (VCN) security list
*and* set the ingress ports in a Network Security Group assigned to your instance
Both of these settings are typically required before you will be able to connect to your VM instance. This is purely configuration related and has nothing to do with the script or the Minecraft server itself.
I do not recommend this platform due to the configuration difficulty but the people who have gone through the pain of configuring an Oracle VM have had good experiences with it after that point. Just keep in mind it’s going to be a rough ride through the configuration for most people.
Troubleshooting Note – Hyper-V
There is a weird bug in Hyper-V that breaks UDP connections on the Minecraft server. The fix for this is that you have to use a Generation 1 VM with the Legacy LAN network driver.
Version History
To view the version history check out the GitHub README here:
My server has been running fine for several days but when I went to use it this morning, it was not running. When I investigated, there seems to be an error with sudo and “pam_unix”. I don’t think this is related to your script (other than the 4am reboot) but was hoping you might know what is happening here.
Thanks
—-
Mar 23 11:44:55 paynecraft systemd[1]: Starting Minecraft Server Service...
Mar 23 11:44:55 paynecraft bash[948]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or confi>
Mar 23 11:44:55 paynecraft bash[948]: sudo: a password is required
Mar 23 11:44:55 paynecraft sudo[948]: pam_unix(sudo:auth): conversation failed
Mar 23 11:44:55 paynecraft sudo[948]: pam_unix(sudo:auth): auth could not identify password for [mcadmin]
Mar 23 11:44:55 paynecraft bash[956]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or confi>
Mar 23 11:44:55 paynecraft sudo[956]: pam_unix(sudo:auth): conversation failed
Mar 23 11:44:55 paynecraft sudo[956]: pam_unix(sudo:auth): auth could not identify password for [mcadmin]
Mar 23 11:44:55 paynecraft bash[956]: sudo: a password is required
Mar 23 11:44:55 paynecraft bash[957]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or confi>
Mar 23 11:44:55 paynecraft bash[957]: sudo: a password is required
Mar 23 11:44:55 paynecraft sudo[957]: pam_unix(sudo:auth): conversation failed
Mar 23 11:44:55 paynecraft sudo[957]: pam_unix(sudo:auth): auth could not identify password for [mcadmin]
Mar 23 11:44:55 paynecraft bash[946]: Backing up server (to cd minecraft/backups folder)
Welcome back! That is a strange one. The first thing I would try is running SetupMinecraft.sh as this will set up your sudoers file again (it’s at /etc/sudoers.d/minecraft). I believe you are correct that it’s sudoers related. You could also manually look in there and see if the file is missing or if something looks wrong / different.
Since you weren’t trying to update nothing should have changed (other than maybe Paper Minecraft itself updating to a newer revision that is doing this) at least with the scripts themselves. I’d be curious what you find the sudoers file situation is as this is what I would expect to happen if it was disabled / missing (maybe due to apt updates, hard to say why).
Which OS and version is this? I’m guessing your pam.d configuration is modified / messed up (probably from apt updates). This seems especially likely since it happened out of the blue.
I’m not sure what your comfort level is here but it looks like if the apt breaking pam.d theory is correct then you may just need to add:
auth sufficient pam_permit.so
in the file like the answer that was picked in my link has. If this is Raspberry Pi OS or Ubuntu installed on the Pi I would say it would be very wise to reimage this Pi and then just restore the server from a backup. It seems like the Pi’s image is not sound here. You can definitely try just patching it up and see if that resolves it though!
Mar 23 23:27:14 paynecraft sudo[5588]: mcadmin : PWD=/home/mcadmin/minecraft ; USER=root ; COMMAND=/usr/bin/sh -c echo 1 > /proc/sys/vm/drop_caches
Mar 23 23:27:14 paynecraft sudo[5588]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Mar 23 23:27:15 paynecraft sudo[5588]: pam_unix(sudo:session): session closed for user root
Mar 23 23:27:15 paynecraft sudo[5597]: mcadmin : PWD=/home/mcadmin/minecraft ; USER=root ; COMMAND=/usr/bin/chown -Rv mcadmin /home/mcadmin/minecraft
Mar 23 23:27:15 paynecraft sudo[5597]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Mar 23 23:27:16 paynecraft sudo[5597]: pam_unix(sudo:session): session closed for user root
Mar 23 23:27:16 paynecraft sudo[5599]: mcadmin : PWD=/home/mcadmin/minecraft ; USER=root ; COMMAND=/usr/bin/chmod -Rv 755 /home/mcadmin/minecraft/fix>
Mar 23 23:27:16 paynecraft sudo[5599]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Mar 23 23:27:16 paynecraft sudo[5599]: pam_unix(sudo:session): session closed for user root
Mar 23 23:27:16 paynecraft bash[5586]: Backing up server (to cd minecraft/backups folder)
But the server still will not start. When I run “screen -r” I see Bukkit starting for a second and then the screen exits.
Can you try reverting your snap package like he did to OpenJDK 17? This is going to be a pretty big problem since the OpenJDK in the Raspberry Pi repositories is still too old (Ubuntu’s might be new enough). I think if you try those instructions it should get it going!
It looks like they made it impossible to install an older version than OpenJDK 18 from snap as of March 22nd (yesterday). Hopefully I removed it before too many people get upgraded. You should still have your old snap that you can revert to.
Raspberry Pi OS only seems to have OpenJDK 17 documentation available on the armhf version in the apt repositories. I’ll have to test the 64 bit version and see if it’s available on there. I did add some additional apt selections to SetupMinecraft.sh that should pick these up on Ubuntu and Raspberry Pi OS (if it’s available, doesn’t seem to be on 32 bit but I’m testing now as I’ll probably have to find a compatible package to download).
It’s going to be trickier for new installations. I’m going to have to figure out how to install OpenJDK 17 reliably for now. That’s very annoying it’s still not compatible when it’s now in the main release channel but I’ll figure something out!
Thanks for thinking to run it manually like that as this was a very unexpected outcome. They literally just switched the repositories over on the 22nd so this basically just happened. The snapd installation is already out of SetupMinecraft.sh and you thinking to test this manually definitely helped me to find it faster!
Now that it’s out of SetupMinecraft.sh anyone who is upgrading from an old installation would just keep whatever version they have installed on there (far more likely to be right than the current snapd version which is certain to be wrong). I just need to find a way to get OpenJDK 17 on new installations which I’ll be working on for the next day or two on 32 and 64 bit test systems.
My “snap” syntax appears to be different than Markus’:
snap list openjdk --all
sudo snap revert openjdk --revision=818
But after applying that, my server is back up and running.
Who would have guessed that a guy who sells QA software for a living could actually help debug an issue. Thanks for your help…and your wonderful script.
Excellent, sorry that you got caught up in that! That makes sense that you have a different revision. The OpenJDK snap doesn’t “allow” reverting to older versions *unless* you already had one installed. This will be different for everyone unfortunately as it will be whatever OpenJDK revision was live when they installed that they will have. This means this trick won’t work for new users, or if you wiped the Pi it would also no longer work. It also means everyone has a different version which makes it a lot dicier to try to automate this. That’s why I outright removed the snapd installation piece as since only OpenJDK 18+ is available now via snap literally nobody should ever use it now as it will only break it.
This appears to be a design choice by OpenJDK (they do HARD cutoffs of the OpenJDK versions and they only last for about 6 months with the exception of a LTS release that is still only good for 2 years) and it is going to be a nightmare to deal with. I just updated my Pi 400 to Bullseye to see what is available there but it looks like builds of OpenJDK 17 for the Pi are very difficult to find. They also work on different platforms. The Pi Zero W for example doesn’t have an OpenJDK anywhere near 16 available. It may look something like the 64 bit flavor of Raspberry Pi OS can install a new enough one and the 32 bit one can’t unless I give it some help with a third party package.
There’s a few different ways I could deal with this and I’m trying to find the easiest and most widely compatible / portable one. One option I’ve been thinking is I could have the project have it’s own “JDK” where it downloads and stores the JDK in your minecraft folder such as ~/minecraft/jdk for example as a base JDK for all servers installed. I’m really hoping I can find a way to take care of this with apt though (at least on Bullseye for Raspberry Pi OS).
I’m a little stunned that Paper still doesn’t support OpenJDK 18. They’ve always caught up before this happened in the past and it was safe to just keep using the newer OpenJDK but it’s different this time. This may not be entirely Paper’s fault. There was some very strange behavior happening with OpenJDK 18 to you guys that I’ve never seen before. It’s probably structural changes within Java that are making this a challenge.
I’m always thrilled to have you help debug! It’s kind of funny because all of the issues for the past few days have required me to make fixes to the script. Usually I go months at a time without anyone finding one of these. It definitely comes in waves.
I’ve fixed this once and for all. The new version of SetupMinecraft actually sets up a dedicated OpenJDK (stored in your server’s folder under jre). It’s never too hard though thanks to you guys, it is appreciated!
I’m having the same issue. Pretty newbie question (which I am), but what’s the easiest way to perform the update of SetupMinecraft, when you have an existing installation?
Great question! Basically just run SetupMinecraft.sh again and leave everything default / the same as it was and it will detect and update your existing server.
The new version takes care of OpenJDK a different way by downloading OpenJDK 17 and putting it in the Minecraft folder.
You’ll basically just enter the same things for the server as you did the first time (or leave them as default). If you have any trouble let me know!
Thanks James. In that case, I’m not sure what I’ve done. Should SetupMinecraft.sh still be present from the original install or do I need to re-download it? I only ask, as I can’t see it installed currently.
In other words, the process to update the scripts is the same as installing it in the first place. It’s designed to safely do this and update the existing scripts.
Sorry for not being more clear the first time, hopefully that gets it for you!
I’ve been struggling with this error too – server works fine, but those log messages are annoying, particularly as I’m also adding a notification on minecraft join/leave via a comobination of tail (the log file) and ssmtp – so given the mailbad statement means I get emails telling me about these failures. #
The fixes noted in the original thread (editing PAM modules) looked a bit dangerous to me (basically making sudo passwordless throughout) as highlighted in the linked StackOverflow thread.
Looking at the code I wonder if it is an issue with fixpermissions.sh and some of the calls? Two potential areas to look at (and apologies for formatting – first post here so this will be garbled, no idea how to quote etc):
1. in fixPermissions.sh there is this segment when testing the options for flags (starts line 20)…
—————— while getopts ":a:" opt; do
case $opt in
t)
case $OPTARG in
case $OPTARG in
''|*[!0-9]*)
Automated=1
;;
*)
Automated=1
;;
esac
—————–
That case statement looks a little funny? Shouldn’t it be a) rather than t)? (the case statement goes on to note countdown etc rather than automation). Similarly, we are not after an actual value are we, so we could just have getopts “:a” (i.e. no second colon), and simplify the case statement? The upshot of this is that when fixPermissions,sh is called from start.sh, it is not setting the Automated flag and therefore not using the sudo -n statement further on down. However, even if you make those changes, so still get similar sudo errors requiring passwords, even if sudoers.d/minecraft is all OK
2. The second problem then is looking in sudoers.d/minecraft. That gives the local user passwordless permission to run fixpermissions.sh, start.sh etc. However, the actual sudo commands (e.g. chown in fixpermissions.sh, sh in start.sh) will only inherit that status, if those scripts are called via sudo. The minecraft service definition however (under /etc/systemd/ssytem/minecraft.service) includes this:
———————- [Service]
User=[user]
WorkingDirectory=/home/[user]/minecraft
Type=forking
ExecStart=/bin/bash /home/charon/minecraft/start.sh
———————-
Note the User= command. The default for systemd on startup is to run as root, however as I understand it, when User is set in the service that is no longer the case. Therefore, that ExecStart will not be called via su, and therefore that elevated status not passed onto calls, so when the script eventually hits the sudo commands, the owning script has not been invoked by sudo and therefore scripts are required. Two solutions, either add chown and sh to sudoers.d (not a good idea possibly), or force a call via sudo in the service i.e. change the ExecStart line to:
————————-
ExecStart=/usr/bin/sudo /bin/bash /home/charon/minecraft/start.sh
————————–
I made all the above changes and rebooted and those errors have now gone away (no idea if it makes any difference at all).
As noted, well beyond my comfort zone with a lot of this, so apologies if foolish errors (and lack of formatting)
It basically already works this way. You can see that fixpermissions is allowed to run in it’s entirety sudoless. Is this a weird operating system like a Red Hat operating system? Some of those have different sudoers file formats and maybe yours is one of the unsupported/untested platforms. I’m aware that these guys are coming to Raspberry Pi real soon as well via Fedora and I’m not sure if that one uses a nonstandard sudoers file yet (I hope not because that one will be officially supported and I don’t want a bunch of janky code making special exceptions for Red Hat in this, there’s nothing special about not conforming to standards other than being garbage/annoying).
If you aren’t on the Raspberry Pi that is okay. This works fine on everything else as well (usually, you may have found one of the exceptions). That’s just traditionally what was popular about this script and is why it hasn’t come up a whole lot. It’s possible if Fedora requires a different sudoers format that I will have to upgrade this somehow to be able to tell the difference between them.
I can just delete all that case stuff out of there. I didn’t write it. It was part of a pull request and I’m totally fine with deleting all of it. I can delete 95% of that code without any consequences. I just did this. All of that code is now deleted and fixpermissions only contains this:
echo "Taking ownership of all server files/folders in dirname/minecraft..."
sudo chown -Rv userxname dirname/minecraft
sudo chmod -Rv 755 dirname/minecraft/*.sh
That script shouldn’t have mattered unless your server was already broken. Did you manually edit server files / permissions or move them around? Fixpermissions is not required to run the server. It only fixes it if the server files have the wrong permissions on it.
I’m going to guess you are running one of the few operating system that implements their own format of sudo that is different from every other operating system out there. Can you confirm this? Needless to say I disagree with them that their sudoers format is the one true format and do not plan on supporting it if that’s the case (gladly, without standards most of the advantages of Linux start to disappear). I hate operating systems / flavors that do it completely different from any others for no reason and I feel like that has to be part of this one!
It has come up once or twice before but it has been a while since I saw one with a different sudo system. If you have this I would highly recommend using the Docker version because you don’t have a compatible flavor of Linux that implements standard sudo (if this is the case). They basically aren’t conforming to Linux standards and are doing their own thing in some cases (and a couple of big names do this although they are usually premium/enterprise like Red Hat).
It’s also possible you just had broken permissions on one of the files. If this is all it was then this is now fixed. I completely revamped that automated code part (as in deleted it). It was throwing Snyk warnings so I have no doubt something was wrong with that piece of it but again this code does nothing unless your server’s permissions were wrong in the first place (which should never happen unless things have been moved around or files were edited by a different user and now they have ownership).
Given how you described your issue though I feel like if you did a fresh install then you’d have to make similar modifications. If that is the case then this is a non-standard Linux distro using whatever sudo format they feel like I guess. If you are comfortable with this then there’s no reason now that you know what to do you couldn’t absolutely move forward as the updates will all still happen normally and should not be impacted by any of this. Hopefully that helps!
Except that now checking it I see that the server is not running on reboot – start.sh traps out root calls at the start (sensibly). So above a broken solution (sorry) but hopefully helpful in some way.
Can you verify your /etc/sudoers.d/minecraft file? It just sounds like it’s not working to me. I think you’re right that you’re having sudo issues. I’d imagine it’s because of your operating system though! If you verify the sudoers file you’ll likely find it’s the wrong format for your OS. I can’t help but notice you did not state which OS this is. If you ran this on Ubuntu or Debian or one of those flavors this does not happen so I am going to need to know that to do anything to help.
This is exactly what I would expect to happen if the sudoers file isn’t working. It can’t run sudo systemctl start without it.
This is not something I could fix for you without writing special code for your operating system to make a sudoers file in the format your OS will take. I would need to know what OS we’re talking about as well as this cannot be a standard/common one. It has also only come up 1 or 2 times ever in the 1000+ comments on this article.
I’m definitely willing to explore fixing this depending on how obscure your OS is. If it’s Red Hat/RHEL after all then I may do it just to prepare for Fedora to come to the Raspberry Pi officially. That is what we are talking about though. We are talking about supporting your flavor of Linux. It already works on the OS that 90%+ of people are running. It does not look like it currently supports whichever one you are running though and frankly you may be the first person to ever try depending on which one it is.
There is nothing wrong with the script. I need to know what uncommon OS is being ran here. I will then explain and link to explanations about why your OS doesn’t work the same way as everyone else’s and what you can do about it (or potentially what I can do about it as well depending on what the problem is). Issues like this are why I have moved away from standalone scripts to Docker for everything new and things like my Geyser + Floodgate container don’t have a standalone version. It’s so difficult to support standalone when there are different competing standards being used.
I have 2 standalone scripts and 4 Docker containers at this point and the 2 standalone ones I did are from 4-5 years ago and all the Docker containers are pretty much from this year. That is basically directly because of issues like this. Everyone using the Docker container will have a better experience and never see any issues like this ever because it’s all standardized. Because of the container everyone is really running Ubuntu within the container. Linux distros aren’t always similarly standardized unfortunately (but they’re usually close or this wouldn’t have worked at all in general and would have had to have a list of specifically supported operating systems/flavors).
Been looking at this for a few hours or so on the internet, and a common issue apparently is overwrites in /etc/sudoers, but the @includedir /etc/sudoers.d is at the bottom of that file so can’t be that.
For confirmation, sudo -ll gives the following:
——————————————- Matching Defaults entries for charon on pluto:
env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/snap/bin, use_pty
User charon may run the following commands on pluto:
Sudoers entry:
RunAsUsers: ALL
RunAsGroups: ALL
Commands:
ALL
Sudoers entry:
RunAsUsers: ALL
Options: !authenticate
Commands:
/bin/bash /home/charon/minecraft/fixpermissions.sh
/bin/systemctl start minecraft
/bin/bash /home/charon/minecraft/start.sh
/sbin/reboot
———————————–
… so assuming that !authenticate means not authenticate, then sudoers.d looks to have bedded in as intended.
Have tried rerunning the original script a couple of times to reset my tweaks to scripts back to default entries … and no change to those log entries. Just spied your update to fixpermissions.sh on github though so will retry that.
Hope something useful in there.
Re your message above, this is my ignorance I’m afraid. The way I was figuring it was that that the systemctl service is *not* running as root (as User is defined in the service definition), therefore fixpermissions will not be called as root/su and therefore the sudoers.d settings will not be invoked when called and so the embedded “sudo” in that script will want interactive authentication. Similarly to the sudo call in start.sh. Probably some garbled logic in there.
Thanks by the way – great script and very easy to set up. Best by far I found on the internet (and as noted on an N2+ as well). As noted the server is running fine, it is just those sudo calls and log entries, which it appears do not hamper the default functionality too much (noting what you highlight re permissions in the top of the fixpermissions script)
Okay, so what I did was run SetupMinecraft.sh fresh and had it set up a new server with the name “james”. I did this on the default odroid user.
Everything seems to be okay on initial first startup. I did get a really weird dripstone block error for some reason but I’m guessing that’s related to the seed / random. I typed screen -r and the server was running:
Starting org.bukkit.craftbukkit.Main
System Info: Java 18 (OpenJDK 64-Bit Server VM 18.0.2+9) Host: Linux 4.9.312-125 (aarch64)
Loading libraries, please wait...
[20:20:10 INFO]: Building unoptimized datafixer
[20:20:14 INFO]: Environment: authHost='https://authserver.mojang.com', accountsHost='https://api.mojang.com', sessionHost='https://sessionserver.mojang.com', servicesHost='https://api.minecraftservices.com', name='PROD'
[20:20:14 INFO]: Found new data pack file/bukkit, loading it automatically
[20:20:19 INFO]: Loaded 7 recipes
[20:20:23 INFO]: Starting minecraft server version 1.19.2
[20:20:23 INFO]: Loading properties
[20:20:24 INFO]: This server is running Paper version git-Paper-138 (MC: 1.19.2) (Implementing API version 1.19.2-R0.1-SNAPSHOT) (Git: 09904fd)
[20:20:24 INFO]: Server Ping Player Sample Count: 12
[20:20:24 INFO]: Using 4 threads for Netty based IO
[20:20:25 INFO]: Default game type: SURVIVAL
[20:20:25 INFO]: Generating keypair
[20:20:25 INFO]: Starting Minecraft server on *:25565
[20:20:25 INFO]: Using epoll channel type
[20:20:25 INFO]: Paper: Using libdeflate (Linux aarch64) compression from Velocity.
[20:20:25 INFO]: Paper: Using OpenSSL (Linux aarch64) cipher from Velocity.
[20:20:26 INFO]: Preparing level "world"
[20:21:14 INFO]: Preparing start region for dimension minecraft:overworld
[20:21:16 INFO]: Time elapsed: 1470 ms
[20:21:16 INFO]: Preparing start region for dimension minecraft:the_nether
[20:21:16 INFO]: Preparing spawn area: 0%
[20:21:17 INFO]: Time elapsed: 1307 ms
[20:21:17 INFO]: Preparing start region for dimension minecraft:the_end
[20:21:18 INFO]: Preparing spawn area: 0%
[20:21:18 INFO]: Preparing spawn area: 0%
[20:21:18 INFO]: Preparing spawn area: 0%
[20:21:19 INFO]: Preparing spawn area: 0%
[20:21:20 INFO]: Preparing spawn area: 0%
[20:21:20 INFO]: Preparing spawn area: 0%
[20:21:20 INFO]: Preparing spawn area: 0%
[20:21:21 INFO]: Time elapsed: 4012 ms
[20:21:21 INFO]: Running delayed init tasks
[20:21:21 INFO]: Done (57.814s)! For help, type "help"
[20:21:21 INFO]: Timings Reset
[20:21:34 ERROR]: Detected setBlock in a far chunk [-6, -1], pos: BlockPosition{x=-82, y=-11, z=-1}, status: minecraft:features, currently generating: ResourceKey[minecraft:worldgen/placed_feature / minecraft:large_dripstone]
>
Okay, so then I answered “Y” on the daily reboot and “Y” on the start the server automatically at startup. I verified my crontab with crontab -e as:
Okay, let’s test rebooting it with sudo reboot and see if it will come back on it’s own. It did. You can verify this with:
odroid@odroid:/etc$ sudo systemctl status minecraft
● minecraft.service - Minecraft Server Service
Loaded: loaded (/etc/systemd/system/minecraft.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-09-04 20:27:23 UTC; 24s ago
Process: 4006 ExecStart=/bin/bash /home/odroid/minecraft/start.sh (code=exited, status=0/SUCCESS)
Tasks: 45 (limit: 3832)
Memory: 745.6M
CGroup: /system.slice/minecraft.service
├─4073 SCREEN -dmS minecraft /home/odroid/minecraft/jre/bin/java -DPaper.IgnoreJavaVersion=true -jar -Xms400M -Xmx2000M /home/odroid/>
└─4074 /home/odroid/minecraft/jre/bin/java -DPaper.IgnoreJavaVersion=true -jar -Xms400M -Xmx2000M /home/odroid/minecraft/paperclip.jar
Sep 04 20:27:20 odroid bash[4061]: % Total % Received % Xferd Average Speed Time Time Time Current
Sep 04 20:27:20 odroid bash[4061]: Dload Upload Total Spent Left Speed
Sep 04 20:27:21 odroid bash[4061]: [237B blob data]
Sep 04 20:27:21 odroid bash[4006]: Latest paperclip build found: 138
Sep 04 20:27:21 odroid bash[4070]: % Total % Received % Xferd Average Speed Time Time Time Current
Sep 04 20:27:21 odroid bash[4070]: Dload Upload Total Spent Left Speed
Sep 04 20:27:23 odroid bash[4070]: [237B blob data]
Sep 04 20:27:23 odroid bash[4006]: Starting Minecraft server. To view window type screen -r minecraft.
Sep 04 20:27:23 odroid bash[4006]: To minimize the window and let the server run in the background, press Ctrl+A then Ctrl+D
Sep 04 20:27:23 odroid systemd[1]: Started Minecraft Server Service.
This image is basically brand new. Maybe a week ago. I had 13 updates since my initial image of it at least. I’m guessing if you do a:
sudo systemctl status
sudo systemctl list-units --failed
that you have more failed units than just the minecraft server. In the past when people have had startup/service related issues like this it’s usually the sign of a sick systemd/bad image. If this has been upgraded from a previous version like 20.04 and hasn’t been reimaged with a clean 22.04 image that is the first thing I would try with this one. I definitely was able to do this properly on a clean image without anything strange.
Some of the commands I gave you may help in diagnosing this as well. The useful ones are sudo systemctl status minecraft (especially right after you reboot and it’s not coming up) and then looking in the “logs” folder for “latest.log”. Those will have the most technical information about what is going wrong. It should not happen on a clean N2+ 22.04.1 image (or at least I was able to verify that it didn’t). The good news is I’m sure this is resolvable and that your platform is supported. Hopefully that helps!
Thanks James – I did upgrade from an older version (20 or 21 can’t remember) to 22 so that may be it, so I’ll try a download and clean reinstall and report back here (wont be tonight)
No problem, I was not at all sure what was going to happen with this one because I have never tried running it on my ODROIDs yet! I just got my few ODROID models I have here in the past few weeks so this had not come up.
It does look image related though. I had a *ton* of image related issues when I upgraded my ODROID M1 from 20.04 to 22.04 in my ODROID M1 Review. At the time there was no 22.04 image available for the M1 so I ended up recommending that people use the server image first and then upgrade to get the cleanest upgrade possible and then to install their desktop later with apt (sudo apt install ubuntu-desktop or ubuntu-desktop-full).
The N2+ seemed to not have as many issues upgrading from 20.04 to 22.04 as the M1 (as in the M1 straight up couldn’t do it with the desktop image) but that doesn’t mean everything upgraded clean and given the M1 experience there’s no way that this one was just perfect (and indeed some weird things seem to be happening that shouldn’t be). This newer image is definitely available fortunately for the N2+ right now as I used it last week.
Sometimes upgrade problems can also depend on what you answer to the questions about whether to replace config files with new copies or leave the existing copy (sometimes the right answer is keep the custom one if it’s just one you’ve customized and sometimes the right answer is replace it with the newer config if it’s a major overhaul of an existing system because the old config will break it, it just depends). I honestly think that the images just don’t upgrade very cleanly a lot of the time though and that definitely isn’t unique to ODROID for sure (and others are often much, much worse).
In fact my N2+ installation on the eMMC is actually having some upgrade related problems from going from 20.04 to 22.04 (the one I tried last week was for my SSD booting guide for the N2+ and that was a clean install on the SSD but my eMMC install is an upgrade). It’s two annoying snap packages and I honestly don’t even know how to fix this one (it’s the chromium-browser and firefox packages). Observe:
Preconfiguring packages ...
(Reading database ... 220363 files and directories currently installed.)
Preparing to unpack .../00-chromium-browser_1%3a85.0.4183.83-0ubuntu2_arm64.deb ...
=> Installing the chromium snap
==> Checking connectivity with the snap store
==> Installing the chromium snap
error: cannot perform the following tasks:
- Run configure hook of "chromium" snap if present (run hook "configure": cannot perform operation: mount --bind /snap/core20/current/etc/nsswitch.
conf /tmp/snap.rootfs_WcYly0/etc/nsswitch.conf: Permission denied)
dpkg: error processing archive /tmp/apt-dpkg-install-p5rOd0/00-chromium-browser_1%3a85.0.4183.83-0ubuntu2_arm64.deb (--unpack):
new chromium-browser package pre-installation script subprocess returned error exit status 1
Preparing to unpack .../01-firefox_1%3a1snap1-0ubuntu2_arm64.deb ...
=> Installing the firefox snap
==> Checking connectivity with the snap store
==> Installing the firefox snap
They fail every time so even this N2+ upgrade from 20.04 on the eMMC is broken. Even if I can fix this minor package issue I don’t trust it. Who knows what else is wrong with it that isn’t as obvious. This one is due for a clean 22.04.1 reimage just like you’re doing.
Thanks for sticking with it. This was something I definitely needed to test as it’s a little embarrassing to have 3 ODROIDs now and to admit I have not ran my Minecraft server script on any of them yet. Let me know what you find!
Just to close this out, I reinstalled from the ground up tonight, moving to an SSD while I did. I used a totally fresh install of 22.04 (using netboot_install ubuntu mate minimal 22.04 from Petitboot), and beyond installing e.g. wifi drivers, ssh servers, xrdp installed this script. Alas the same warnings appear in journalctl, so not sure what is going on. To be honest it is a nice to have, and I’ll live with it. The Docker solution is one option, but I slightly prefer having the server not in a virtual environment as it facliattes integration with other things going on on the machine.
Thanks for all your help and your great sight – looking through all your old posts now and some really valuable stuff in here
One aside… I was following your guide for booting N2+ from SSD (I’m using a USB one), and interesting the boot arguments in Petitboot was already set as you highlight to sda, it seemed to have autodetected it.
Excellent, I’m very glad you met with success here! I was able to eliminate those warnings by doing a clean 22.04 install but all of my upgrades have them. Most of those services are configured to fall back when this is the case so they just generate annoying warnings. If it’s an actual error that is a different thing / a problem but the warnings usually indicate it’s just going to fall back to the old functionality.
For what it’s worth I was able to get a clean install without the journalctl warnings by completely reinstalling (i.e. not using an upgrade install) of 22.04 right from the start. I did not try with the minimal though. I’m surprised you still get those journalctl warnings using that method as I got the impression it has to do with the partitions and /etc/fstab content when these are set up (related to mounting flags a lot of the time). They definitely do seem to be harmless though and even my upgrade installs with the journalctl warnings functioned well and ran everything I tried to do as normally as the clean install that did not have those warnings present (again for what it’s worth).
My Petitboot did seem to autodetect it as well. I believe I had other choices showing up as well was the issue on my end (I had several other storage devices connected during some of the testing as I was trying different methods of cloning them and ended up landing on this one). It’s really quite smart. I’m honestly impressed by Petitboot and would definitely look on any ODROID board’s having that as having a definite advantage. I wish my XU4Q had a Petitboot loader for example.
Thank you for closing this out with all your final results / conclusions, take care!
Patrick
2 years ago
Not sure what happened, my server suddenly rebooted tonight and stopped working afterwards. I attempted to backup the world folders and deleted the minecraft folder in an attempt to start from scratch again after trying the restart script with no luck. When running the setup script now, I keep getting stuck at the openjdk version is less or greater than 16 message.
Checked and can see that openjdk 18 was released today and that is showing as the current version when I run java -version from the command line. Tried several methods to uninstall version 18 and reinstall the openjdk version 16 but all attempts have failed so far.
Open to any other suggestions I could try out here. Awesome blog James, even if I can’t get this to work, you’ve already provided hours of fun for myself and kids with our previous server world. 🙂
It sounds like you’re trying to reuse an older version of SetupMinecraft.sh. I would recommend deleting that version and running the latest version from GitHub as it doesn’t have that requirement anymore.
The latest version should update itself and I’m guessing yours is right before that. Can you give that a try?
I used the curl https://raw.githubusercontent.com/TheRemote/RaspberryPiMinecraft/master/SetupMinecraft.sh | bash command and ended up with the message:
Correct OpenJDK (16) was not found in apt repositories and needs to be installed via snapd.
Checking for snapd...
Installing OpenJDK....
"snap "core" already installed snap "openjdk" is already installed and snap "openjdk" has no updates available
That helps! I think I’ve found the problem. You’re definitely running it the right way. I’ve actually just updated and uploaded the fix on GitHub just now as there was still some lines in there that weren’t allowing OpenJDK 18 or higher.
This has been fixed. Can you give it another try here and see if that took care of this for you? To be clear go ahead and run SetupMinecraft.sh again now the same way you posted (it will use the updated file).
No problem at all! Thanks for stopping by to report it. I think this may be related to a couple of other people that I was helping investigate some issues and I’m sure there’s many more that didn’t post.
The reason this happened is that Paper didn’t used to support running OpenJDK versions above 17. I honestly thought I had removed this check completely already (maybe I did and there were still a few pieces left). Thanks again for reporting it!
Jeff
2 years ago
Hi James,
Thank you so much for the scripts, I ran a 1.17 server on my Pi for a while with no issues! I just upgraded to 1.18.2 and for some reason, the automatic startup is giving me issues. When the Pi reboots automatically or manually it seems like the start.sh begins (the backup successfully completes) however the server does not actually start. When I run start.sh manually, it begins no problem. Any idea what might be happening? Thanks for your help and useful page!
Can you try checking the logs in the logs folder and sudo systemctl status minecraft? It may be worth it to run SetupMinecraft.sh again one more time as this will recreate the service. We can definitely narrow it down from there!
Hi Jeff, there’s nothing in the logs folder unless I manually start the server, so I don’t think it’s relevant. Pasted below is the results of checking the status of minecraft service. Thanks again for your help!
● minecraft.service - Minecraft Server Service
Loaded: loaded (/etc/systemd/system/minecraft.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-03-17 01:35:46 CDT; 6s ago
Process: 903 ExecStart=/bin/bash /home/pi/minecraft/start.sh (code=exited, status=0/SUCCESS)
Process: 961 ExecStop=/bin/bash /home/pi/minecraft/stop.sh (code=exited, status=1/FAILURE)
Mar 17 01:35:43 raspberrypi bash[903]: ./world/region/r.5.2.mca
Mar 17 01:35:44 raspberrypi bash[903]: ./world/region/r.3.-5.mca
Mar 17 01:35:44 raspberrypi bash[903]: ./world/session.lock
Mar 17 01:35:44 raspberrypi bash[903]: Starting Minecraft server. To view window type screen -r minecraft.
Mar 17 01:35:44 raspberrypi bash[903]: To minimize the window and let the server run in the background, press Ctrl+A then Ctrl+D
Mar 17 01:35:44 raspberrypi systemd[1]: Started Minecraft Server Service.
Mar 17 01:35:46 raspberrypi bash[961]: Unable to set path variable. You likely need an updated version of SetupMinecraft.sh from GitHub!
Mar 17 01:35:46 raspberrypi bash[961]: Server is not currently running!
Mar 17 01:35:46 raspberrypi systemd[1]: minecraft.service: Control process exited, code=exited, status=1/FAILURE
Mar 17 01:35:46 raspberrypi systemd[1]: minecraft.service: Failed with result 'exit-code'.
Oh wow, that was not at all what I expected! No worries about there not being anything in the logs, you got exactly what I needed!
The first thing I want to check is that you should delete your copy of SetupMinecraft.sh and download the latest one from GitHub. I think you may be using an old enough version that it won’t autoupdate itself (the new version does this automatically now unless you enable the override switch at the top of the script).
If that isn’t it the other thing I’d like to see is the top of your start.sh script. This should have a PATH variable section where it sets the path variable to the ones that were active when you ran SetupMinecraft.sh.
Hopefully you just need the latest SetupMinecraft.sh from GitHub which will add that path variable in for you. If not then the top section of start.sh should reveal the issue. Hopefully that helps!
CardinalFang36
2 years ago
James.
Every time I try to “search” the extensive comments on this page, after entering 2-3 characters in the search box, I get a popup that wants me to sign in. Is this the expected behavior? I don’t see any way to create an account.
Ultimately, I was searching to see if you had any experience running “Multiverse” or similar mods on the RasPI. Do you know if they would overwhelm the server if I only have a couple of occasional users on a RPI 4 w/8GB?
Sorry about that! I found the problem and got that fixed up here. I actually had to pay something like a $99 lifetime license fee for that system and it looks like it was broken the whole time. It works great too if it didn’t require an admin password (doh). It doesn’t require the page to be refreshed or anything and will search as you are typing past something like a few characters (which is when it was triggering the password prompt). It turns out this plugin calls one file from the wp-admin folder which I have protected by multiple passwords. I just needed to write a security rule to allow the comment search through.
This has honestly probably been hurting my traffic as it is impossible to find some of those buried comments without it and that’s definitely what some people come here for. This was caused by my much higher than normal security I have on the site. It’s attacked at least a few times per day and sometimes more. It’s probably my cryptocurrency content that makes me a target as it didn’t used to be this way before that (now they know I have cryptocurrency and want to take it). They’ve never got me yet though (and I keep an eye on them to make sure it stays that way) but I never want it to interfere with normal site operation. Thanks for letting me know so I could get that fixed!
So you should be absolutely fine with these mods and some people have done them here before. I don’t think you’ll have much trouble. People have done Bungeecord and even some other ones (you can probably find these now that the search is fixed too for a *lot* more information). I have a Docker container that lets Bedrock users connect to Java servers using Geyser / Floodgate here.
It depends on how deep into the multiverse you want to go but this shouldn’t be a problem for the Pi to handle as it’s not particularly CPU / storage / memory intensive to link these together. Some multiverse systems may be more efficient than others but it should be the hardest on your networking so if you have a high quality / fast network I don’t see it being much of a bottleneck problem unless we’re talking about a whole lot of people / actions going on.
Definitely let me know if you try it how it works out!
Paul
2 years ago
Hi James,
I only stumbled across your blog while doing some extra searches on performance in relation to me trying to document my own Pi setup experimentation – so I won’t forget how to, if my nephew asks again :). My experimentation after setting him up with a server on laptop moved onto me using vannila Minecraft, Raspian aarch64 version, MS Java 17.0.2 on a RPi4 2GB @2146Mhz (actively cooling) and using UHS-I sandisk micro-sd.
From finding your blog I am very appreciative of the information in this – and your previous version of this article – about papermc and the pre-generating of the worlds info.
From your excellent info I downloaded the jar for papermc and chunky to pre-generate my map – allowing me to compare paper to vanilla minecraft performance – with my config using draw-distance of 16 and simulation of 10 – my start flags don’t use nogui and I still run the vncserver on the Pi4, but still find my linux OS is still only showing about 250MB-300MB for the OS when running “top” in a terminal and checking the buff/cache memory usage at first boot.
Sadly, I haven’t got a spare sd card or portable ssd, or the option to image my working Pi3 or Pi4’s cards to free one up currently, to try your blogs amazingly cool solution just yet, so I was wondering if you could give any indication of what type of server performance you are seeing on paper’s gui: avg tick time? or what ticks per second is paper on the Pi4 giving you – after an initial 10mins of uptime with one user logged in and when logged out after 10mins?
vannila Minecraft sadly doesn’t track the 1min, 5min, 20min TPS info AFAIK, and doesn’t have the amazing “timings report” detail that I’ve only just discovered indirectly with paper from your blog, but the “/perf start” on my vannila config – after pre-generation and user logged in – now reports ~20.04 TPS and in normal activities even with 10 TNT explosions – but not flying, not teleporting, not using fill. And now my typical avg tick is around the 10-20ms, occasionally stressing at around the 35ms mark, and after logging out returns to 1-2ms, which is what it eventually reaches after the initial 5-10min server start up load – until “htop –tree” in a terminal is no longer showing 4 core utilisation, or my CPU-voltage-throttling script shows temperatures have dropped back to mid-40degs from mid-70degs. Which is a bit of a quandary because the idle and load tick ms numbers I get running paper – even with some small tweaks to paper.yml and bukkit.yml to lower tick frequency of items and using further tweaks to GC flags still seems to indicate vannila performs marginally better than paper – for me -with its lowest idle tick in ms of about 4.5ms (from initial 10-15ms after different GC start flags) – which I assume is wrong, and I’m leaving performance untapped in paper somehow, either not allocating enough memory with the xms/xmx flags using 1450M, or too big a swap file at 512MB or something.
From anecdotal observation, using chunky on paper and moving the map back to vannila seemed to eliminate performance inconsistency on my vannila setup. Watching “htop –tree” early on with default settings for paper’s startup, it seemed to stress the CPUs cores all the time, but at half load and somehow seemed to load the server and world faster than vannila.
Using my CPU monitoring script through startup it also seemed to keep the temperatures lower, which is interesting because on my Pi3, that isn’t actively cooled, vannila MC server isn’t reliable when I tried, because the initial startup load would frequently cause thermal throttling – dropping the core frequencies down to 800Mhz – when allocating 750MB to the server, and would then cause the server to exceed the “max-tick-time=60000” and shutdown automatically. So on passively cooled Pis paper seems more robust and maybe smoother at handling lag all round on any setup compared to vannila, so naturally I want to beat vannila on my RPI4 setup to justify the switch to paper. Any ideas of things to try would be most welcome.
Thanks for stopping by, I’m glad you found the site! This sounds like a great build and I’ll definitely fill in everything I can between my experience with it and other people’s reports posted here.
That is really interesting that you pregenerated the chunks in Paper and then moved it to vanilla! I’ve never heard of anyone doing this before but it makes perfect sense. The chunk generation hits the Pi in 3 places where it hurts: the memory, the storage, and the CPU. There isn’t any reason this shouldn’t work and it’s a great idea!
This script has been around for a long time now. It looks like the first build I released it on was 1.12 back in 2017. Back then most people were using Pis with 512MB to 1GB. It was pretty much required to get full TPS to pregenerate your chunks back then.
There have been several developments that have helped though. The biggest thing with the chunks is upgrading from a SD card to something like a SSD. This will make the largest difference of any upgrade you can make. The 2GB of RAM should be enough for a few players. It has been working with a few players on at full speed for quite a few years! The quality of your SD card can impact this quite a bit too (something like a SanDisk Extreme vs a generic SD card can be quite a bit, you can test them at pibenchmarks.com to see how fast they are exactly). It has been able to achieve a solid 20 TPS with a few players on for many years now (with fast storage).
It sounds like you have good CPU monitoring set up. Have you done any IO monitoring by chance? My guess is that this is where any slowness is. You could get a program like iotop or your OS may have iostat built in. You can do a:
iostat -dxm
Linux 5.16.0-kali1-amd64 (kali) 03/06/2022 _x86_64_ (8 CPU)
This is on my PC and you can see my drive utilization is extremely low (the last column). This is a very useful tool to run if you are having lag / TPS drops as if you find the utilization is at 100% then it is likely that the Minecraft server is actually just sitting there waiting for the storage to finish writing the new blocks. I do have a bunch of articles on here for setting up SSDs in case you are finding you want higher IO performance. I know you mentioned that you didn’t have one yet but I think you are on the right track here. Running some iostat measurements to get some measurements similar to your CPU and memory measurements should fill in the rest of the picture here.
For the 4 core utilization issue I definitely know what you’re talking about and have seen that. When you initially load the server it will be basically pegged for several minutes (it will literally say like 400% CPU usage in regular top, not sure about htop). It typically settles down after that and the number that pops in my head is it would still usually be over 100% in old top for the utilization but nowhere near 400% after the initial startup. It’s a pretty dramatic drop when it does and I’m not surprised you’re seeing thermal performance. The best I can tell this is mostly IO activity. The server is loading in a bunch of chunks and data (even when nobody is connected for the first few minutes). It has done this as long as I can remember and it actually used to be much worse. Back in those days with 512MB – 1GB at the most Pis it wasn’t too unusual to have to wait for this to stop before performance would stabilize (monsters would be warping around, blocks would have a break delay, etc then once it finished whatever it does during this time it would be perfect).
The vanilla server having higher performance in any way than Paper is almost shocking. The reason I say that is that Paper turns a *lot* of stuff off that ticks in normal vanilla Minecraft. All of the options (including a bunch that you mentioned) are basically additions to let you control those variables because it is the main source of “lag” or TPS drop in vanilla. I can’t think of many reasons this should be ever be the case but one of them would be plugins. This is just a hunch, but you may want to try disabling all the Paper plugins (including any chunk generation ones, just for a test) and see if that behavior goes away.
If it doesn’t then it would almost suggest to me that this is some kind of IO / storage bottleneck. I know Paper batches some things together (meaning it tries to update a bunch of things at once instead of sporadically here and there like vanilla) and theoretically if the storage is bottlenecking that could be resulting in some TPS drops that are more severe than vanilla. This definitely shouldn’t happen though as Paper doesn’t really “add” any weight other than the plugin system. It basically only tries to fix vanilla sources of lag if that makes sense.
The RAM sounds tight but that is to be expected on 2GB. It should be okay. It would be tough to run a version like 1.18 on 512MB anymore but even 1GB should work (but it will be very tight with the OS). Have you experienced any memory related crashes or had the Minecraft server terminated? This is typically what will happen when you allocate too much memory to the Minecraft process. Counter intuitively it should often be lowered. I didn’t see you mention any crashes so it sounds like you have the memory tuned okay. It may be worth doing an experiment and turning it down slightly more and see how that impacts your performance numbers (if things are really tight on the OS it may actually help).
My gut feeling (if possible) is that if you could pick up a cheap <$20 SSD and maybe the StarTech USB to SATA adapter (or even better if you already have a spare SSD you would just need the adapter) this could be a <$30-$35 upgrade even buying both brand new which puts it in the appropriate price range for the Pi (and even starting to approach the price of some higher capacity premium performance SD cards but for an order of magnitude of better performance). Fortunately though if you use something like iostat or iotop I think we can get your complete performance picture here even though vanilla doesn't have as fancy of a timings tool as Aikar's!
It seems like there's a bottleneck here and given your thorough measurement of CPU and memory I think it will show up on the io tools. I think it should be performing a little bit better than it is from your description (but it sounds like it's really getting close) and I suspect this is your impression as well. The fact that vanilla is performing at least the same or maybe even slightly better tells us that there has to be some kind of bottleneck outside of your Paper or vanilla configuration. I think we can definitely find it and get the server performance to perfect!
I made a site called Pi Benchmarks that you may find interesting. It’s benchmarks from all over the world specifically on Raspberry Pis. People test SSDs, eMMC, SD cards, anything you can connect to a Pi basically. This breaks down the stats and generates a “score” giving you a quick comparison on performance. An average SD card’s IO performance would score something like 700-1200 and an average SSD is more like 5000-8000 and even higher for NVMe drives. I made this originally specifically for this Minecraft script project as IO performance has been a huge issue since day 1. Like I was saying earlier the difference between a generic / cheap SD and a high performance A1 (application class) can literally be twice as fast in some cases. IO is always the most commonly missed / most poorly understood bottleneck (partially because you need different tools to measure it effectively and they vary more across distros than top/htop/etc.) and I made that site because it was even worse on the Pi and we used to have literally 0 way to know or compare how fast any of them were.
I’d love to hear what you find and can definitely help further if you wanted to bounce any ideas off me to see if I’ve tried them / others have tried them!
Thanks for the great reply. I probably should have elaborated more than “sandisk uhs-I” and fully said it was a SanDisk Extreme v30 U3 A1 32GB card in my Pi4(and 3) in my first post – which I think is fine, going by the raspian benchmark (rpdiag.txt) file I generated today, gave:
Test : SD Card Speed Test
Run 1
prepare-file;0;0;38258;74
seq-write;0;0;37816;73
rand-4k-write;0;0;2951;737
rand-4k-read;11049;2762;0;0
Sequential write speed 37816 KB/sec (target 10000) - PASS
Random write speed 737 IOPS (target 500) - PASS
Random read speed 2762 IOPS (target 1500) - PASS
Test PASS
I took your good advice and installed iostat on my Pi4 and ran: “iostat -d -h 2” throughout the startup and stabilization time of both paper and vanilla on my server setup and after graphing one of them in scalc, the graph looked similar to the server monitoring graph of memory use, massive activity through setup and then stable with virtual nothing other than small megabyte or kilobyte blips. So it doesn’t appear that the I/O is having either config waiting.
I tried some different things like reducing the number of threads for netty-io down to 2 then 1, but saw no real difference to the idle avg tick time in ms – going no lower than 4ms – and watching htop it looks like paper never gets as much utilisation of all cores as vanilla – whether startup or when handling load it never sustains 4 full cores for more than second for me – but never fully settles either, unlike vanilla that will drop to almost nothing – on all 4 cores – when the server is fully stable after 10mins – under no load – when it has a constant tick time between 1.6ms and 2ms and has dropped to a low 40 deg temp, whereas when running paper, the Pi4 never really drops fully back below 45deg and stays there.
My gut feeling is that the vanilla performance number advantage at idle on my setup is a placebo effect that wouldn’t translate into a better experience than paper in actual use. Sadly, AFAIK there isn’t a free way to simulate a 2 to 4 player additional load on a personal server – without real players, which seems like a massive oversight for Majong and now Microsoft for such an important aspect of a game with self serving.
From the adjustments I tried I think the numbers now show paper to be faster, as it seems like the change in avg tick time with an active player versus idle is less (only 12ms per player if it scales linearly) with paper, where as it seems more like 15-20ms per player with vannila in my tests..
I’m now intrigued enough to move some hardware around to try out your setup script on a Micro-SD or a usb ssd with my Pi4, as I’m sure if you’ve had older server versions running on the 512MB, there’s more performance I missing.
If there is any aspect of my vanilla setup with an advantage to paper, I think it could be because I’m using a Java SDK (installed through sdkman) that Microsoft probably use and profile against vannila, but the paper team probably use the oracle openjdk which is maybe a better fit for it. I selected the Microsoft Java 17.0.2 release because it seemed to be smaller in size than openjdk and oracle, etcs efforts and seemed to give better performance numbers for me. Alternatively if the higher CPU utilisation I’m seeing is a cause, I wonder if there is heavy sorting taking place in the minecraft servers, as a sort algorithm with data supplied in a way that provided superior balancing might give better multithreading performance, and as paper’s adaption of vannila could upset such an algorithm I wonder if that is the performance in idle I can’t seem to get back.
To answer your other question, I did previously have situations where I was running out of memory and crashing the server program because I was experimenting with a ramfs – that needs everything manually accounted for. I was using it for combinations of the location for the world data, the java sdk and the jar file and the path using flag“-DbundlerRepoDir=” for the extracted MainClass, with a 4 or 8GB Pi I think all in ramfs would be really good. But I eventually lowered the vm.swappiness value in the “/etc/sysctl.conf” file, and now only keep the jar and extracted class in a ramfs – to eliminate storage IO as a bottleneck – it seems like linux gives back more of the buffer/cache memory (100MB) by swapping itself into the file with a very low swappiness value, rather than Minecraft’s time sensitive data, and seemingly stopped lag-crashing the server program after idles, where I assume it moved some of the server program data out to swap and then couldn’t access it all quick enough after idle to avoid having a tick longer than maximum allowed.
Thanks again, and will hopefully report back when I get a chance to try your setup script.
I think I can explain some of this behavior. Basically the Minecraft server is single “cored”. What people mean when they say this is that the bulk of the Minecraft server data processing is done in the same thread. It still uses a few threads (sockets use some threads for example) but the main thread that is “ticking” (thus TPS / ticks per second) is happening typically on one CPU core. If one thing is slowing it down everything has to wait basically. This means CPUs that have high single-core performance (vs the multiple performance benchmarks, you’ll usually see one of each for most CPUs) perform better with Minecraft and other similarly structured application.
This may look like it’s performing better and with lower CPU usage but it partially just means vanilla isn’t utilizing your other cores at all (and can’t). Paper can’t really do it that well either but it does spread things out a little bit over a few threads which is just about right for the Pis CPU (3-4 cores). I believe if you have multiple people online with vanilla you will “feel” that everything is even more heavily loaded in a single core and that people’s experience will be worse. You kind of alluded to this in your post that you suspected this may be the case and I think you are correct / thinking about it the right way. This doesn’t explain all of it though and there is in fact more going on here (it’s related to some of the optimizations Paper does).
Another important difference is that Paper actually grows plants and some other basic things when people are offline. Vanilla doesn’t do this generally and pretty much shuts everything off if nobody is online. You can measure this using Paper’s timings tool you mentioned earlier. Everything it does is documented in there including when nobody is online. Every procedure call and everything it’s doing as well as how much time it takes. I would imagine this would answer this question as to what Paper is doing. I covered how to do this in a really old post here.
It’s pretty much as simple as typing timings on in the server console and then waiting a period of time and typing timings report. This will give you a URL to paste into your browser and it will appear as an interactive report like my screenshots here.
Check out the breakdown. This breaks down everything that happened during a server “tick”. You can sort the data a few different ways but it knows how many times each function was called, the name of the function and a sort of “stack trace” that makes it easy to see what part of the code it is and what it’s generally doing. I would definitely run this tool on there to determine your Paper tick usage and you can figure out what is ticking. The % of your overall tick is really useful. If you see things like entities using a huge % of your tick rate then if you reduce those entities in the .yml config files and run the test again you’ll see it’s using much less of a % of your server tick.
Most of this stuff can be disabled. If you wanted to match Paper to vanilla’s settings you can pretty much disable all of it. If you find that it’s the plants growing / that type of tile ticking going on you could disable that in the various .yml config files such as paper.yml, spigot.yml etc. and all of these settings can be viewed here.
Specifically for yours I bet your behavior is related to Paper Minecraft not unloading chunks like the starting area. It will keep those chunks loaded when nobody is online so that when people join there isn’t a delay as it normally has to load and fetch those chunks from the server’s storage (cold storage retrieval). Paper’s default settings keep it loaded since the spawn region is typically the most high traffic area of a lot of servers. You could probably greatly reduce the CPU usage with settings like keep-spawn-loaded to false (this is for sure some of the behavior, your spawn isn’t unloaded on Paper without changing this setting and vanilla does nothing like this). Maybe set keep-spawn-loaded-range to 0 for good measure.
That should explain most of this as those chunks are still live on Paper when no one is online. Even the monsters aren’t unloaded when this setting is active (they do stop moving at least if nobody is close by but many types won’t despawn). The spawn region is basically protected from being unloaded even if nobody is playing whereas vanilla destroys everything.
There may be more of these as well you can disable. You can basically start the timings tool on Paper and have it record everything that is happening. You can just walk away and have a coffee or do something else and let it slowly benchmark and count everything it’s doing. When you stop the tool and generate the report you can pretty much Ctrl+F the Paper.yml page I linked earlier to find settings related to it. You should be able to shut this off to get more comparable behavior for the tests. This may help explain some of these performance differences (beyond just the keep-spawn-loaded setting) for sure!
Unfortunately as we’ve lamented before vanilla doesn’t have the timings tool but we at least do have 50% of the information and I believe you’ll be able to probably get things pretty close if not better with some additional configuration in the yml files. Fortunately in this case Paper is also the one we need this data for as we already know what vanilla is doing when everyone is offline (nothing). We need to know what Paper is doing to explain this and by using the timings tool like I described in the post linked above (it’s old, for 1.13 but it hasn’t changed at all to use the tool) you will end up with a complete stack trace of every single instruction the server executed while running the tool. It will answer these lingering questions that are not really showing up well in the standard toolset (not uncommon with something as complex as the Minecraft server is, it just means we need better / more specific tools to get the data we need to explain the discrepancies).
Your memory configuration sounds excellent. You already went through and tuned this appropriately. That SD card is excellent as well, I have a couple of those SD card and they score very well. Everything is looking good. I’m curious if you can narrow down some flags using the Timings tool that can explain the rest (if any) discrepancies between the two. With any luck the keep-spawn-loaded behavior will explain 90%+ of this and simply disabling it will drop the temps back down to match vanilla. Hopefully that helps!
That’s the info I was missing from my newbie knowledge of actually playing MC, so not knowing the main virtues of papermc, like its world persistence for ticking crops etc – beyond plug-ins, and performance that I was struggling to prove to myself, numerically – and me just being the IT person getting a laptop server functional for my nephew, with no background in MC serving specifics or awareness of bukkit/spigot/paper then, and then hobby-ing with a RPi4 replacement as a potential solution, or future server for my kids.
Changing the “keep-spawn-loaded” to “false” as you suggested improved the synthetic(& real) performance numbers massively. Producing a significantly lower avg time in ms per tick, and stabilized as low as 0.2ms, 5-10x less time than 1-2ms of my attempts to optimise Vanilla – even with Vanilla not ticking crops.
The server also has a lower avg tick time most of the time, but the range being much bigger tells me I need to either revert the setting and take the inferior but stable numbers, or look again at maybe reducing memory use in the start flags – as you’ve suggested the server can work well with the pi3’s lesser gigabyte too – and look again at the ramfs, for the java sdk, world files, maybe even revert to a tiny 100MB swapfile too, and then alter some other settings in my yml files to get the better performance numbers along with more stable numbers.
Changing the “chunk-tasks-per-tick” down from 1000 to 500 helped a bit, but one of the problems seems to be the Salmon tick-time for me – using more resources than zombies on lag ticks – but being unable to see a setting that matches in my yml files, and my lack of papermc knowledge means I’m not sure how to handle that best, or the occasional Server.Oversleep adding 40ms when “keep-spawn-loaded“ is set to “false”.
I suspect on balance “keep-spawn-loaded” set to “true” is going to be the better option, but I am still very appreciative that you were able to point me in the right direction to validate that paper is vastly more efficient than Vanilla on my setup. With “keep-spawn-loaded“ set to “false” for the time being on my setup running “/perf start” on a papermc server (not needed obviously with timgsV2 being the definitive tool) now works without throwing errors.
I’ve also freed up a card with a new Bulleyes install, so my next task is to try your script out which I assume has custom yml settings you feel work best, yes?
No problem at all, all of those numbers sound like everything is really coming together!
I’ve also ran into the entities tick issue. It’s usually passive animals that are the most offensive. You have a few different settings that can improve / reduce the amount of TPS being burned on this. One set of settings is the despawn range for these creators. If you lower the range it will despawn them more aggressively (without the player wandering as far away):
despawn-ranges
monster:
soft
default: 32
description: The number of blocks away from a player in which monsters will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which monsters will be forcibly despawned.
creature:
soft
default: 32
description: The number of blocks away from a player in which creatures will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which creatures will be forcibly despawned.
ambient:
soft
default: 32
description: The number of blocks away from a player in which ambient entities will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which ambient entities will be forcibly despawned.
underground_water_creature:
soft
default: 32
description: The number of blocks away from a player in which underground water creatures will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which underground water creatures will be forcibly despawned.
water_creature:
soft
default: 32
description: The number of blocks away from a player in which water creatures will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which water creatures will be forcibly despawned.
water_ambient:
soft
default: 32
description: The number of blocks away from a player in which water ambient entities will be randomly selected to be despawned.
hard
default: 64
description: The number of blocks away from a player in which water ambient entities will be forcibly despawned.
misc:
soft
default: 32
description: The number of blocks away from a player in which misc entities will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which misc entities will be forcibly despawned.
Basically you can see here they have different categories. The Salmon should hopefully be in the underground waters section, but be advised that sometimes the monsters are classified in really odd ways. If you encounter this you may need to look up the entity ID by doing some Google queries (usually will take you to the minecraft wiki or something like that) and can figure out which category some of the stranger ones are in.
It had been years since I looked at them but I double checked which flags my script applied automatically. They are related to this conversation and may be of some assistance while you’re tuning / testing / planning:
# Configure paper.yml options
if [ -f "paper.yml" ]; then
# early-warning-delay, early-warning-every
# Disables constant error spam of chunk unloading warnings
sed -i "s/early-warning-delay: 10000/early-warning-delay: 120000/g" paper.yml
sed -i "s/early-warning-every: 5000/early-warning-every: 60000/g" paper.yml
# optimize-explosions
# Paper applies a custom and far more efficient algorithm for explosions. It has no impact on gameplay.
sed -i "s/optimize-explosions: false/optimize-explosions: true/g" paper.yml
# mob-spawner-tick-rate
# This is the delay (in ticks) before an activated spawner attempts to spawn mobs. Doubling the rate to 2 should have no impact on spawn rates.
# Only go higher if you have severe load from ticking spawners. Keep below 10.
sed -i "s/mob-spawner-tick-rate: 1/mob-spawner-tick-rate: 3/g" paper.yml
# container-update-tick-rate
# This changes how often your containers/inventories are refreshed while open. Do not go higher than 3.
sed -i "s/container-update-tick-rate: 1/container-update-tick-rate: 2/g" paper.yml
# max-entity-collisions
# Crammed entities (grinders, farms, etc.) will collide less and consume less TPS in the process.
sed -i "s/max-entity-collisions: 8/max-entity-collisions: 2/g" paper.yml
# fire-physics-event-for-redstone
# This stops active redstone from firing BlockPhysicsEvent and can salvage some TPS from a cosmetic task.
# Note: If you have a rare plugin that listens to BlockPhysicsEvent, leave this on.
sed -i "s/fire-physics-event-for-redstone: true/fire-physics-event-for-redstone: false/g" paper.yml
# use-faster-eigencraft-redstone
# This setting eliminates redundant redstone updates by as much as 95% without breaking vanilla mechanics/devices (pretty sure). Empirical testing shows a speedup by as much as 10x!
sed -i "s/use-faster-eigencraft-redstone: false/use-faster-eigencraft-redstone: true/g" paper.yml
# grass-spread-tick
# The time (in ticks) before the server attempts to spread grass in loaded chunks. This will have minimal gameplay impact on most game types.
sed -i "s/grass-spread-tick-rate: 1/grass-spread-tick-rate: 3/g" paper.yml
# despawn-ranges
# Soft = The distance (in blocks) from a player where mobs will be periodically removed.
# Hard = Distance where mobs will be removed instantly.
sed -i "s/soft: 32/soft: 28/g" paper.yml
sed -i "s/hard: 128/hard: 96/g" paper.yml
# hopper.disable-move-event
# This will significantly reduce hopper lag by preventing InventoryMoveItemEvent being called for EVERY slot in a container.
# Warning: If you have a plugin that listens to InventoryMoveItemEvent, do not set true.
sed -i "s/hopper.disable-move-event: false/hopper.disable-move-event: true/g" paper.yml
# non-player-arrow-despawn-rate, creative-arrow-despawn-rate
# Similar to arrow-despawn-rate in Spigot, but targets skeleton arrows. Since players cannot retrieve mob-fired arrows, this setting is only a cosmetic change.
sed -i "s/creative-arrow-despawn-rate: -1/creative-arrow-despawn-rate: 60/g" paper.yml
sed -i "s/non-player-arrow-despawn-rate: -1/non-player-arrow-despawn-rate: 60/g" paper.yml
# prevent-moving-into-unloaded-chunks
# Prevents players from entering an unloaded chunk (due to lag), which causes more TPS loss. The true setting will rubberband them back to a "safe" area.
# Note: If you did not pregenerate your world (what's wrong with you?!), this setting might be a godsend.
sed -i "s/prevent-moving-into-unloaded-chunks: false/prevent-moving-into-unloaded-chunks: true/g" paper.yml
# disable-chest-cat-detection
# By default, chests scan for a cat/ocelot on top of it when opened. While this eliminates a vanilla mechanic (cats block chest opening), do you really need this silly mechanic?
sed -i "s/disable-chest-cat-detection: false/disable-chest-cat-detection: true/g" paper.yml
# bungee-online-mode
# disable Bungee online mode
sed -i "s/bungee-online-mode: true/bungee-online-mode: false/g" paper.yml
# keep-spawn-loaded, keep-spawn-loaded-range
# This causes the nether and the end to be ticked and save so we are going to disable it
# This setting makes sense on high player count servers but for the Pi it just wastes resources
sed -i "s/keep-spawn-loaded: true/keep-spawn-loaded: false/g" paper.yml
sed -i "s/keep-spawn-loaded-range: 10/keep-spawn-loaded-range: -1/g" paper.yml
fi
# Configure bukkit.yml options
if [ -f "bukkit.yml" ]; then
# monster-spawns
# This dictates how often (in ticks) the server will attempt to spawn a monster in a legal location. Doubling the time between attempts helps performance without hurting spawn rates.
sed -i "s/monster-spawns: 1/monster-spawns: 2/g" bukkit.yml
# autosave
# This enables Bukkit's world saving function and how often it runs (in ticks). It should be 6000 (5 minutes) by default.
# This is causing 10 second lag spikes in 1.14 so we are going to increase it to 18000 (15 minutes).
sed -i "s/autosave: 6000/autosave: 18000/g" bukkit.yml
# warn-on-overload
# Disables annoying server is overloaded messages
sed -i "s/warn-on-overload: true/warn-on-overload: false/g" bukkit.yml
fi
# Configure spigot.yml options
if [ -f "spigot.yml" ]; then
# Merging items has a huge impact on tick consumption for ground items. Higher values allow more items to be swept into piles and allow you to avoid plugins like ClearLag.
# Note: Merging items will lead to the occasional illusion of items disappearing as they merge together a few blocks away. A minor annoyance.
sed -i "s/exp: 3.0/exp: 6.0/g" spigot.yml
sed -i "s/item: 2.5/item: 4.0/g" spigot.yml
# max-entity-collisions
# Crammed entities (grinders, farms, etc.) will collide less and consume less TPS in the process.
sed -i "s/max-entity-collisions: 8/max-entity-collisions: 2/g" spigot.yml
# mob-spawn-range
# Crammed entities (grinders, farms, etc.) will collide less and consume less TPS in the process.
sed -i "s/mob-spawn-range: 8/mob-spawn-range: 6/g" spigot.yml
# entity-activation-range:
sed -i -z "s/entity-activation-range:\n animals: 32\n monsters: 32\n raiders: 48\n misc: 16\n tick-inactive-villagers: true/entity-activation-range:\n animals: 24\n monsters: 24\n raiders: 48\n misc: 12\n tick-inactive-villagers: false/g" spigot.yml
fi
You can see a lot of entities related settings in here. Most of them are nibbling around the edges of things that “tug” at your TPS. Notice that I slightly tweak a bunch of settings related to them but mostly not in a very dramatic way. It turns out my script does disable the spawn loading as we can see here (I even made a note I forgot about, this setting also causes The Nether and The End to also tick all of the time when they normally don’t without this setting unless someone is in there). Undoubtedly you are going on a very similar journey that I did years ago to discover which flags give you the best performance! These are all settings that caused me (or the people using the script) a problem that I had to usually locate through the timings and find the setting to alleviate the issue.
You may be able to cheat and use mine but they’ve added some new configuration flags over the past few years that mine likely isn’t using. There’s probably some additional performance that can be gained here and I definitely don’t insist it’s the perfectly correct set of flags. These were chosen as an inoffensive and pretty close to vanilla experience while greatly improving performance on the Pi. Some of them are related to freeing up the storage for example which is already slow on the Pi. I change the autosave interval from something like 5 minutes to 15 minutes.
The scripts are meant to be easily modified. You could actually just put your changes right in start.sh if you follow my lead and replace them right in the .yml files. You could also just edit the files as it’s mostly to automate it that I did that but it is nice to have a comments area for each configuration change so you can document what you did (otherwise the next time you spin up a server you’ll likely forget a few of these settings). That’s one way to stay organized at least! I literally wouldn’t have remembered a single flag or why I did it if I didn’t document and comment it for myself (you think you will but outside of a few memorable ones like the keep-spawn-loaded one that I remembered off the top of my head the minor ones I wouldn’t have been able to to save my life).
I’m curious what your experience will be when you test the script! As I was saying earlier you may be able to find a few surprises in there that will yield additional gains. This is especially likely for flags that have only been added in the past couple of years. If you do let me know and I may modify the default flags if it’s clearly better performance without any drastic quality of life reduction in game.
I typically play “vanilla survival” mode with my friends. Obviously it’s not 100% pure vanilla but the settings are meant to be as close as possible. View distance 10 is maintained for example (a common setting to lower that drastically reduces quality of life / game experience) and the entities while having a few optimizations around the edges aren’t reduced by 90% or anything drastic that some people have attempted before. The settings are inspired by this goal though and therefore may not yield maximum performance in some cases where I made the tradeoff to keep the experience as close to vanilla as possible. Different people may have different goals though and want that extra performance depending on what their server needs.
You’re getting to the “tradeoff” point but there still are probably a few “low-hanging fruit” in the settings file before you start to have to chop away at the qualify of life on the game. As you pointed out it is a tradeoff to disable keeping the spawn loaded. Especially when your server is starting out people are going to feel it if they are the first one logging on and they’re the one that has to wait for the spawn to load. For the Pi because of the CPU usage (especially years ago when there was a lot less CPU to go around) that tradeoff didn’t make sense to me as the default setting but it undoubtedly will be better for some servers (no question).
Hopefully those help as well and definitely let us know how it turns out / what flags (if any) improve the experience further!
I got your script installed on my other card using my RPI4 2GB using the raspian bulleyes aarch64 os and can see why you’ve had lots of people using your setup successfully over the years. It seems to work really well by default with very little user input, other than needing to pick wisely with the memory allocation.
Even without the other optimisations I use, like the swappiness change and use of the ramfs for staging files/symlinks, etc. It is a day and night comparison to just running Vannila as is on an overclocked RPi4 from the performance I was noting using my testworld from my other card seup.
After lots of timings with different OS tweaks from my other setup and trying with and without a GUI – using your scripts “switch” console setup – and various java garbage collection flags, with different memory -Xmx allocations, numerically the idling wasn’t as low as my other setup and edge cases where lag really hits, it seemed higher – but by default I think your script still uses “keep-spawn-loaded: true” and the range for the loaded area set to 10 chunks IIRC, which made sense, so I was largely seeing better normal server performance with your paper.yml settings, but better idling and peak lag handling with my paper.yml. Using screen gave back more memory, but didn’t improve the performance on TimingsV2, versus using the Raspian desktop remotely with papermc’s GUI through realvncserver, so I can only assume the ssh and screen programs consume more CPU time on the main core – by redirecting – than GPU accelerated GUI with the java awt gui for the paper server do.
Eventually I downloaded diffuse(a Gui diff) and sorting the two files so your flags order matched the ordering of my recently generated paper.yml – and merged in any missing flags from each other that are still used, and use a copy of the variables with the default value behind a # # in the paper.yml file, for any variable changed from normal – except for the long list of soft:hard:, where I just note it at the top of the section once, as their defaults are all the same.
Some of the entries in the file are left as is, but have a copy with “# #” at the start,because these were entries I was changing at some point, but gave inconclusive results.
This is a link to my last timings run without the initial server startup in the timings. The server.properties file uses a “view-distance=16”,and “network-compression-threshold=512” too.
I this is as good as I’ll get other than me getting a RPI400 or 8GB Pi4 with more ramfs options for the world data, and/or maybe using an SSD to gain a little extra performance from the OS itself, hobbling the game features too much.
And this is the java flags I use, I think 768k being better IMHO than 1MB jvm would override a bigger value with – and better than 512K I tried – because half of the 768K is a better alignment to the CPU caches on the ARM chip IIRC and seems to yield better performance. I also allocate a large 2048MB swapfile, so the OS doesn’t react to the server consuming all the memory at once and can hopefully not crash the RPI4 if the paper server crashes. I also change the vm.swappiness value down to 5, from the default of 60 in the OS
980M seems to be the maximum I could allocated with a large 200MB world, without more than 30MB of swap being used – working on the basis that storage on a RPi4 has a read limit probably around 30MB/s, so 40ticks stall at most for any minor swapping.
Everything is looking good! My guess from your timings is you would get a pretty decent boost from a SSD. Your highest scores when you choose the “Lag” option (vs the All option) is all chunk saving / storage. There isn’t a ton you can do with paper.yml flags to alleviate this except for save and tick the chunks less but in reality you are hitting both some SD card limitations and some Pi 3 limitations.
I have a few suggestions for you since you are on a Pi 3. You are able to overclock the SD card port from 50MHz to 100Mhz by adding:
dtparam=sd_overclock=100
to your config.txt. You may want to try this and repeat my storage benchmark as it does increase the throughput (it won’t be a 100% increase but it’s noticeable, maybe more like 10-15%). My storage benchmark checks for this setting and measures it.
You can still have some substantial storage/IO gains on a Pi 3. Check out these scores people submitted for Pi 3 here: Pi3
Your upper end is probably 2500-3000 as far as a raw performance score on the Pi 3 with a SSD but that’s still easily 2-3x faster IO performance. This is especially true on random read/writes (as opposed to sequential read/writes) which is what the Minecraft server uses. The highest for the SD card you have is something like 1000-1200 I believe. There’s actually a Pi 3 score on there of exactly 1000 for that card. Notice that even $20 SSDs (literally, well $23 at the moment) like the Kingston A400 are doing 2500. You can see the exact IO stats by clicking into those benchmarks (MB/s, IOPS, all of it).
Bumping up the core speed and CPU speed wouldn’t hurt on the Pi 3 either. Some of those benchmarks submitted will have overclocks. You don’t want to overdo it but you could do what we call a “safe” or “conservative” overclock on it in config.txt without any risk. You have to set a special setting (force_turbo, voids the warranty) to be able to do too much damage to it.
The SD overclock and the core speed are the most likely to reduce those chunk save times a little bit followed by the CPU at a close second. Check out the RetroPie settings examples for some safe ones I’ve used before here (Ctrl+F then “Settings Examples”). This includes a safe RAM overclock (sdram_schmoo=0x02000020) as well which is huge for the Pi 3.
I wish I had a Pi 3 setup (I’d have to dig it out, I have it somewhere but it would be fresh imaging and the works) that I could show you how big of a difference it makes to have a SSD in there on those ticks. You’ve pretty much recreated my entire journey of creating this script except the SSD part. That was when I finally reached the performance level I wanted (view distance 10, as close to vanilla settings as possible for multiplayer survival). I’ve hosted it on application class SD cards with a few friends before successfully but you can definitely feel the difference. The players online can feel the difference between a SD card and a SSD (during world saves you won’t go over 100% on your tick like that with a SSD). It’s that noticeable for sure!
Because of the single “core” nature of the Minecraft server we talked about it really can’t do anything else when it’s waiting for the storage to read / write these chunks to / from disk via random IO operations. Most of this time you are losing it’s just waiting for the storage device. It’s not because the Pi is too busy and can’t keep up (that looks very different in the timings, yours look fantastic other than the expensive saving / reading IO operations involving chunks). If the Minecraft server was designed to split things up into separate threads / processes like many modern servers are this wouldn’t be the case and it could be doing other things at the same time. Unfortunately with how the Java version was built and designed basically it can’t do anything else. Everything stops until that chunk is saved / read so people will notice blocks won’t break for a few seconds or however long it takes etc.
I think I agree with your assessment that it’s as good as it’s getting without a SSD with the exceptions that I mentioned here of you should give overclocking the SD card and probably even the core/CPU a try. The reason this can make a difference is you’re a generation back. It closes the gap in performance anywhere from a little bit to a moderate amount and honestly yours is so close that a little bit is all you need. Your TPS is completely under control other than the chunk saving which according to the “Lag” chart did spike over 100% during some of those saves (people will feel this, that’s a lag spike).
The RAM in the Pi 4 is much faster so you can’t catch up there but on core and even CPU it’s not that far behind for sure. This is more “nibbling around the edges” and all of that nibbling combined resulted in a smooth playable experience even on the Pi 3! I won’t say that the Pi 3 is indistinguishable from a PC hosting it (yours is going to be close I can already tell) but I would say that the Pi 4 on a SSD is.
Hopefully that helps and let us know how it turns out!
Oh. sorry I think my aggressive reduction in Xmx value – to below 1GB and mentioning me taking the card from my RPi3, has made you think it wasn’t the 2GB RPi4 I ‘ve been using when it was.
Between 980MB for the server, another 200MB-ish in the ramfs, 250MB-ish overhead for Java’s JVM, and another 300-350MB for the linux OS buffer/cache and needing to leave a small margin free of another 100MB or so for the buffer/cache to extend you actually get back to the 1900MB available.
As for the lag numbers, I was under the impression from a video I watch on timingsV2 that it is near impossible to eliminate the lag ticks entirely, but the importance was to keep the avg TPS at 20 and have the lag frames with a combination of minimal over-spend – by time taking more than 50ms – and to have a low percentage of them – and spaced out – over the range.
I was surprised you hadn’t interpreted the timings the way I had – as being improved for a RPi4 – and so I ran your excellent script freshly, again with the same world data and workload as I’d done for my own setup and did seem to get better performance, than my previous runs, and then from that, I reconfigured my own setup to use screen -dmS minecraft with the nogui option, and saw better performance in both setups, so I can only assume either something in the overnight update for Raspian aarch64 bulleyes fixed a bug in screen or ramfs, or my laziness to duplicate the world data, rather than share them with aliased symbolic links between setups, could be a discrepancy in an area of the micro-sd card that hampered my first timings comparison.
After doing a new comparison, and looking at the timings, both setups allocating 1240MB seemed very similar, and only on trying both with the view-distance – in the server.properties file – set to 16, did I then notice that the line trends, although very similar actually were resulting in more chunks being stored in memory on my config (10 versus 4, typically0 and when the orange loss of TPS sections happened, the green line didn’t ever drop from its 20 TPS average, whereas on the script setup the TPS loss was showing a dip in the green TPS line around these moments – like logging in, a “/save-all” or respawning from death – and then when I looked at the lag frame over-spends and the % number of laggy frames, that data reflected that observation of the lines, and those numbers are getting a 2 or more fold improvement in my comparison timings – unless I’m reading them wrong.
I whole heartedly agree, it is nimbling at the edges and even if the numbers implied more than a small improvement, again it could all be placebo benefits – or a regression – when tried in actual use with multiple client load.
Either way I’m pretty satisfied any more tinkering by me will be me, going in circles.
I’m very thankful for all your great input and producing your superb script that gave me great insight into how you optimised your paper.yml settings, along with the script clawing back about 250MB of RPi memory by using screen from linux in CLI mode.
I probably will get round to trying with an SSD – although seemed to remember my 2146Mhz overclock gets unstable with HDD and SSD storage last time I tried, so might go the RPi400 route as it has 4GB of ram for a 1GB ramfs allocation, and it is actually available in the UK, unlike the 4GB and 8GB RPi4 out of stock at MSRP. Normally I’d have spares of SSDs laying around for no-budget hobby projects, but a computer for each of nephews, and niece in recent months has wiped me out for now.
Whoops, sorry for the confusion on the Pi 3 vs Pi 4! Maybe that will come in handy for someone searching the comments later as people have asked that before and there’s some good tips in there for it. Overclocking on the Pi 4 is not nearly as “good” as the Pi 3 (the SD card port and I believe the RAM also do not overclock on the Pi 4).
You could definitely still do the general Pi 4 CPU overclock if you haven’t done that yet and the few other ones that are modifiable. It’s definitely an easy test and it costs nothing. Even if you set parameters that are too aggressive they can be reverted easily by putting the SD card into a PC and changing config.txt back basically. It’s best to have one of those little heatsink kits to do this but later firmware revisions won’t let the Pi 4 cook itself like the earlier revisions sometimes had a problem with.
That sounds about right for the JVM overhead. One thing that can impact it is 64 bit vs 32 bit. I’m not sure if not we ever talked about those but the 32 bit JVM tends to use a little bit less memory. Other factors can influence it such as what JVM you use. This probably isn’t worth chasing at this point but I figured I’d mention it for completeness.
So to explain my interpretation of the timings a little bit (I thought about going into detail about this but I had already wrote arguably too much) basically there are 2 different main views you will use. There is the “All” view which is the average over all ticks. You’ll notice your numbers are much better here. The default view it puts you in is the “Lag” view. This only shows ticks where there was lag / a problem.
Given that background there’s a few things I look for in the data. The first is I look for things that cause lag spikes. When the usage is near or over 100% (it will be dark red for these) it means that those operations were taking longer than the entire tick to complete. That’s a lag spike that the players will feel. It probably only happens every 5 minutes it so though for the world save tick for example (this is why my script turns it up to 15 minutes, it’s usually only a couple of a second spike even on really barebones setups without many resources).
However, the reason I said your timings look great is that the more important view is to use the “All” tab which is not the default. This is where everything looks great. Your TPS is pretty much at full with only a couple of times where you even had a little bump up in the chart. To sum it all up things look fantastic but it looks like you may have some occasional lag spikes people will “feel” during those expensive chunk saving / ticking operations. Frankly this can happen even on PCs (especially servers with old HDDs).
It’s definitely true they can’t be eliminated entirely. You can definitely get them under 100% though with a SSD but we aren’t talking about a huge lag spike still at over 100% (a BIG lag spike could be thousands of percentage points if it’s like 10-15+ seconds or something insane). I want to say my lag chart looks closer to highs of 30-40% of TPS on a NVMe SSD though and not anywhere near 100%. This is admittedly on an 8GB Pi though or a Compute Module 4. This is mostly just to give a reference of what kind of performance could possibly be achieved if you wanted to get an extremely expensive NVMe drive for it (wouldn’t really make sense to do unless you had one around).
I honestly think everything looks great. You don’t want to go in circles with it like you were saying. It’s all tradeoffs at the point and you’ve more than mastered the art! I thought you were the one that was unhappy with it to be honest with you so I think I may have misinterpreted that you may have wanted to go further still. You are basically right at the point I reached with the script where I was completely happy with it!
Obviously you can always throw more money or hardware at it but that wasn’t the point of it. I just thought it would be really cool and fun to run it on there and all the other guides / scripts out there were either horrible or never kept up to date. Since I was a hardware tech I had access to plenty of free SSDs that were being pulled out of destroyed / surplused computers.
The SSD would be the big upgrade you could do if you found that those occasional spikes were even noticeable or actually a problem. I totally understand about not having the spares around (they come and go, sometimes lots of RAM is surplused, sometimes lots of HDDs/SSDs and then sometimes there is droughts). I’m sure one will come back into your hands to give it a go at some point and it would be great to see a direct TPS comparison on the same Pi with a SSD and get a modern measurement of what the exact performance gains look like!
I do have the RPI 400 and it’s great! It’s the most powerful Pi you can get (it has the most MHz CPU). The CM4s are extremely powerful as well but that’s mostly due to the fact they have an IO board. This is basically a motherboard that they plug into so that the CM4 gains whatever capabilities the IO board has (including native PCIe 1x ports and other fun stuff). The cost overall is a little higher because of all the accessories you need to get that you don’t need for a regular Pi like the IO boards and things to connect them altogether but you can do some really cool stuff with them.
I’m grateful to you for the great conversation! I’m not sure if you could tell but I definitely enjoyed it as you asked me in a lot more detail than I think anyone ever has about some of these performance questions. There’s very few people out there that would be able to bear listening to me talking about the strange lengths I went to even getting this playable on those old Pis with so much less than even your 2GB Pi4 has (this should be a great setup).
It did gave me a chance to write down all of this too in one place outlining my “process” for creating and optimizing this script (using data and testing with the timings tool so we can know instead of suspect, there’s a lot of voodoo in old optimization guides still posted that does not bear out in the numbers). These techniques we’ve discussed could be used to optimize any Minecraft server. It would be the same process on the PC and Paper is still one of the best choices (some configuration required for best performance as you discovered).
People have done really impressive things with the script I definitely didn’t design it for or even imagine so I’m happy to share anything I know as a lot of it I learned from comments / conversations here just like this with others. My intention was for it to be an easy to modify base/template if that makes sense and people can run with it beyond anything I could design for / imagine.
Definitely feel free to stop by any time if you run into anything to bounce off me or find anything interesting you want to share!
CardinalFang36
2 years ago
Do you know if v1.17 has the needed fixes to protect against the log4j security flaw? I was under the impression that 1.18 was required. Any thoughts on this?
I’m honestly not sure. I do not think so. I looked at Paper’s downloads and 1.17.1 is considered a “legacy” version that is no longer supported and they will not support according to papermc.io/legacy
That does not look very encouraging. They don’t appear to really be supporting 1.17 at all anymore. I’ve heard it’s extremely dangerous to run even if you “think” you’ve fixed it. It doesn’t appear to be getting patched anymore and if you submit an issue related to that version they say you will receive 0 support and it will be closed. That’s very harsh compared to the past and I imagine it’s because they want *nothing* to do with the log4j exploits.
Basically the exploits equate to not only your server getting pwned 100% but every single player on there also getting pwned 100%. It’s basically as bad as it gets.
If you had to run 1.17 for some reason you’d want to do the official Minecraft fixes but they are honestly complicated.
1.18: Upgrade to 1.18.1, if possible. If not, use the same approach as for 1.17.x:
1.17: Add the following JVM arguments to your startup command line:
-Dlog4j2.formatMsgNoLookups=true
1.12-1.16.5: Download this file to the working directory where your server runs. Then add the following JVM arguments to your startup command line:
-Dlog4j.configurationFile=log4j2_112-116.xml
1.7-1.11.2: Download this file to the working directory where your server runs. Then add the following JVM arguments to your startup command line:
-Dlog4j.configurationFile=log4j2_17-111.xml
Versions below 1.7 are not affected
There’s the list of steps you’d need to do. You’d basically need to modify start.sh to include this JVM command line argument as well as download the files. You’d also have to be sure it’s working if you wanted to risk if it wasn’t that your server and everyone on it will get completely obliterated by hackers if you made a mistake. If you are comfortable trying to run the exploits against the server and testing it that is really the only way to be sure.
Because of the complexity of the fix (and really no easy way to tell it’s working other than patch it) I would say that I, like Paper, want nothing to do with telling anyone it’s safe to run those versions under any circumstances. The only exception to that would be if you wanted to do the manual fixes and run the exploits against your server to make sure they aren’t working.
My understanding is that there are all these fixes and plugins out there for this but I see just as many people saying they’re easily bypassed if all of those fixes aren’t done 100% correctly. It does not look like Paper is attempting to patch it to fix these though, and if they are they didn’t mention it anywhere and in fact made it sound like quite the opposite from their legacy section.
Basically the issue with these exploits is they are at a pretty low level (log4j). That’s why this same exploit can be used on thousands and thousands of different pieces of software. It’s an extremely devastating hack. That’s why you’re seeing people like me (someone who has published security research before and has some more upcoming CVEs to publish) really squirm with this one. It’s complicated to fix but more importantly you have to be *sure* because if you’re wrong it’s game over. Unlike being blown up by a creeper though this game over could result in your entire network being filled with malware / your identity being stolen / whatever else evil hackers are trying to do.
If this was a minor exploit I would have personally patched it and said it’s safe for all versions. It’s a lot more complicated with this one though because it’s so low-level. It’s very rare that an exploit this devastating and widespread is found (maybe once every few years?). You really have to get into the JVM and the libraries and start configuring them to fix this. Given that a lot of people may have multiple JVMs on their machine and other configuration differences this becomes impossible to make sure you’ve made every single setup 100% safe. I think that’s why Paper and others like me aren’t even trying. We can’t guarantee that will be safe no matter what we try to do unless you literally download the exploits and check it.
I just want you guys to stay as safe as possible and these fixes are anything but a sure thing. Everyone (including Mojang) has been saying the only true safe thing is to upgrade to the latest and I’ve pretty much only heard horror stories from people who didn’t. Hopefully that helps!
Thanks for your quick response….but I remain confused. It is my understanding that your script uses v1.17 of Minecraft yet the only “safe’ version we should be running these days (because of the log4j exploit) is v1.18. Is this correct, or do I not understand the details of Minecraft versioning?
No problem at all, I’m happy to clarify. This is a very important issue so I’d rather have any questions about it asked and fully answer them for sure. It will help others who come and see this conversation who may have the same question as well!
If you’re hosting your own Minecraft: Java Edition server, you’ll need to take different steps depending on which version you’re using, in order to secure it.
1.18: Upgrade to 1.18.1, if possible. If not, use the same approach as for 1.17.x:
1.17: Add the following JVM arguments to your startup command line:
-Dlog4j2.formatMsgNoLookups=true
1.12-1.16.5: Download this file to the working directory where your server runs. Then add the following JVM arguments to your startup command line:
-Dlog4j.configurationFile=log4j2_112-116.xml
1.7-1.11.2: Download this file to the working directory where your server runs. Then add the following JVM arguments to your startup command line:
-Dlog4j.configurationFile=log4j2_17-111.xml
Versions below 1.7 are not affected
According to the official Minecraft post for versions 1.17 you would add -Dlog4j2.formatMsgNoLookups=true to your server launch line. You’d essentially put this in start.sh. It says you don’t need the .xml fixes for version 1.17 according to Microsoft / Mojang. If it’s older than that there are more fixes you need to do with xml files to be safe. I found another source with this information here.
It would be difficult to automate doing the correct fixes for each one of these old versions (there’s up to 3 different fixes you have to do depending on how old it is). I can’t be confident they are using a Java version that will even respect the log4j flag I’m passing to it. It would basically be guarantees / promises I can’t really make because I can’t be sure of everyone’s exact configuration (and when I’ve made assumptions like this in the past people always let me know hey we were using it that way your assumption was wrong)!
My version does not use 1.17 (although you may have that if you installed during that time) and I don’t recommend using it to be honest. The only exception is if you are comfortable testing the exploits against your server after doing the fixes for the version you want to run which is the only way to be sure it’s fixed (other than upgrading). The current version will be using version 1.18.2. If you are still on 1.17 (with a previous install) you may just need to run SetupMinecraft.sh again. This will update you to the latest version (1.18.2). It is safe to do this on an existing server (it’s designed for it) but it’s always a good idea to make sure you have your backups saved outside of the Minecraft backups folder before upgrading.
Hopefully that clears things up and let me know if you have any more questions!
My confusion was around the fact that this page starts with “Minecraft 1.17 (Caves and Cliffs Update) is here! ” so I thought the latest version your script would support was currently 1.17. I see now that it is using the latest “1.18.2”. I just did a clean install on my Raspberry Pi 4 (8GB) as the prior setup was on Ubuntu and I seem to be having “apt get update” issues with that…so used this as an opportunity to start with a fresh install overall. I was also concerned that if someone had exploited the log4j issue there is no way I would ever know.
I actually had thought that I had been hacked. I had not used the server for a while and found out it was down. When I investigated, I found the server was out of disk space. I panicked but upon investigation found that the daily backups of the server data was the cause of the disk space issue.
That solves the mysteries! So the backup issue has actually been fixed. For most of 1.18 being out it has only kept the most recent 10 backups. I had the same thing happen to my server and added this in to prevent this so at least that shouldn’t happen again on the new version!
I’m glad you weren’t actually hacked. I think you should be all safe and secured now and the backups won’t be a problem with the new “pruning” system that keeps the most recent 10 backups. Cheers and have fun!
Andre
2 years ago
is there any way to install a snapshot version, for instance the new deep dark snapshot thats not in the launcher yet?
That’s a great question. I think the problem is mainly that you would need a link to the zip file that has the dedicated server for that snapshot.
Theoretically if we had this you absolutely could do that. Do you know of a public download somewhere? It might actually work to just extract that zip over your existing server (make sure you have backups before trying this). Theoretically that would work if there is a public download!
I tried using winscp to move the file for the server.jar file into the minecraft folder on the server for the newest deep dark snapshot and ran it like a normal server without your script and it worked
James,
My server has been running fine for several days but when I went to use it this morning, it was not running. When I investigated, there seems to be an error with sudo and “pam_unix”. I don’t think this is related to your script (other than the 4am reboot) but was hoping you might know what is happening here.
Thanks
—-
Mar 23 11:44:55 paynecraft systemd[1]: Starting Minecraft Server Service...Mar 23 11:44:55 paynecraft bash[948]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or confi>
Mar 23 11:44:55 paynecraft bash[948]: sudo: a password is required
Mar 23 11:44:55 paynecraft sudo[948]: pam_unix(sudo:auth): conversation failed
Mar 23 11:44:55 paynecraft sudo[948]: pam_unix(sudo:auth): auth could not identify password for [mcadmin]
Mar 23 11:44:55 paynecraft bash[956]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or confi>
Mar 23 11:44:55 paynecraft sudo[956]: pam_unix(sudo:auth): conversation failed
Mar 23 11:44:55 paynecraft sudo[956]: pam_unix(sudo:auth): auth could not identify password for [mcadmin]
Mar 23 11:44:55 paynecraft bash[956]: sudo: a password is required
Mar 23 11:44:55 paynecraft bash[957]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or confi>
Mar 23 11:44:55 paynecraft bash[957]: sudo: a password is required
Mar 23 11:44:55 paynecraft sudo[957]: pam_unix(sudo:auth): conversation failed
Mar 23 11:44:55 paynecraft sudo[957]: pam_unix(sudo:auth): auth could not identify password for [mcadmin]
Mar 23 11:44:55 paynecraft bash[946]: Backing up server (to cd minecraft/backups folder)
Hey CardinalFang36,
Welcome back! That is a strange one. The first thing I would try is running SetupMinecraft.sh as this will set up your sudoers file again (it’s at /etc/sudoers.d/minecraft). I believe you are correct that it’s sudoers related. You could also manually look in there and see if the file is missing or if something looks wrong / different.
Since you weren’t trying to update nothing should have changed (other than maybe Paper Minecraft itself updating to a newer revision that is doing this) at least with the scripts themselves. I’d be curious what you find the sudoers file situation is as this is what I would expect to happen if it was disabled / missing (maybe due to apt updates, hard to say why).
This is my sudoers file:
cat /etc/sudoers.d/minecraftmcadmin ALL=(ALL) NOPASSWD: /bin/bash /home/mcadmin/minecraft/fixpermissions.sh, /bin/systemctl start minecraft, /bin/bash /home/mcadmin/minecraft/start.sh, /sbin/reboot
I reran SetupMinecraft.sh (seemed fine except this: OpenJDK installation failed. Java version is still reporting as less or greater than OpenJDK 16!)
after, my sudoers looks like:
cat /etc/sudoers.d/minecraftmcadmin ALL=(ALL) NOPASSWD: /bin/bash /home/mcadmin/minecraft/fixpermissions.sh, /bin/systemctl start minecraft, /bin/bash /home/mcadmin/minecraft/start.sh, /sbin/reboot
Looks to be the same,
It does appear that a new paperclip.jar appeared last night
Hey CardinalFang,
Which OS and version is this? I’m guessing your pam.d configuration is modified / messed up (probably from apt updates). This seems especially likely since it happened out of the blue.
Here is a StackOverflow thread that may help
I’m not sure what your comfort level is here but it looks like if the apt breaking pam.d theory is correct then you may just need to add:
auth sufficient pam_permit.soin the file like the answer that was picked in my link has. If this is Raspberry Pi OS or Ubuntu installed on the Pi I would say it would be very wise to reimage this Pi and then just restore the server from a backup. It seems like the Pi’s image is not sound here. You can definitely try just patching it up and see if that resolves it though!
I applied the fix from StackOverflow
Now see this:
Mar 23 23:27:14 paynecraft sudo[5588]: mcadmin : PWD=/home/mcadmin/minecraft ; USER=root ; COMMAND=/usr/bin/sh -c echo 1 > /proc/sys/vm/drop_cachesMar 23 23:27:14 paynecraft sudo[5588]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Mar 23 23:27:15 paynecraft sudo[5588]: pam_unix(sudo:session): session closed for user root
Mar 23 23:27:15 paynecraft sudo[5597]: mcadmin : PWD=/home/mcadmin/minecraft ; USER=root ; COMMAND=/usr/bin/chown -Rv mcadmin /home/mcadmin/minecraft
Mar 23 23:27:15 paynecraft sudo[5597]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Mar 23 23:27:16 paynecraft sudo[5597]: pam_unix(sudo:session): session closed for user root
Mar 23 23:27:16 paynecraft sudo[5599]: mcadmin : PWD=/home/mcadmin/minecraft ; USER=root ; COMMAND=/usr/bin/chmod -Rv 755 /home/mcadmin/minecraft/fix>
Mar 23 23:27:16 paynecraft sudo[5599]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Mar 23 23:27:16 paynecraft sudo[5599]: pam_unix(sudo:session): session closed for user root
Mar 23 23:27:16 paynecraft bash[5586]: Backing up server (to cd minecraft/backups folder)
But the server still will not start. When I run “screen -r” I see Bukkit starting for a second and then the screen exits.
I guess the next step is rebuild and restore.
When I try to run manually, I am now getting:
Starting org.bukkit.craftbukkit.MainUnsupported Java detected (62.0). Only up to Java 17 is supported.
FYI: Running on Raspberry OS
Linux paynecraft 5.10.103-v8+ #1530 SMP PREEMPT Tue Mar 8 13:06:35 GMT 2022 aarch64 GNU/LinuxHey CardinalFang,
Great find! Check out Markus’ post here.
Can you try reverting your snap package like he did to OpenJDK 17? This is going to be a pretty big problem since the OpenJDK in the Raspberry Pi repositories is still too old (Ubuntu’s might be new enough). I think if you try those instructions it should get it going!
It looks like they made it impossible to install an older version than OpenJDK 18 from snap as of March 22nd (yesterday). Hopefully I removed it before too many people get upgraded. You should still have your old snap that you can revert to.
Raspberry Pi OS only seems to have OpenJDK 17 documentation available on the armhf version in the apt repositories. I’ll have to test the 64 bit version and see if it’s available on there. I did add some additional apt selections to SetupMinecraft.sh that should pick these up on Ubuntu and Raspberry Pi OS (if it’s available, doesn’t seem to be on 32 bit but I’m testing now as I’ll probably have to find a compatible package to download).
It’s going to be trickier for new installations. I’m going to have to figure out how to install OpenJDK 17 reliably for now. That’s very annoying it’s still not compatible when it’s now in the main release channel but I’ll figure something out!
Thanks for thinking to run it manually like that as this was a very unexpected outcome. They literally just switched the repositories over on the 22nd so this basically just happened. The snapd installation is already out of SetupMinecraft.sh and you thinking to test this manually definitely helped me to find it faster!
Now that it’s out of SetupMinecraft.sh anyone who is upgrading from an old installation would just keep whatever version they have installed on there (far more likely to be right than the current snapd version which is certain to be wrong). I just need to find a way to get OpenJDK 17 on new installations which I’ll be working on for the next day or two on 32 and 64 bit test systems.
James,
My “snap” syntax appears to be different than Markus’:
snap list openjdk --allsudo snap revert openjdk --revision=818
But after applying that, my server is back up and running.
Who would have guessed that a guy who sells QA software for a living could actually help debug an issue. Thanks for your help…and your wonderful script.
Tom Cardinal Fang
Hey CardinalFang,
Excellent, sorry that you got caught up in that! That makes sense that you have a different revision. The OpenJDK snap doesn’t “allow” reverting to older versions *unless* you already had one installed. This will be different for everyone unfortunately as it will be whatever OpenJDK revision was live when they installed that they will have. This means this trick won’t work for new users, or if you wiped the Pi it would also no longer work. It also means everyone has a different version which makes it a lot dicier to try to automate this. That’s why I outright removed the snapd installation piece as since only OpenJDK 18+ is available now via snap literally nobody should ever use it now as it will only break it.
This appears to be a design choice by OpenJDK (they do HARD cutoffs of the OpenJDK versions and they only last for about 6 months with the exception of a LTS release that is still only good for 2 years) and it is going to be a nightmare to deal with. I just updated my Pi 400 to Bullseye to see what is available there but it looks like builds of OpenJDK 17 for the Pi are very difficult to find. They also work on different platforms. The Pi Zero W for example doesn’t have an OpenJDK anywhere near 16 available. It may look something like the 64 bit flavor of Raspberry Pi OS can install a new enough one and the 32 bit one can’t unless I give it some help with a third party package.
There’s a few different ways I could deal with this and I’m trying to find the easiest and most widely compatible / portable one. One option I’ve been thinking is I could have the project have it’s own “JDK” where it downloads and stores the JDK in your minecraft folder such as ~/minecraft/jdk for example as a base JDK for all servers installed. I’m really hoping I can find a way to take care of this with apt though (at least on Bullseye for Raspberry Pi OS).
I’m a little stunned that Paper still doesn’t support OpenJDK 18. They’ve always caught up before this happened in the past and it was safe to just keep using the newer OpenJDK but it’s different this time. This may not be entirely Paper’s fault. There was some very strange behavior happening with OpenJDK 18 to you guys that I’ve never seen before. It’s probably structural changes within Java that are making this a challenge.
I’m always thrilled to have you help debug! It’s kind of funny because all of the issues for the past few days have required me to make fixes to the script. Usually I go months at a time without anyone finding one of these. It definitely comes in waves.
I’ve fixed this once and for all. The new version of SetupMinecraft actually sets up a dedicated OpenJDK (stored in your server’s folder under jre). It’s never too hard though thanks to you guys, it is appreciated!
I’m having the same issue. Pretty newbie question (which I am), but what’s the easiest way to perform the update of SetupMinecraft, when you have an existing installation?
Hey David,
Great question! Basically just run SetupMinecraft.sh again and leave everything default / the same as it was and it will detect and update your existing server.
The new version takes care of OpenJDK a different way by downloading OpenJDK 17 and putting it in the Minecraft folder.
You’ll basically just enter the same things for the server as you did the first time (or leave them as default). If you have any trouble let me know!
Thanks James. In that case, I’m not sure what I’ve done. Should SetupMinecraft.sh still be present from the original install or do I need to re-download it? I only ask, as I can’t see it installed currently.
Hey David,
No worries at all. You basically run it the same way as before (you don’t have to download it basically). All you will do is run:
sudo curl https://raw.githubusercontent.com/TheRemote/PiBenchmarks/master/Storage.sh | sudo bashIn other words, the process to update the scripts is the same as installing it in the first place. It’s designed to safely do this and update the existing scripts.
Sorry for not being more clear the first time, hopefully that gets it for you!
You meant this, right? 😉
curl https://raw.githubusercontent.com/TheRemote/RaspberryPiMinecraft/master/SetupMinecraft.sh | bashThanks for the help here – that’s worked. I’m such a noob with all this, I keep considering giving up but, you know, there’s only one way to learn!
Hey David,
That is it! Thanks for leaving your resolution so I know you got taken care of, I’m very glad you’re back up and running!
I’ve been struggling with this error too – server works fine, but those log messages are annoying, particularly as I’m also adding a notification on minecraft join/leave via a comobination of tail (the log file) and ssmtp – so given the mailbad statement means I get emails telling me about these failures. #
The fixes noted in the original thread (editing PAM modules) looked a bit dangerous to me (basically making sudo passwordless throughout) as highlighted in the linked StackOverflow thread.
Looking at the code I wonder if it is an issue with fixpermissions.sh and some of the calls? Two potential areas to look at (and apologies for formatting – first post here so this will be garbled, no idea how to quote etc):
1. in fixPermissions.sh there is this segment when testing the options for flags (starts line 20)…
——————
while getopts ":a:" opt; docase $opt in
t)
case $OPTARG in
case $OPTARG in
''|*[!0-9]*)
Automated=1
;;
*)
Automated=1
;;
esac
—————–
That case statement looks a little funny? Shouldn’t it be a) rather than t)? (the case statement goes on to note countdown etc rather than automation). Similarly, we are not after an actual value are we, so we could just have getopts “:a” (i.e. no second colon), and simplify the case statement? The upshot of this is that when fixPermissions,sh is called from start.sh, it is not setting the Automated flag and therefore not using the sudo -n statement further on down. However, even if you make those changes, so still get similar sudo errors requiring passwords, even if sudoers.d/minecraft is all OK
2. The second problem then is looking in sudoers.d/minecraft. That gives the local user passwordless permission to run fixpermissions.sh, start.sh etc. However, the actual sudo commands (e.g. chown in fixpermissions.sh, sh in start.sh) will only inherit that status, if those scripts are called via sudo. The minecraft service definition however (under /etc/systemd/ssytem/minecraft.service) includes this:
———————-
[Service]User=[user]
WorkingDirectory=/home/[user]/minecraft
Type=forking
ExecStart=/bin/bash /home/charon/minecraft/start.sh
———————-
Note the User= command. The default for systemd on startup is to run as root, however as I understand it, when User is set in the service that is no longer the case. Therefore, that ExecStart will not be called via su, and therefore that elevated status not passed onto calls, so when the script eventually hits the sudo commands, the owning script has not been invoked by sudo and therefore scripts are required. Two solutions, either add chown and sh to sudoers.d (not a good idea possibly), or force a call via sudo in the service i.e. change the ExecStart line to:
————————-
ExecStart=/usr/bin/sudo /bin/bash /home/charon/minecraft/start.sh
————————–
I made all the above changes and rebooted and those errors have now gone away (no idea if it makes any difference at all).
As noted, well beyond my comfort zone with a lot of this, so apologies if foolish errors (and lack of formatting)
So I’m not sure why you had to do any of this to be honest. The sudoers file is:
/etc/sudoers.d/minecraftIt has the following contents:
james ALL=(ALL) NOPASSWD: /bin/bash /home/james/minecraft/fixpermissions.sh, /bin/systemctl start minecraft, /bin/bash /home/james/minecraft/start.sh, /sbin/rebootIt basically already works this way. You can see that fixpermissions is allowed to run in it’s entirety sudoless. Is this a weird operating system like a Red Hat operating system? Some of those have different sudoers file formats and maybe yours is one of the unsupported/untested platforms. I’m aware that these guys are coming to Raspberry Pi real soon as well via Fedora and I’m not sure if that one uses a nonstandard sudoers file yet (I hope not because that one will be officially supported and I don’t want a bunch of janky code making special exceptions for Red Hat in this, there’s nothing special about not conforming to standards other than being garbage/annoying).
If you aren’t on the Raspberry Pi that is okay. This works fine on everything else as well (usually, you may have found one of the exceptions). That’s just traditionally what was popular about this script and is why it hasn’t come up a whole lot. It’s possible if Fedora requires a different sudoers format that I will have to upgrade this somehow to be able to tell the difference between them.
I can just delete all that case stuff out of there. I didn’t write it. It was part of a pull request and I’m totally fine with deleting all of it. I can delete 95% of that code without any consequences. I just did this. All of that code is now deleted and fixpermissions only contains this:
echo "Taking ownership of all server files/folders in dirname/minecraft..."sudo chown -Rv userxname dirname/minecraft
sudo chmod -Rv 755 dirname/minecraft/*.sh
That script shouldn’t have mattered unless your server was already broken. Did you manually edit server files / permissions or move them around? Fixpermissions is not required to run the server. It only fixes it if the server files have the wrong permissions on it.
I’m going to guess you are running one of the few operating system that implements their own format of sudo that is different from every other operating system out there. Can you confirm this? Needless to say I disagree with them that their sudoers format is the one true format and do not plan on supporting it if that’s the case (gladly, without standards most of the advantages of Linux start to disappear). I hate operating systems / flavors that do it completely different from any others for no reason and I feel like that has to be part of this one!
It has come up once or twice before but it has been a while since I saw one with a different sudo system. If you have this I would highly recommend using the Docker version because you don’t have a compatible flavor of Linux that implements standard sudo (if this is the case). They basically aren’t conforming to Linux standards and are doing their own thing in some cases (and a couple of big names do this although they are usually premium/enterprise like Red Hat).
It’s also possible you just had broken permissions on one of the files. If this is all it was then this is now fixed. I completely revamped that automated code part (as in deleted it). It was throwing Snyk warnings so I have no doubt something was wrong with that piece of it but again this code does nothing unless your server’s permissions were wrong in the first place (which should never happen unless things have been moved around or files were edited by a different user and now they have ownership).
Given how you described your issue though I feel like if you did a fresh install then you’d have to make similar modifications. If that is the case then this is a non-standard Linux distro using whatever sudo format they feel like I guess. If you are comfortable with this then there’s no reason now that you know what to do you couldn’t absolutely move forward as the updates will all still happen normally and should not be impacted by any of this. Hopefully that helps!
Except that now checking it I see that the server is not running on reboot – start.sh traps out root calls at the start (sensibly). So above a broken solution (sorry) but hopefully helpful in some way.
Hey Funkyl,
Can you verify your /etc/sudoers.d/minecraft file? It just sounds like it’s not working to me. I think you’re right that you’re having sudo issues. I’d imagine it’s because of your operating system though! If you verify the sudoers file you’ll likely find it’s the wrong format for your OS. I can’t help but notice you did not state which OS this is. If you ran this on Ubuntu or Debian or one of those flavors this does not happen so I am going to need to know that to do anything to help.
This is exactly what I would expect to happen if the sudoers file isn’t working. It can’t run sudo systemctl start without it.
This is not something I could fix for you without writing special code for your operating system to make a sudoers file in the format your OS will take. I would need to know what OS we’re talking about as well as this cannot be a standard/common one. It has also only come up 1 or 2 times ever in the 1000+ comments on this article.
I’m definitely willing to explore fixing this depending on how obscure your OS is. If it’s Red Hat/RHEL after all then I may do it just to prepare for Fedora to come to the Raspberry Pi officially. That is what we are talking about though. We are talking about supporting your flavor of Linux. It already works on the OS that 90%+ of people are running. It does not look like it currently supports whichever one you are running though and frankly you may be the first person to ever try depending on which one it is.
There is nothing wrong with the script. I need to know what uncommon OS is being ran here. I will then explain and link to explanations about why your OS doesn’t work the same way as everyone else’s and what you can do about it (or potentially what I can do about it as well depending on what the problem is). Issues like this are why I have moved away from standalone scripts to Docker for everything new and things like my Geyser + Floodgate container don’t have a standalone version. It’s so difficult to support standalone when there are different competing standards being used.
I have 2 standalone scripts and 4 Docker containers at this point and the 2 standalone ones I did are from 4-5 years ago and all the Docker containers are pretty much from this year. That is basically directly because of issues like this. Everyone using the Docker container will have a better experience and never see any issues like this ever because it’s all standardized. Because of the container everyone is really running Ubuntu within the container. Linux distros aren’t always similarly standardized unfortunately (but they’re usually close or this wouldn’t have worked at all in general and would have had to have a list of specifically supported operating systems/flavors).
Hi…. I had been going round and round in circles re the sudoers.d file… more on that in a mo.
In terms of my environment – this is on an Odroid N2+ running their flavour of Ubuntu Mate (22.04) . I’m doing all this headless via ssh.
In terms of sudoers.d/minecraft, this is what it looks like:
charon ALL=(ALL) NOPASSWD: /bin/bash /home/charon/minecraft/fixpermissions.sh, /bin/systemctl start minecraft, /bin/bash /home/charon/minecraft/start.sh, /sbin/rebootBeen looking at this for a few hours or so on the internet, and a common issue apparently is overwrites in /etc/sudoers, but the @includedir /etc/sudoers.d is at the bottom of that file so can’t be that.
For confirmation, sudo -ll gives the following:
——————————————-
Matching Defaults entries for charon on pluto:env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/snap/bin, use_pty
User charon may run the following commands on pluto:
Sudoers entry:
RunAsUsers: ALL
RunAsGroups: ALL
Commands:
ALL
Sudoers entry:
RunAsUsers: ALL
Options: !authenticate
Commands:
/bin/bash /home/charon/minecraft/fixpermissions.sh
/bin/systemctl start minecraft
/bin/bash /home/charon/minecraft/start.sh
/sbin/reboot
———————————–
… so assuming that !authenticate means not authenticate, then sudoers.d looks to have bedded in as intended.
Have tried rerunning the original script a couple of times to reset my tweaks to scripts back to default entries … and no change to those log entries. Just spied your update to fixpermissions.sh on github though so will retry that.
Hope something useful in there.
Re your message above, this is my ignorance I’m afraid. The way I was figuring it was that that the systemctl service is *not* running as root (as User is defined in the service definition), therefore fixpermissions will not be called as root/su and therefore the sudoers.d settings will not be invoked when called and so the embedded “sudo” in that script will want interactive authentication. Similarly to the sudo call in start.sh. Probably some garbled logic in there.
Thanks by the way – great script and very easy to set up. Best by far I found on the internet (and as noted on an N2+ as well). As noted the server is running fine, it is just those sudo calls and log entries, which it appears do not hamper the default functionality too much (noting what you highlight re permissions in the top of the fixpermissions script)
Hey FunkyL,
That’s fascinating, I actually have one of those I can test! I’ve recently done some coverage of the N2+ on the site.
Let me take a look on there and see what is going on as this happens to be one that is easily available for me to check.
Okay, so what I did was run SetupMinecraft.sh fresh and had it set up a new server with the name “james”. I did this on the default odroid user.
Everything seems to be okay on initial first startup. I did get a really weird dripstone block error for some reason but I’m guessing that’s related to the seed / random. I typed screen -r and the server was running:
Starting org.bukkit.craftbukkit.MainSystem Info: Java 18 (OpenJDK 64-Bit Server VM 18.0.2+9) Host: Linux 4.9.312-125 (aarch64)
Loading libraries, please wait...
[20:20:10 INFO]: Building unoptimized datafixer
[20:20:14 INFO]: Environment: authHost='https://authserver.mojang.com', accountsHost='https://api.mojang.com', sessionHost='https://sessionserver.mojang.com', servicesHost='https://api.minecraftservices.com', name='PROD'
[20:20:14 INFO]: Found new data pack file/bukkit, loading it automatically
[20:20:19 INFO]: Loaded 7 recipes
[20:20:23 INFO]: Starting minecraft server version 1.19.2
[20:20:23 INFO]: Loading properties
[20:20:24 INFO]: This server is running Paper version git-Paper-138 (MC: 1.19.2) (Implementing API version 1.19.2-R0.1-SNAPSHOT) (Git: 09904fd)
[20:20:24 INFO]: Server Ping Player Sample Count: 12
[20:20:24 INFO]: Using 4 threads for Netty based IO
[20:20:25 INFO]: Default game type: SURVIVAL
[20:20:25 INFO]: Generating keypair
[20:20:25 INFO]: Starting Minecraft server on *:25565
[20:20:25 INFO]: Using epoll channel type
[20:20:25 INFO]: Paper: Using libdeflate (Linux aarch64) compression from Velocity.
[20:20:25 INFO]: Paper: Using OpenSSL (Linux aarch64) cipher from Velocity.
[20:20:26 INFO]: Preparing level "world"
[20:21:14 INFO]: Preparing start region for dimension minecraft:overworld
[20:21:16 INFO]: Time elapsed: 1470 ms
[20:21:16 INFO]: Preparing start region for dimension minecraft:the_nether
[20:21:16 INFO]: Preparing spawn area: 0%
[20:21:17 INFO]: Time elapsed: 1307 ms
[20:21:17 INFO]: Preparing start region for dimension minecraft:the_end
[20:21:18 INFO]: Preparing spawn area: 0%
[20:21:18 INFO]: Preparing spawn area: 0%
[20:21:18 INFO]: Preparing spawn area: 0%
[20:21:19 INFO]: Preparing spawn area: 0%
[20:21:20 INFO]: Preparing spawn area: 0%
[20:21:20 INFO]: Preparing spawn area: 0%
[20:21:20 INFO]: Preparing spawn area: 0%
[20:21:21 INFO]: Time elapsed: 4012 ms
[20:21:21 INFO]: Running delayed init tasks
[20:21:21 INFO]: Done (57.814s)! For help, type "help"
[20:21:21 INFO]: Timings Reset
[20:21:34 ERROR]: Detected setBlock in a far chunk [-6, -1], pos: BlockPosition{x=-82, y=-11, z=-1}, status: minecraft:features, currently generating: ResourceKey[minecraft:worldgen/placed_feature / minecraft:large_dripstone]
>
Okay, so then I answered “Y” on the daily reboot and “Y” on the start the server automatically at startup. I verified my crontab with crontab -e as:
0 4 * * * /home/odroid/minecraft/restart.sh 2>&1I then checked my /etc/sudoers.d/minecraft file:
odroid@odroid:~$ cat /etc/sudoers.d/minecraftodroid ALL=(ALL) NOPASSWD: /bin/bash /home/odroid/minecraft/fixpermissions.sh, /bin/systemctl start minecraft, /bin/bash /home/odroid/minecraft/start.sh, /sbin/reboot
Okay, let’s test rebooting it with sudo reboot and see if it will come back on it’s own. It did. You can verify this with:
odroid@odroid:/etc$ sudo systemctl status minecraft● minecraft.service - Minecraft Server Service
Loaded: loaded (/etc/systemd/system/minecraft.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-09-04 20:27:23 UTC; 24s ago
Process: 4006 ExecStart=/bin/bash /home/odroid/minecraft/start.sh (code=exited, status=0/SUCCESS)
Tasks: 45 (limit: 3832)
Memory: 745.6M
CGroup: /system.slice/minecraft.service
├─4073 SCREEN -dmS minecraft /home/odroid/minecraft/jre/bin/java -DPaper.IgnoreJavaVersion=true -jar -Xms400M -Xmx2000M /home/odroid/>
└─4074 /home/odroid/minecraft/jre/bin/java -DPaper.IgnoreJavaVersion=true -jar -Xms400M -Xmx2000M /home/odroid/minecraft/paperclip.jar
Sep 04 20:27:20 odroid bash[4061]: % Total % Received % Xferd Average Speed Time Time Time Current
Sep 04 20:27:20 odroid bash[4061]: Dload Upload Total Spent Left Speed
Sep 04 20:27:21 odroid bash[4061]: [237B blob data]
Sep 04 20:27:21 odroid bash[4006]: Latest paperclip build found: 138
Sep 04 20:27:21 odroid bash[4070]: % Total % Received % Xferd Average Speed Time Time Time Current
Sep 04 20:27:21 odroid bash[4070]: Dload Upload Total Spent Left Speed
Sep 04 20:27:23 odroid bash[4070]: [237B blob data]
Sep 04 20:27:23 odroid bash[4006]: Starting Minecraft server. To view window type screen -r minecraft.
Sep 04 20:27:23 odroid bash[4006]: To minimize the window and let the server run in the background, press Ctrl+A then Ctrl+D
Sep 04 20:27:23 odroid systemd[1]: Started Minecraft Server Service.
This image is basically brand new. Maybe a week ago. I had 13 updates since my initial image of it at least. I’m guessing if you do a:
sudo systemctl statussudo systemctl list-units --failed
that you have more failed units than just the minecraft server. In the past when people have had startup/service related issues like this it’s usually the sign of a sick systemd/bad image. If this has been upgraded from a previous version like 20.04 and hasn’t been reimaged with a clean 22.04 image that is the first thing I would try with this one. I definitely was able to do this properly on a clean image without anything strange.
Some of the commands I gave you may help in diagnosing this as well. The useful ones are sudo systemctl status minecraft (especially right after you reboot and it’s not coming up) and then looking in the “logs” folder for “latest.log”. Those will have the most technical information about what is going wrong. It should not happen on a clean N2+ 22.04.1 image (or at least I was able to verify that it didn’t). The good news is I’m sure this is resolvable and that your platform is supported. Hopefully that helps!
Thanks James – I did upgrade from an older version (20 or 21 can’t remember) to 22 so that may be it, so I’ll try a download and clean reinstall and report back here (wont be tonight)
Hey FunkyL,
No problem, I was not at all sure what was going to happen with this one because I have never tried running it on my ODROIDs yet! I just got my few ODROID models I have here in the past few weeks so this had not come up.
It does look image related though. I had a *ton* of image related issues when I upgraded my ODROID M1 from 20.04 to 22.04 in my ODROID M1 Review. At the time there was no 22.04 image available for the M1 so I ended up recommending that people use the server image first and then upgrade to get the cleanest upgrade possible and then to install their desktop later with apt (sudo apt install ubuntu-desktop or ubuntu-desktop-full).
The N2+ seemed to not have as many issues upgrading from 20.04 to 22.04 as the M1 (as in the M1 straight up couldn’t do it with the desktop image) but that doesn’t mean everything upgraded clean and given the M1 experience there’s no way that this one was just perfect (and indeed some weird things seem to be happening that shouldn’t be). This newer image is definitely available fortunately for the N2+ right now as I used it last week.
Sometimes upgrade problems can also depend on what you answer to the questions about whether to replace config files with new copies or leave the existing copy (sometimes the right answer is keep the custom one if it’s just one you’ve customized and sometimes the right answer is replace it with the newer config if it’s a major overhaul of an existing system because the old config will break it, it just depends). I honestly think that the images just don’t upgrade very cleanly a lot of the time though and that definitely isn’t unique to ODROID for sure (and others are often much, much worse).
In fact my N2+ installation on the eMMC is actually having some upgrade related problems from going from 20.04 to 22.04 (the one I tried last week was for my SSD booting guide for the N2+ and that was a clean install on the SSD but my eMMC install is an upgrade). It’s two annoying snap packages and I honestly don’t even know how to fix this one (it’s the chromium-browser and firefox packages). Observe:
Preconfiguring packages ...(Reading database ... 220363 files and directories currently installed.)
Preparing to unpack .../00-chromium-browser_1%3a85.0.4183.83-0ubuntu2_arm64.deb ...
=> Installing the chromium snap
==> Checking connectivity with the snap store
==> Installing the chromium snap
error: cannot perform the following tasks:
- Run configure hook of "chromium" snap if present (run hook "configure": cannot perform operation: mount --bind /snap/core20/current/etc/nsswitch.
conf /tmp/snap.rootfs_WcYly0/etc/nsswitch.conf: Permission denied)
dpkg: error processing archive /tmp/apt-dpkg-install-p5rOd0/00-chromium-browser_1%3a85.0.4183.83-0ubuntu2_arm64.deb (--unpack):
new chromium-browser package pre-installation script subprocess returned error exit status 1
Preparing to unpack .../01-firefox_1%3a1snap1-0ubuntu2_arm64.deb ...
=> Installing the firefox snap
==> Checking connectivity with the snap store
==> Installing the firefox snap
They fail every time so even this N2+ upgrade from 20.04 on the eMMC is broken. Even if I can fix this minor package issue I don’t trust it. Who knows what else is wrong with it that isn’t as obvious. This one is due for a clean 22.04.1 reimage just like you’re doing.
Thanks for sticking with it. This was something I definitely needed to test as it’s a little embarrassing to have 3 ODROIDs now and to admit I have not ran my Minecraft server script on any of them yet. Let me know what you find!
Just to close this out, I reinstalled from the ground up tonight, moving to an SSD while I did. I used a totally fresh install of 22.04 (using netboot_install ubuntu mate minimal 22.04 from Petitboot), and beyond installing e.g. wifi drivers, ssh servers, xrdp installed this script. Alas the same warnings appear in journalctl, so not sure what is going on. To be honest it is a nice to have, and I’ll live with it. The Docker solution is one option, but I slightly prefer having the server not in a virtual environment as it facliattes integration with other things going on on the machine.
Thanks for all your help and your great sight – looking through all your old posts now and some really valuable stuff in here
One aside… I was following your guide for booting N2+ from SSD (I’m using a USB one), and interesting the boot arguments in Petitboot was already set as you highlight to sda, it seemed to have autodetected it.
Hey FunkyL,
Excellent, I’m very glad you met with success here! I was able to eliminate those warnings by doing a clean 22.04 install but all of my upgrades have them. Most of those services are configured to fall back when this is the case so they just generate annoying warnings. If it’s an actual error that is a different thing / a problem but the warnings usually indicate it’s just going to fall back to the old functionality.
For what it’s worth I was able to get a clean install without the journalctl warnings by completely reinstalling (i.e. not using an upgrade install) of 22.04 right from the start. I did not try with the minimal though. I’m surprised you still get those journalctl warnings using that method as I got the impression it has to do with the partitions and /etc/fstab content when these are set up (related to mounting flags a lot of the time). They definitely do seem to be harmless though and even my upgrade installs with the journalctl warnings functioned well and ran everything I tried to do as normally as the clean install that did not have those warnings present (again for what it’s worth).
My Petitboot did seem to autodetect it as well. I believe I had other choices showing up as well was the issue on my end (I had several other storage devices connected during some of the testing as I was trying different methods of cloning them and ended up landing on this one). It’s really quite smart. I’m honestly impressed by Petitboot and would definitely look on any ODROID board’s having that as having a definite advantage. I wish my XU4Q had a Petitboot loader for example.
Thank you for closing this out with all your final results / conclusions, take care!
Not sure what happened, my server suddenly rebooted tonight and stopped working afterwards. I attempted to backup the world folders and deleted the minecraft folder in an attempt to start from scratch again after trying the restart script with no luck. When running the setup script now, I keep getting stuck at the openjdk version is less or greater than 16 message.
Checked and can see that openjdk 18 was released today and that is showing as the current version when I run java -version from the command line. Tried several methods to uninstall version 18 and reinstall the openjdk version 16 but all attempts have failed so far.
Open to any other suggestions I could try out here. Awesome blog James, even if I can’t get this to work, you’ve already provided hours of fun for myself and kids with our previous server world. 🙂
Hey Patrick,
It sounds like you’re trying to reuse an older version of SetupMinecraft.sh. I would recommend deleting that version and running the latest version from GitHub as it doesn’t have that requirement anymore.
The latest version should update itself and I’m guessing yours is right before that. Can you give that a try?
I used the
curl https://raw.githubusercontent.com/TheRemote/RaspberryPiMinecraft/master/SetupMinecraft.sh | bashcommand and ended up with the message:Correct OpenJDK (16) was not found in apt repositories and needs to be installed via snapd.Checking for snapd...
Installing OpenJDK....
"snap "core" already installed snap "openjdk" is already installed and snap "openjdk" has no updates available
then the OpenJDK failed in red text
sorry there were no quotation marks around the curl address
Hey Patrick,
That helps! I think I’ve found the problem. You’re definitely running it the right way. I’ve actually just updated and uploaded the fix on GitHub just now as there was still some lines in there that weren’t allowing OpenJDK 18 or higher.
This has been fixed. Can you give it another try here and see if that took care of this for you? To be clear go ahead and run SetupMinecraft.sh again now the same way you posted (it will use the updated file).
That has done the trick James! Thank you for getting back so quickly and for continuing to update this
Hey Patrick,
No problem at all! Thanks for stopping by to report it. I think this may be related to a couple of other people that I was helping investigate some issues and I’m sure there’s many more that didn’t post.
The reason this happened is that Paper didn’t used to support running OpenJDK versions above 17. I honestly thought I had removed this check completely already (maybe I did and there were still a few pieces left). Thanks again for reporting it!
Hi James,
Thank you so much for the scripts, I ran a 1.17 server on my Pi for a while with no issues! I just upgraded to 1.18.2 and for some reason, the automatic startup is giving me issues. When the Pi reboots automatically or manually it seems like the start.sh begins (the backup successfully completes) however the server does not actually start. When I run start.sh manually, it begins no problem. Any idea what might be happening? Thanks for your help and useful page!
Hey Jeff,
Can you try checking the logs in the logs folder and sudo systemctl status minecraft? It may be worth it to run SetupMinecraft.sh again one more time as this will recreate the service. We can definitely narrow it down from there!
Hi Jeff, there’s nothing in the logs folder unless I manually start the server, so I don’t think it’s relevant. Pasted below is the results of checking the status of minecraft service. Thanks again for your help!
● minecraft.service - Minecraft Server ServiceLoaded: loaded (/etc/systemd/system/minecraft.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-03-17 01:35:46 CDT; 6s ago
Process: 903 ExecStart=/bin/bash /home/pi/minecraft/start.sh (code=exited, status=0/SUCCESS)
Process: 961 ExecStop=/bin/bash /home/pi/minecraft/stop.sh (code=exited, status=1/FAILURE)
Mar 17 01:35:43 raspberrypi bash[903]: ./world/region/r.5.2.mca
Mar 17 01:35:44 raspberrypi bash[903]: ./world/region/r.3.-5.mca
Mar 17 01:35:44 raspberrypi bash[903]: ./world/session.lock
Mar 17 01:35:44 raspberrypi bash[903]: Starting Minecraft server. To view window type screen -r minecraft.
Mar 17 01:35:44 raspberrypi bash[903]: To minimize the window and let the server run in the background, press Ctrl+A then Ctrl+D
Mar 17 01:35:44 raspberrypi systemd[1]: Started Minecraft Server Service.
Mar 17 01:35:46 raspberrypi bash[961]: Unable to set path variable. You likely need an updated version of SetupMinecraft.sh from GitHub!
Mar 17 01:35:46 raspberrypi bash[961]: Server is not currently running!
Mar 17 01:35:46 raspberrypi systemd[1]: minecraft.service: Control process exited, code=exited, status=1/FAILURE
Mar 17 01:35:46 raspberrypi systemd[1]: minecraft.service: Failed with result 'exit-code'.
Hey Jeff,
Oh wow, that was not at all what I expected! No worries about there not being anything in the logs, you got exactly what I needed!
The first thing I want to check is that you should delete your copy of SetupMinecraft.sh and download the latest one from GitHub. I think you may be using an old enough version that it won’t autoupdate itself (the new version does this automatically now unless you enable the override switch at the top of the script).
If that isn’t it the other thing I’d like to see is the top of your start.sh script. This should have a PATH variable section where it sets the path variable to the ones that were active when you ran SetupMinecraft.sh.
Hopefully you just need the latest SetupMinecraft.sh from GitHub which will add that path variable in for you. If not then the top section of start.sh should reveal the issue. Hopefully that helps!
James.
Every time I try to “search” the extensive comments on this page, after entering 2-3 characters in the search box, I get a popup that wants me to sign in. Is this the expected behavior? I don’t see any way to create an account.
Ultimately, I was searching to see if you had any experience running “Multiverse” or similar mods on the RasPI. Do you know if they would overwhelm the server if I only have a couple of occasional users on a RPI 4 w/8GB?
Tom
Hey CardinalFang,
Sorry about that! I found the problem and got that fixed up here. I actually had to pay something like a $99 lifetime license fee for that system and it looks like it was broken the whole time. It works great too if it didn’t require an admin password (doh). It doesn’t require the page to be refreshed or anything and will search as you are typing past something like a few characters (which is when it was triggering the password prompt). It turns out this plugin calls one file from the wp-admin folder which I have protected by multiple passwords. I just needed to write a security rule to allow the comment search through.
This has honestly probably been hurting my traffic as it is impossible to find some of those buried comments without it and that’s definitely what some people come here for. This was caused by my much higher than normal security I have on the site. It’s attacked at least a few times per day and sometimes more. It’s probably my cryptocurrency content that makes me a target as it didn’t used to be this way before that (now they know I have cryptocurrency and want to take it). They’ve never got me yet though (and I keep an eye on them to make sure it stays that way) but I never want it to interfere with normal site operation. Thanks for letting me know so I could get that fixed!
So you should be absolutely fine with these mods and some people have done them here before. I don’t think you’ll have much trouble. People have done Bungeecord and even some other ones (you can probably find these now that the search is fixed too for a *lot* more information). I have a Docker container that lets Bedrock users connect to Java servers using Geyser / Floodgate here.
It depends on how deep into the multiverse you want to go but this shouldn’t be a problem for the Pi to handle as it’s not particularly CPU / storage / memory intensive to link these together. Some multiverse systems may be more efficient than others but it should be the hardest on your networking so if you have a high quality / fast network I don’t see it being much of a bottleneck problem unless we’re talking about a whole lot of people / actions going on.
Definitely let me know if you try it how it works out!
Hi James,
I only stumbled across your blog while doing some extra searches on performance in relation to me trying to document my own Pi setup experimentation – so I won’t forget how to, if my nephew asks again :). My experimentation after setting him up with a server on laptop moved onto me using vannila Minecraft, Raspian aarch64 version, MS Java 17.0.2 on a RPi4 2GB @2146Mhz (actively cooling) and using UHS-I sandisk micro-sd.
From finding your blog I am very appreciative of the information in this – and your previous version of this article – about papermc and the pre-generating of the worlds info.
From your excellent info I downloaded the jar for papermc and chunky to pre-generate my map – allowing me to compare paper to vanilla minecraft performance – with my config using draw-distance of 16 and simulation of 10 – my start flags don’t use nogui and I still run the vncserver on the Pi4, but still find my linux OS is still only showing about 250MB-300MB for the OS when running “top” in a terminal and checking the buff/cache memory usage at first boot.
Sadly, I haven’t got a spare sd card or portable ssd, or the option to image my working Pi3 or Pi4’s cards to free one up currently, to try your blogs amazingly cool solution just yet, so I was wondering if you could give any indication of what type of server performance you are seeing on paper’s gui: avg tick time? or what ticks per second is paper on the Pi4 giving you – after an initial 10mins of uptime with one user logged in and when logged out after 10mins?
vannila Minecraft sadly doesn’t track the 1min, 5min, 20min TPS info AFAIK, and doesn’t have the amazing “timings report” detail that I’ve only just discovered indirectly with paper from your blog, but the “/perf start” on my vannila config – after pre-generation and user logged in – now reports ~20.04 TPS and in normal activities even with 10 TNT explosions – but not flying, not teleporting, not using fill. And now my typical avg tick is around the 10-20ms, occasionally stressing at around the 35ms mark, and after logging out returns to 1-2ms, which is what it eventually reaches after the initial 5-10min server start up load – until “htop –tree” in a terminal is no longer showing 4 core utilisation, or my CPU-voltage-throttling script shows temperatures have dropped back to mid-40degs from mid-70degs. Which is a bit of a quandary because the idle and load tick ms numbers I get running paper – even with some small tweaks to paper.yml and bukkit.yml to lower tick frequency of items and using further tweaks to GC flags still seems to indicate vannila performs marginally better than paper – for me -with its lowest idle tick in ms of about 4.5ms (from initial 10-15ms after different GC start flags) – which I assume is wrong, and I’m leaving performance untapped in paper somehow, either not allocating enough memory with the xms/xmx flags using 1450M, or too big a swap file at 512MB or something.
From anecdotal observation, using chunky on paper and moving the map back to vannila seemed to eliminate performance inconsistency on my vannila setup. Watching “htop –tree” early on with default settings for paper’s startup, it seemed to stress the CPUs cores all the time, but at half load and somehow seemed to load the server and world faster than vannila.
Using my CPU monitoring script through startup it also seemed to keep the temperatures lower, which is interesting because on my Pi3, that isn’t actively cooled, vannila MC server isn’t reliable when I tried, because the initial startup load would frequently cause thermal throttling – dropping the core frequencies down to 800Mhz – when allocating 750MB to the server, and would then cause the server to exceed the “max-tick-time=60000” and shutdown automatically. So on passively cooled Pis paper seems more robust and maybe smoother at handling lag all round on any setup compared to vannila, so naturally I want to beat vannila on my RPI4 setup to justify the switch to paper. Any ideas of things to try would be most welcome.
Hey Paul,
Thanks for stopping by, I’m glad you found the site! This sounds like a great build and I’ll definitely fill in everything I can between my experience with it and other people’s reports posted here.
That is really interesting that you pregenerated the chunks in Paper and then moved it to vanilla! I’ve never heard of anyone doing this before but it makes perfect sense. The chunk generation hits the Pi in 3 places where it hurts: the memory, the storage, and the CPU. There isn’t any reason this shouldn’t work and it’s a great idea!
This script has been around for a long time now. It looks like the first build I released it on was 1.12 back in 2017. Back then most people were using Pis with 512MB to 1GB. It was pretty much required to get full TPS to pregenerate your chunks back then.
There have been several developments that have helped though. The biggest thing with the chunks is upgrading from a SD card to something like a SSD. This will make the largest difference of any upgrade you can make. The 2GB of RAM should be enough for a few players. It has been working with a few players on at full speed for quite a few years! The quality of your SD card can impact this quite a bit too (something like a SanDisk Extreme vs a generic SD card can be quite a bit, you can test them at pibenchmarks.com to see how fast they are exactly). It has been able to achieve a solid 20 TPS with a few players on for many years now (with fast storage).
It sounds like you have good CPU monitoring set up. Have you done any IO monitoring by chance? My guess is that this is where any slowness is. You could get a program like iotop or your OS may have iostat built in. You can do a:
iostat -dxmLinux 5.16.0-kali1-amd64 (kali) 03/06/2022 _x86_64_ (8 CPU)
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 0.40 0.01 0.30 42.90 0.66 17.88 0.54 0.00 0.23 30.07 1.05 7.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07
sr0 0.00 0.00 0.00 0.00 0.12 25.61 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
This is on my PC and you can see my drive utilization is extremely low (the last column). This is a very useful tool to run if you are having lag / TPS drops as if you find the utilization is at 100% then it is likely that the Minecraft server is actually just sitting there waiting for the storage to finish writing the new blocks. I do have a bunch of articles on here for setting up SSDs in case you are finding you want higher IO performance. I know you mentioned that you didn’t have one yet but I think you are on the right track here. Running some iostat measurements to get some measurements similar to your CPU and memory measurements should fill in the rest of the picture here.
For the 4 core utilization issue I definitely know what you’re talking about and have seen that. When you initially load the server it will be basically pegged for several minutes (it will literally say like 400% CPU usage in regular top, not sure about htop). It typically settles down after that and the number that pops in my head is it would still usually be over 100% in old top for the utilization but nowhere near 400% after the initial startup. It’s a pretty dramatic drop when it does and I’m not surprised you’re seeing thermal performance. The best I can tell this is mostly IO activity. The server is loading in a bunch of chunks and data (even when nobody is connected for the first few minutes). It has done this as long as I can remember and it actually used to be much worse. Back in those days with 512MB – 1GB at the most Pis it wasn’t too unusual to have to wait for this to stop before performance would stabilize (monsters would be warping around, blocks would have a break delay, etc then once it finished whatever it does during this time it would be perfect).
The vanilla server having higher performance in any way than Paper is almost shocking. The reason I say that is that Paper turns a *lot* of stuff off that ticks in normal vanilla Minecraft. All of the options (including a bunch that you mentioned) are basically additions to let you control those variables because it is the main source of “lag” or TPS drop in vanilla. I can’t think of many reasons this should be ever be the case but one of them would be plugins. This is just a hunch, but you may want to try disabling all the Paper plugins (including any chunk generation ones, just for a test) and see if that behavior goes away.
If it doesn’t then it would almost suggest to me that this is some kind of IO / storage bottleneck. I know Paper batches some things together (meaning it tries to update a bunch of things at once instead of sporadically here and there like vanilla) and theoretically if the storage is bottlenecking that could be resulting in some TPS drops that are more severe than vanilla. This definitely shouldn’t happen though as Paper doesn’t really “add” any weight other than the plugin system. It basically only tries to fix vanilla sources of lag if that makes sense.
The RAM sounds tight but that is to be expected on 2GB. It should be okay. It would be tough to run a version like 1.18 on 512MB anymore but even 1GB should work (but it will be very tight with the OS). Have you experienced any memory related crashes or had the Minecraft server terminated? This is typically what will happen when you allocate too much memory to the Minecraft process. Counter intuitively it should often be lowered. I didn’t see you mention any crashes so it sounds like you have the memory tuned okay. It may be worth doing an experiment and turning it down slightly more and see how that impacts your performance numbers (if things are really tight on the OS it may actually help).
My gut feeling (if possible) is that if you could pick up a cheap <$20 SSD and maybe the StarTech USB to SATA adapter (or even better if you already have a spare SSD you would just need the adapter) this could be a <$30-$35 upgrade even buying both brand new which puts it in the appropriate price range for the Pi (and even starting to approach the price of some higher capacity premium performance SD cards but for an order of magnitude of better performance). Fortunately though if you use something like iostat or iotop I think we can get your complete performance picture here even though vanilla doesn't have as fancy of a timings tool as Aikar's! It seems like there's a bottleneck here and given your thorough measurement of CPU and memory I think it will show up on the io tools. I think it should be performing a little bit better than it is from your description (but it sounds like it's really getting close) and I suspect this is your impression as well. The fact that vanilla is performing at least the same or maybe even slightly better tells us that there has to be some kind of bottleneck outside of your Paper or vanilla configuration. I think we can definitely find it and get the server performance to perfect! I made a site called Pi Benchmarks that you may find interesting. It’s benchmarks from all over the world specifically on Raspberry Pis. People test SSDs, eMMC, SD cards, anything you can connect to a Pi basically. This breaks down the stats and generates a “score” giving you a quick comparison on performance. An average SD card’s IO performance would score something like 700-1200 and an average SSD is more like 5000-8000 and even higher for NVMe drives. I made this originally specifically for this Minecraft script project as IO performance has been a huge issue since day 1. Like I was saying earlier the difference between a generic / cheap SD and a high performance A1 (application class) can literally be twice as fast in some cases. IO is always the most commonly missed / most poorly understood bottleneck (partially because you need different tools to measure it effectively and they vary more across distros than top/htop/etc.) and I made that site because it was even worse on the Pi and we used to have literally 0 way to know or compare how fast any of them were.
I’d love to hear what you find and can definitely help further if you wanted to bounce any ideas off me to see if I’ve tried them / others have tried them!
Thanks for the great reply. I probably should have elaborated more than “sandisk uhs-I” and fully said it was a SanDisk Extreme v30 U3 A1 32GB card in my Pi4(and 3) in my first post – which I think is fine, going by the raspian benchmark (rpdiag.txt) file I generated today, gave:
Test : SD Card Speed TestRun 1
prepare-file;0;0;38258;74
seq-write;0;0;37816;73
rand-4k-write;0;0;2951;737
rand-4k-read;11049;2762;0;0
Sequential write speed 37816 KB/sec (target 10000) - PASS
Random write speed 737 IOPS (target 500) - PASS
Random read speed 2762 IOPS (target 1500) - PASS
Test PASS
I took your good advice and installed iostat on my Pi4 and ran: “iostat -d -h 2” throughout the startup and stabilization time of both paper and vanilla on my server setup and after graphing one of them in scalc, the graph looked similar to the server monitoring graph of memory use, massive activity through setup and then stable with virtual nothing other than small megabyte or kilobyte blips. So it doesn’t appear that the I/O is having either config waiting.
I tried some different things like reducing the number of threads for netty-io down to 2 then 1, but saw no real difference to the idle avg tick time in ms – going no lower than 4ms – and watching htop it looks like paper never gets as much utilisation of all cores as vanilla – whether startup or when handling load it never sustains 4 full cores for more than second for me – but never fully settles either, unlike vanilla that will drop to almost nothing – on all 4 cores – when the server is fully stable after 10mins – under no load – when it has a constant tick time between 1.6ms and 2ms and has dropped to a low 40 deg temp, whereas when running paper, the Pi4 never really drops fully back below 45deg and stays there.
My gut feeling is that the vanilla performance number advantage at idle on my setup is a placebo effect that wouldn’t translate into a better experience than paper in actual use. Sadly, AFAIK there isn’t a free way to simulate a 2 to 4 player additional load on a personal server – without real players, which seems like a massive oversight for Majong and now Microsoft for such an important aspect of a game with self serving.
From the adjustments I tried I think the numbers now show paper to be faster, as it seems like the change in avg tick time with an active player versus idle is less (only 12ms per player if it scales linearly) with paper, where as it seems more like 15-20ms per player with vannila in my tests..
I’m now intrigued enough to move some hardware around to try out your setup script on a Micro-SD or a usb ssd with my Pi4, as I’m sure if you’ve had older server versions running on the 512MB, there’s more performance I missing.
If there is any aspect of my vanilla setup with an advantage to paper, I think it could be because I’m using a Java SDK (installed through sdkman) that Microsoft probably use and profile against vannila, but the paper team probably use the oracle openjdk which is maybe a better fit for it. I selected the Microsoft Java 17.0.2 release because it seemed to be smaller in size than openjdk and oracle, etcs efforts and seemed to give better performance numbers for me. Alternatively if the higher CPU utilisation I’m seeing is a cause, I wonder if there is heavy sorting taking place in the minecraft servers, as a sort algorithm with data supplied in a way that provided superior balancing might give better multithreading performance, and as paper’s adaption of vannila could upset such an algorithm I wonder if that is the performance in idle I can’t seem to get back.
To answer your other question, I did previously have situations where I was running out of memory and crashing the server program because I was experimenting with a ramfs – that needs everything manually accounted for. I was using it for combinations of the location for the world data, the java sdk and the jar file and the path using flag“-DbundlerRepoDir=” for the extracted MainClass, with a 4 or 8GB Pi I think all in ramfs would be really good. But I eventually lowered the vm.swappiness value in the “/etc/sysctl.conf” file, and now only keep the jar and extracted class in a ramfs – to eliminate storage IO as a bottleneck – it seems like linux gives back more of the buffer/cache memory (100MB) by swapping itself into the file with a very low swappiness value, rather than Minecraft’s time sensitive data, and seemingly stopped lag-crashing the server program after idles, where I assume it moved some of the server program data out to swap and then couldn’t access it all quick enough after idle to avoid having a tick longer than maximum allowed.
Thanks again, and will hopefully report back when I get a chance to try your setup script.
Hey Paul,
I think I can explain some of this behavior. Basically the Minecraft server is single “cored”. What people mean when they say this is that the bulk of the Minecraft server data processing is done in the same thread. It still uses a few threads (sockets use some threads for example) but the main thread that is “ticking” (thus TPS / ticks per second) is happening typically on one CPU core. If one thing is slowing it down everything has to wait basically. This means CPUs that have high single-core performance (vs the multiple performance benchmarks, you’ll usually see one of each for most CPUs) perform better with Minecraft and other similarly structured application.
This may look like it’s performing better and with lower CPU usage but it partially just means vanilla isn’t utilizing your other cores at all (and can’t). Paper can’t really do it that well either but it does spread things out a little bit over a few threads which is just about right for the Pis CPU (3-4 cores). I believe if you have multiple people online with vanilla you will “feel” that everything is even more heavily loaded in a single core and that people’s experience will be worse. You kind of alluded to this in your post that you suspected this may be the case and I think you are correct / thinking about it the right way. This doesn’t explain all of it though and there is in fact more going on here (it’s related to some of the optimizations Paper does).
Another important difference is that Paper actually grows plants and some other basic things when people are offline. Vanilla doesn’t do this generally and pretty much shuts everything off if nobody is online. You can measure this using Paper’s timings tool you mentioned earlier. Everything it does is documented in there including when nobody is online. Every procedure call and everything it’s doing as well as how much time it takes. I would imagine this would answer this question as to what Paper is doing. I covered how to do this in a really old post here.
It’s pretty much as simple as typing
timings onin the server console and then waiting a period of time and typingtimings report. This will give you a URL to paste into your browser and it will appear as an interactive report like my screenshots here.Check out the breakdown. This breaks down everything that happened during a server “tick”. You can sort the data a few different ways but it knows how many times each function was called, the name of the function and a sort of “stack trace” that makes it easy to see what part of the code it is and what it’s generally doing. I would definitely run this tool on there to determine your Paper tick usage and you can figure out what is ticking. The % of your overall tick is really useful. If you see things like entities using a huge % of your tick rate then if you reduce those entities in the .yml config files and run the test again you’ll see it’s using much less of a % of your server tick.
Most of this stuff can be disabled. If you wanted to match Paper to vanilla’s settings you can pretty much disable all of it. If you find that it’s the plants growing / that type of tile ticking going on you could disable that in the various .yml config files such as paper.yml, spigot.yml etc. and all of these settings can be viewed here.
Specifically for yours I bet your behavior is related to Paper Minecraft not unloading chunks like the starting area. It will keep those chunks loaded when nobody is online so that when people join there isn’t a delay as it normally has to load and fetch those chunks from the server’s storage (cold storage retrieval). Paper’s default settings keep it loaded since the spawn region is typically the most high traffic area of a lot of servers. You could probably greatly reduce the CPU usage with settings like
keep-spawn-loadedto false (this is for sure some of the behavior, your spawn isn’t unloaded on Paper without changing this setting and vanilla does nothing like this). Maybe setkeep-spawn-loaded-rangeto 0 for good measure.That should explain most of this as those chunks are still live on Paper when no one is online. Even the monsters aren’t unloaded when this setting is active (they do stop moving at least if nobody is close by but many types won’t despawn). The spawn region is basically protected from being unloaded even if nobody is playing whereas vanilla destroys everything.
There may be more of these as well you can disable. You can basically start the timings tool on Paper and have it record everything that is happening. You can just walk away and have a coffee or do something else and let it slowly benchmark and count everything it’s doing. When you stop the tool and generate the report you can pretty much Ctrl+F the Paper.yml page I linked earlier to find settings related to it. You should be able to shut this off to get more comparable behavior for the tests. This may help explain some of these performance differences (beyond just the keep-spawn-loaded setting) for sure!
Unfortunately as we’ve lamented before vanilla doesn’t have the timings tool but we at least do have 50% of the information and I believe you’ll be able to probably get things pretty close if not better with some additional configuration in the yml files. Fortunately in this case Paper is also the one we need this data for as we already know what vanilla is doing when everyone is offline (nothing). We need to know what Paper is doing to explain this and by using the timings tool like I described in the post linked above (it’s old, for 1.13 but it hasn’t changed at all to use the tool) you will end up with a complete stack trace of every single instruction the server executed while running the tool. It will answer these lingering questions that are not really showing up well in the standard toolset (not uncommon with something as complex as the Minecraft server is, it just means we need better / more specific tools to get the data we need to explain the discrepancies).
Your memory configuration sounds excellent. You already went through and tuned this appropriately. That SD card is excellent as well, I have a couple of those SD card and they score very well. Everything is looking good. I’m curious if you can narrow down some flags using the Timings tool that can explain the rest (if any) discrepancies between the two. With any luck the
keep-spawn-loadedbehavior will explain 90%+ of this and simply disabling it will drop the temps back down to match vanilla. Hopefully that helps!Great reply, James!
That’s the info I was missing from my newbie knowledge of actually playing MC, so not knowing the main virtues of papermc, like its world persistence for ticking crops etc – beyond plug-ins, and performance that I was struggling to prove to myself, numerically – and me just being the IT person getting a laptop server functional for my nephew, with no background in MC serving specifics or awareness of bukkit/spigot/paper then, and then hobby-ing with a RPi4 replacement as a potential solution, or future server for my kids.
Changing the “keep-spawn-loaded” to “false” as you suggested improved the synthetic(& real) performance numbers massively. Producing a significantly lower avg time in ms per tick, and stabilized as low as 0.2ms, 5-10x less time than 1-2ms of my attempts to optimise Vanilla – even with Vanilla not ticking crops.
The server also has a lower avg tick time most of the time, but the range being much bigger tells me I need to either revert the setting and take the inferior but stable numbers, or look again at maybe reducing memory use in the start flags – as you’ve suggested the server can work well with the pi3’s lesser gigabyte too – and look again at the ramfs, for the java sdk, world files, maybe even revert to a tiny 100MB swapfile too, and then alter some other settings in my yml files to get the better performance numbers along with more stable numbers.
Changing the “chunk-tasks-per-tick” down from 1000 to 500 helped a bit, but one of the problems seems to be the Salmon tick-time for me – using more resources than zombies on lag ticks – but being unable to see a setting that matches in my yml files, and my lack of papermc knowledge means I’m not sure how to handle that best, or the occasional Server.Oversleep adding 40ms when “keep-spawn-loaded“ is set to “false”.
I suspect on balance “keep-spawn-loaded” set to “true” is going to be the better option, but I am still very appreciative that you were able to point me in the right direction to validate that paper is vastly more efficient than Vanilla on my setup. With “keep-spawn-loaded“ set to “false” for the time being on my setup running “/perf start” on a papermc server (not needed obviously with timgsV2 being the definitive tool) now works without throwing errors.
I’ve also freed up a card with a new Bulleyes install, so my next task is to try your script out which I assume has custom yml settings you feel work best, yes?
Hey Paul,
No problem at all, all of those numbers sound like everything is really coming together!
I’ve also ran into the entities tick issue. It’s usually passive animals that are the most offensive. You have a few different settings that can improve / reduce the amount of TPS being burned on this. One set of settings is the despawn range for these creators. If you lower the range it will despawn them more aggressively (without the player wandering as far away):
despawn-rangesmonster:
soft
default: 32
description: The number of blocks away from a player in which monsters will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which monsters will be forcibly despawned.
creature:
soft
default: 32
description: The number of blocks away from a player in which creatures will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which creatures will be forcibly despawned.
ambient:
soft
default: 32
description: The number of blocks away from a player in which ambient entities will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which ambient entities will be forcibly despawned.
underground_water_creature:
soft
default: 32
description: The number of blocks away from a player in which underground water creatures will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which underground water creatures will be forcibly despawned.
water_creature:
soft
default: 32
description: The number of blocks away from a player in which water creatures will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which water creatures will be forcibly despawned.
water_ambient:
soft
default: 32
description: The number of blocks away from a player in which water ambient entities will be randomly selected to be despawned.
hard
default: 64
description: The number of blocks away from a player in which water ambient entities will be forcibly despawned.
misc:
soft
default: 32
description: The number of blocks away from a player in which misc entities will be randomly selected to be despawned.
hard
default: 128
description: The number of blocks away from a player in which misc entities will be forcibly despawned.
Basically you can see here they have different categories. The Salmon should hopefully be in the underground waters section, but be advised that sometimes the monsters are classified in really odd ways. If you encounter this you may need to look up the entity ID by doing some Google queries (usually will take you to the minecraft wiki or something like that) and can figure out which category some of the stranger ones are in.
It had been years since I looked at them but I double checked which flags my script applied automatically. They are related to this conversation and may be of some assistance while you’re tuning / testing / planning:
# Configure paper.yml optionsif [ -f "paper.yml" ]; then
# early-warning-delay, early-warning-every
# Disables constant error spam of chunk unloading warnings
sed -i "s/early-warning-delay: 10000/early-warning-delay: 120000/g" paper.yml
sed -i "s/early-warning-every: 5000/early-warning-every: 60000/g" paper.yml
# optimize-explosions
# Paper applies a custom and far more efficient algorithm for explosions. It has no impact on gameplay.
sed -i "s/optimize-explosions: false/optimize-explosions: true/g" paper.yml
# mob-spawner-tick-rate
# This is the delay (in ticks) before an activated spawner attempts to spawn mobs. Doubling the rate to 2 should have no impact on spawn rates.
# Only go higher if you have severe load from ticking spawners. Keep below 10.
sed -i "s/mob-spawner-tick-rate: 1/mob-spawner-tick-rate: 3/g" paper.yml
# container-update-tick-rate
# This changes how often your containers/inventories are refreshed while open. Do not go higher than 3.
sed -i "s/container-update-tick-rate: 1/container-update-tick-rate: 2/g" paper.yml
# max-entity-collisions
# Crammed entities (grinders, farms, etc.) will collide less and consume less TPS in the process.
sed -i "s/max-entity-collisions: 8/max-entity-collisions: 2/g" paper.yml
# fire-physics-event-for-redstone
# This stops active redstone from firing BlockPhysicsEvent and can salvage some TPS from a cosmetic task.
# Note: If you have a rare plugin that listens to BlockPhysicsEvent, leave this on.
sed -i "s/fire-physics-event-for-redstone: true/fire-physics-event-for-redstone: false/g" paper.yml
# use-faster-eigencraft-redstone
# This setting eliminates redundant redstone updates by as much as 95% without breaking vanilla mechanics/devices (pretty sure). Empirical testing shows a speedup by as much as 10x!
sed -i "s/use-faster-eigencraft-redstone: false/use-faster-eigencraft-redstone: true/g" paper.yml
# grass-spread-tick
# The time (in ticks) before the server attempts to spread grass in loaded chunks. This will have minimal gameplay impact on most game types.
sed -i "s/grass-spread-tick-rate: 1/grass-spread-tick-rate: 3/g" paper.yml
# despawn-ranges
# Soft = The distance (in blocks) from a player where mobs will be periodically removed.
# Hard = Distance where mobs will be removed instantly.
sed -i "s/soft: 32/soft: 28/g" paper.yml
sed -i "s/hard: 128/hard: 96/g" paper.yml
# hopper.disable-move-event
# This will significantly reduce hopper lag by preventing InventoryMoveItemEvent being called for EVERY slot in a container.
# Warning: If you have a plugin that listens to InventoryMoveItemEvent, do not set true.
sed -i "s/hopper.disable-move-event: false/hopper.disable-move-event: true/g" paper.yml
# non-player-arrow-despawn-rate, creative-arrow-despawn-rate
# Similar to arrow-despawn-rate in Spigot, but targets skeleton arrows. Since players cannot retrieve mob-fired arrows, this setting is only a cosmetic change.
sed -i "s/creative-arrow-despawn-rate: -1/creative-arrow-despawn-rate: 60/g" paper.yml
sed -i "s/non-player-arrow-despawn-rate: -1/non-player-arrow-despawn-rate: 60/g" paper.yml
# prevent-moving-into-unloaded-chunks
# Prevents players from entering an unloaded chunk (due to lag), which causes more TPS loss. The true setting will rubberband them back to a "safe" area.
# Note: If you did not pregenerate your world (what's wrong with you?!), this setting might be a godsend.
sed -i "s/prevent-moving-into-unloaded-chunks: false/prevent-moving-into-unloaded-chunks: true/g" paper.yml
# disable-chest-cat-detection
# By default, chests scan for a cat/ocelot on top of it when opened. While this eliminates a vanilla mechanic (cats block chest opening), do you really need this silly mechanic?
sed -i "s/disable-chest-cat-detection: false/disable-chest-cat-detection: true/g" paper.yml
# bungee-online-mode
# disable Bungee online mode
sed -i "s/bungee-online-mode: true/bungee-online-mode: false/g" paper.yml
# keep-spawn-loaded, keep-spawn-loaded-range
# This causes the nether and the end to be ticked and save so we are going to disable it
# This setting makes sense on high player count servers but for the Pi it just wastes resources
sed -i "s/keep-spawn-loaded: true/keep-spawn-loaded: false/g" paper.yml
sed -i "s/keep-spawn-loaded-range: 10/keep-spawn-loaded-range: -1/g" paper.yml
fi
# Configure bukkit.yml options
if [ -f "bukkit.yml" ]; then
# monster-spawns
# This dictates how often (in ticks) the server will attempt to spawn a monster in a legal location. Doubling the time between attempts helps performance without hurting spawn rates.
sed -i "s/monster-spawns: 1/monster-spawns: 2/g" bukkit.yml
# autosave
# This enables Bukkit's world saving function and how often it runs (in ticks). It should be 6000 (5 minutes) by default.
# This is causing 10 second lag spikes in 1.14 so we are going to increase it to 18000 (15 minutes).
sed -i "s/autosave: 6000/autosave: 18000/g" bukkit.yml
# warn-on-overload
# Disables annoying server is overloaded messages
sed -i "s/warn-on-overload: true/warn-on-overload: false/g" bukkit.yml
fi
# Configure spigot.yml options
if [ -f "spigot.yml" ]; then
# Merging items has a huge impact on tick consumption for ground items. Higher values allow more items to be swept into piles and allow you to avoid plugins like ClearLag.
# Note: Merging items will lead to the occasional illusion of items disappearing as they merge together a few blocks away. A minor annoyance.
sed -i "s/exp: 3.0/exp: 6.0/g" spigot.yml
sed -i "s/item: 2.5/item: 4.0/g" spigot.yml
# max-entity-collisions
# Crammed entities (grinders, farms, etc.) will collide less and consume less TPS in the process.
sed -i "s/max-entity-collisions: 8/max-entity-collisions: 2/g" spigot.yml
# mob-spawn-range
# Crammed entities (grinders, farms, etc.) will collide less and consume less TPS in the process.
sed -i "s/mob-spawn-range: 8/mob-spawn-range: 6/g" spigot.yml
# entity-activation-range:
sed -i -z "s/entity-activation-range:\n animals: 32\n monsters: 32\n raiders: 48\n misc: 16\n tick-inactive-villagers: true/entity-activation-range:\n animals: 24\n monsters: 24\n raiders: 48\n misc: 12\n tick-inactive-villagers: false/g" spigot.yml
fi
You can see a lot of entities related settings in here. Most of them are nibbling around the edges of things that “tug” at your TPS. Notice that I slightly tweak a bunch of settings related to them but mostly not in a very dramatic way. It turns out my script does disable the spawn loading as we can see here (I even made a note I forgot about, this setting also causes The Nether and The End to also tick all of the time when they normally don’t without this setting unless someone is in there). Undoubtedly you are going on a very similar journey that I did years ago to discover which flags give you the best performance! These are all settings that caused me (or the people using the script) a problem that I had to usually locate through the timings and find the setting to alleviate the issue.
You may be able to cheat and use mine but they’ve added some new configuration flags over the past few years that mine likely isn’t using. There’s probably some additional performance that can be gained here and I definitely don’t insist it’s the perfectly correct set of flags. These were chosen as an inoffensive and pretty close to vanilla experience while greatly improving performance on the Pi. Some of them are related to freeing up the storage for example which is already slow on the Pi. I change the autosave interval from something like 5 minutes to 15 minutes.
The scripts are meant to be easily modified. You could actually just put your changes right in start.sh if you follow my lead and replace them right in the .yml files. You could also just edit the files as it’s mostly to automate it that I did that but it is nice to have a comments area for each configuration change so you can document what you did (otherwise the next time you spin up a server you’ll likely forget a few of these settings). That’s one way to stay organized at least! I literally wouldn’t have remembered a single flag or why I did it if I didn’t document and comment it for myself (you think you will but outside of a few memorable ones like the keep-spawn-loaded one that I remembered off the top of my head the minor ones I wouldn’t have been able to to save my life).
I’m curious what your experience will be when you test the script! As I was saying earlier you may be able to find a few surprises in there that will yield additional gains. This is especially likely for flags that have only been added in the past couple of years. If you do let me know and I may modify the default flags if it’s clearly better performance without any drastic quality of life reduction in game.
I typically play “vanilla survival” mode with my friends. Obviously it’s not 100% pure vanilla but the settings are meant to be as close as possible. View distance 10 is maintained for example (a common setting to lower that drastically reduces quality of life / game experience) and the entities while having a few optimizations around the edges aren’t reduced by 90% or anything drastic that some people have attempted before. The settings are inspired by this goal though and therefore may not yield maximum performance in some cases where I made the tradeoff to keep the experience as close to vanilla as possible. Different people may have different goals though and want that extra performance depending on what their server needs.
You’re getting to the “tradeoff” point but there still are probably a few “low-hanging fruit” in the settings file before you start to have to chop away at the qualify of life on the game. As you pointed out it is a tradeoff to disable keeping the spawn loaded. Especially when your server is starting out people are going to feel it if they are the first one logging on and they’re the one that has to wait for the spawn to load. For the Pi because of the CPU usage (especially years ago when there was a lot less CPU to go around) that tradeoff didn’t make sense to me as the default setting but it undoubtedly will be better for some servers (no question).
Hopefully those help as well and definitely let us know how it turns out / what flags (if any) improve the experience further!
Thanks for the in-depth response.
I got your script installed on my other card using my RPI4 2GB using the raspian bulleyes aarch64 os and can see why you’ve had lots of people using your setup successfully over the years. It seems to work really well by default with very little user input, other than needing to pick wisely with the memory allocation.
Even without the other optimisations I use, like the swappiness change and use of the ramfs for staging files/symlinks, etc. It is a day and night comparison to just running Vannila as is on an overclocked RPi4 from the performance I was noting using my testworld from my other card seup.
After lots of timings with different OS tweaks from my other setup and trying with and without a GUI – using your scripts “switch” console setup – and various java garbage collection flags, with different memory -Xmx allocations, numerically the idling wasn’t as low as my other setup and edge cases where lag really hits, it seemed higher – but by default I think your script still uses “keep-spawn-loaded: true” and the range for the loaded area set to 10 chunks IIRC, which made sense, so I was largely seeing better normal server performance with your paper.yml settings, but better idling and peak lag handling with my paper.yml. Using screen gave back more memory, but didn’t improve the performance on TimingsV2, versus using the Raspian desktop remotely with papermc’s GUI through realvncserver, so I can only assume the ssh and screen programs consume more CPU time on the main core – by redirecting – than GPU accelerated GUI with the java awt gui for the paper server do.
Eventually I downloaded diffuse(a Gui diff) and sorting the two files so your flags order matched the ordering of my recently generated paper.yml – and merged in any missing flags from each other that are still used, and use a copy of the variables with the default value behind a # # in the paper.yml file, for any variable changed from normal – except for the long list of soft:hard:, where I just note it at the top of the section once, as their defaults are all the same.
Some of the entries in the file are left as is, but have a copy with “# #” at the start,because these were entries I was changing at some point, but gave inconclusive results.
This is a link to my last timings run without the initial server startup in the timings. The server.properties file uses a “view-distance=16”,and “network-compression-threshold=512” too.
I this is as good as I’ll get other than me getting a RPI400 or 8GB Pi4 with more ramfs options for the world data, and/or maybe using an SSD to gain a little extra performance from the OS itself, hobbling the game features too much.
Aikar Timings Results
And this is the java flags I use, I think 768k being better IMHO than 1MB jvm would override a bigger value with – and better than 512K I tried – because half of the 768K is a better alignment to the CPU caches on the ARM chip IIRC and seems to yield better performance. I also allocate a large 2048MB swapfile, so the OS doesn’t react to the server consuming all the memory at once and can hopefully not crash the RPI4 if the paper server crashes. I also change the vm.swappiness value down to 5, from the default of 60 in the OS
980M seems to be the maximum I could allocated with a large 200MB world, without more than 30MB of swap being used – working on the basis that storage on a RPi4 has a read limit probably around 30MB/s, so 40ticks stall at most for any minor swapping.
java -Xmx980M -Xms980M -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=5 -XX:G1MaxNewSizePercent=5 -XX:G1HeapRegionSize=768K -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=1 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=10 -XX:SurvivorRatio=10 -DbundlerRepoDir=/media/pi/ramdrive -DbundlerMainClass=org.bukkit.craftbukkit.Main -jar ~/Minecraft_files/minecraft/paperclip.jar --server-name RP4B --universe /media/pi/ramdrive/And here is the paper.yml file settings with all your improvements, and one or two adjustments I made
# #max-joins-per-tick: 3max-joins-per-tick: 1
track-plugin-scoreboards: false
fix-entity-position-desync: true
use-display-name-in-quit-message: false
load-permissions-yml-before-plugins: true
region-file-cache-size: 256
enable-player-collisions: true
save-empty-scoreboard-teams: false
bungee-online-mode: false
incoming-packet-spam-threshold: 300
use-alternative-luck-formula: false
velocity-support:
enabled: false
online-mode: false
secret: ''
console-has-all-permissions: false
player-auto-save-rate: -1
# #player-auto-save-rate: 60000
# #max-player-auto-save-per-tick: -1
max-player-auto-save-per-tick: 1
fix-target-selector-tag-completion: true
lag-compensate-block-breaking: true
send-full-pos-for-hard-colliding-entities: true
time-command-affects-all-worlds: false
use-dimension-type-for-custom-spawners: false
log-player-ip-addresses: true
console:
enable-brigadier-highlighting: true
enable-brigadier-completions: true
suggest-player-names-when-null-tab-completions: true
watchdog:
early-warning-every: 5000
early-warning-delay: 10000
spam-limiter:
tab-spam-increment: 1
tab-spam-limit: 500
recipe-spam-increment: 1
recipe-spam-limit: 20
book-size:
page-max: 2560
total-multiplier: 0.98
loggers:
deobfuscate-stacktraces: true
item-validation:
display-name: 8192
loc-name: 8192
lore-line: 8192
book:
title: 8192
author: 8192
page: 16384
chunk-loading:
global-max-concurrent-loads: 500.0
min-load-radius: 1
# #min-load-radius: 2
# #max-concurrent-sends: 2
max-concurrent-sends: 32
autoconfig-send-distance: true
target-player-chunk-send-rate: 100.0
# #target-player-chunk-send-rate: 6.0
global-max-chunk-send-rate: -1.0
# #global-max-chunk-send-rate: 50.0
# #enable-frustum-priority: false
enable-frustum-priority: true
# #global-max-chunk-load-rate: -1.0
global-max-chunk-load-rate: = -1.0
# #player-max-concurrent-loads: 20.0
player-max-concurrent-loads: 20.0
# #global-max-concurrent-loads: 20.0
async-chunks:
threads: -1
# #threads: -1
unsupported-settings:
allow-permanent-block-break-exploits: false
allow-piston-duplication: false
perform-username-validation: true
allow-headless-pistons: false
allow-permanent-block-break-exploits-readme: This setting controls if players
should be able to break bedrock, end portals and other intended to be permanent
blocks.
allow-piston-duplication-readme: This setting controls if player should be able
to use TNT duplication, but this also allows duplicating carpet, rails and potentially
other items
allow-headless-pistons-readme: This setting controls if players should be able
to create headless pistons.
packet-limiter:
kick-message: '&cSent too many packets'
limits:
all:
interval: 7.0
max-packet-rate: 500.0
PacketPlayInAutoRecipe:
interval: 4.0
max-packet-rate: 5.0
action: DROP
world-settings:
default:
prevent-moving-into-unloaded-chunks: false
disable-teleportation-suffocation-check: false
generator-settings:
flat-bedrock: false
# #chunk-tasks-per-tick: 1000
delay-chunk-unloads-by: 10s
# #delay-chunk-unloads-by: 10s
piglins-guard-chests: true
should-remove-dragon: false
max-auto-save-chunks-per-tick: 8
# #max-auto-save-chunks-per-tick: 24
baby-zombie-movement-modifier: 0.5
optimize-explosions: true
use-vanilla-world-scoreboard-name-coloring: false
game-mechanics:
scan-for-legacy-ender-dragon: true
fix-curing-zombie-villager-discount-exploit: true
disable-pillager-patrols: false
pillager-patrols:
spawn-chance: 0.2
spawn-delay:
per-player: false
ticks: 12000
start:
per-player: false
day: 5
disable-chest-cat-detection: true
nerf-pigmen-from-nether-portals: false
disable-end-credits: false
disable-player-crits: false
disable-sprint-interruption-on-attack: false
shield-blocking-delay: 5
disable-unloaded-chunk-enderpearl-exploit: true
disable-relative-projectile-velocity: false
disable-mob-spawner-spawn-egg-transformation: false
# #prevent-moving-into-unloaded-chunks: false
prevent-moving-into-unloaded-chunks: true
count-all-mobs-for-spawning: false
spawn-limits:
monster: -1
creature: -1
ambient: -1
axolotls: -1
underground_water_creature: -1
water_creature: -1
water_ambient: -1
experience-merge-max-value: -1
allow-using-signs-inside-spawn-protection: false
wandering-trader:
spawn-minute-length: 1200
spawn-day-length: 24000
spawn-chance-failure-increment: 25
spawn-chance-min: 25
spawn-chance-max: 75
door-breaking-difficulty:
zombie:
- HARD
vindicator:
- NORMAL
- HARD
max-growth-height:
cactus: 3
reeds: 3
bamboo:
max: 16
min: 11
despawn-ranges:
# # defaults #soft: 32
# # defaults #hard: 128
monster:
soft: 28
hard: 96
creature:
soft: 28
hard: 96
ambient:
soft: 28
hard: 96
axolotls:
soft: 28
hard: 96
underground_water_creature:
soft: 28
hard: 96
water_creature:
soft: 28
hard: 96
water_ambient:
soft: 28
hard: 64
misc:
soft: 28
hard: 96
ender-dragons-death-always-places-dragon-egg: false
falling-block-height-nerf: 0
tnt-entity-height-nerf: 0
slime-spawn-height:
swamp-biome:
maximum: 70.0
minimum: 50.0
slime-chunk:
maximum: 40.0
frosted-ice:
enabled: true
delay:
min: 20
max: 40
lootables:
auto-replenish: false
restrict-player-reloot: true
reset-seed-on-fill: true
max-refills: -1
refresh-min: 12h
refresh-max: 2d
filter-nbt-data-from-spawn-eggs-and-related: true
max-entity-collisions: 8
# #max-entity-collisions: 8
disable-creeper-lingering-effect: false
duplicate-uuid-resolver: saferegen
duplicate-uuid-saferegen-delete-range: 32
hopper:
cooldown-when-full: true
disable-move-event: false
ignore-occluding-blocks: false
mob-effects:
undead-immune-to-certain-effects: true
spiders-immune-to-poison-effect: true
immune-to-wither-effect:
wither: true
wither-skeleton: true
# #update-pathfinding-on-block-update: true
update-pathfinding-on-block-update: true
phantoms-do-not-spawn-on-creative-players: true
phantoms-only-attack-insomniacs: true
mobs-can-always-pick-up-loot:
zombies: false
skeletons: false
map-item-frame-cursor-update-interval: 10
allow-player-cramming-damage: false
anticheat:
obfuscation:
items:
hide-itemmeta: false
hide-durability: false
monster-spawn-max-light-level: -1
light-queue-size: 20
auto-save-interval: -1
# #auto-save-interval: -1
mob-spawner-tick-rate: 1
# #mob-spawner-tick-rate: 3
zombies-target-turtle-eggs: true
all-chunks-are-slime-chunks: false
container-update-tick-rate: 1
water-over-lava-flow-speed: 5
grass-spread-tick-rate: 1
use-faster-eigencraft-redstone: true
# #use-faster-eigencraft-redstone: false
armor-stands-do-collision-entity-lookups: true
keep-spawn-loaded: true
# #keep-spawn-loaded: false
anti-xray:
enabled: false
engine-mode: 1
max-block-height: 64
update-radius: 2
lava-obscures: false
use-permission: false
hidden-blocks:
- copper_ore
- deepslate_copper_ore
- gold_ore
- deepslate_gold_ore
- iron_ore
- deepslate_iron_ore
- coal_ore
- deepslate_coal_ore
- lapis_ore
- deepslate_lapis_ore
- mossy_cobblestone
- obsidian
- chest
- diamond_ore
- deepslate_diamond_ore
- redstone_ore
- deepslate_redstone_ore
- clay
- emerald_ore
- deepslate_emerald_ore
- ender_chest
replacement-blocks:
- stone
- oak_planks
- deepslate
disable-thunder: false
skeleton-horse-thunder-spawn-chance: 0.01
disable-ice-and-snow: false
# #keep-spawn-loaded-range: 10
keep-spawn-loaded-range: 6
nether-ceiling-void-damage-height: 0
only-players-collide: false
allow-vehicle-collisions: true
allow-non-player-entities-on-scoreboards: false
portal-search-radius: 128
portal-create-radius: 16
portal-search-vanilla-dimension-scaling: true
fix-items-merging-through-walls: false
parrots-are-unaffected-by-player-movement: false
disable-explosion-knockback: false
fix-climbing-bypassing-cramming-rule: false
fixed-chunk-inhabited-time: -1
remove-corrupt-tile-entities: false
prevent-tnt-from-moving-in-water: false
show-sign-click-command-failure-msgs-to-player: false
iron-golems-can-spawn-in-air: false
max-leash-distance: 10.0
# # #generator-settings:
# # #flat-bedrock: false
per-player-mob-spawns: true
map-item-frame-cursor-limit: 128
armor-stands-tick: true
non-player-arrow-despawn-rate: 60
# #non-player-arrow-despawn-rate: -1
creative-arrow-despawn-rate: 60
# #creative-arrow-despawn-rate: -1
spawner-nerfed-mobs-should-jump: false
entities-target-with-follow-range: false
wateranimal-spawn-height:
maximum: default
minimum: default
zombie-villager-infection-chance: -1.0
unsupported-settings:
fix-invulnerable-end-crystal-exploit: true
enable-treasure-maps: true
treasure-maps-return-already-discovered: false
split-overstacked-loot: true
fishing-time-range:
MinimumTicks: 100
MaximumTicks: 600
entity-per-chunk-save-limit:
experience_orb: -1
snowball: -1
ender_pearl: -1
arrow: -1
fireball: -1
small_fireball: -1
alt-item-despawn-rate:
enabled: false
items:
COBBLESTONE: 300
tick-rates:
sensor:
villager:
secondarypoisensor: 40
behavior:
villager:
validatenearbypoi: -1
feature-seeds:
generate-random-seeds-for-all: false
Hey Paul,
Everything is looking good! My guess from your timings is you would get a pretty decent boost from a SSD. Your highest scores when you choose the “Lag” option (vs the All option) is all chunk saving / storage. There isn’t a ton you can do with paper.yml flags to alleviate this except for save and tick the chunks less but in reality you are hitting both some SD card limitations and some Pi 3 limitations.
I have a few suggestions for you since you are on a Pi 3. You are able to overclock the SD card port from 50MHz to 100Mhz by adding:
dtparam=sd_overclock=100to your config.txt. You may want to try this and repeat my storage benchmark as it does increase the throughput (it won’t be a 100% increase but it’s noticeable, maybe more like 10-15%). My storage benchmark checks for this setting and measures it.
You can still have some substantial storage/IO gains on a Pi 3. Check out these scores people submitted for Pi 3 here: Pi3
Your upper end is probably 2500-3000 as far as a raw performance score on the Pi 3 with a SSD but that’s still easily 2-3x faster IO performance. This is especially true on random read/writes (as opposed to sequential read/writes) which is what the Minecraft server uses. The highest for the SD card you have is something like 1000-1200 I believe. There’s actually a Pi 3 score on there of exactly 1000 for that card. Notice that even $20 SSDs (literally, well $23 at the moment) like the Kingston A400 are doing 2500. You can see the exact IO stats by clicking into those benchmarks (MB/s, IOPS, all of it).
Bumping up the core speed and CPU speed wouldn’t hurt on the Pi 3 either. Some of those benchmarks submitted will have overclocks. You don’t want to overdo it but you could do what we call a “safe” or “conservative” overclock on it in config.txt without any risk. You have to set a special setting (force_turbo, voids the warranty) to be able to do too much damage to it.
The SD overclock and the core speed are the most likely to reduce those chunk save times a little bit followed by the CPU at a close second. Check out the RetroPie settings examples for some safe ones I’ve used before here (Ctrl+F then “Settings Examples”). This includes a safe RAM overclock (sdram_schmoo=0x02000020) as well which is huge for the Pi 3.
I wish I had a Pi 3 setup (I’d have to dig it out, I have it somewhere but it would be fresh imaging and the works) that I could show you how big of a difference it makes to have a SSD in there on those ticks. You’ve pretty much recreated my entire journey of creating this script except the SSD part. That was when I finally reached the performance level I wanted (view distance 10, as close to vanilla settings as possible for multiplayer survival). I’ve hosted it on application class SD cards with a few friends before successfully but you can definitely feel the difference. The players online can feel the difference between a SD card and a SSD (during world saves you won’t go over 100% on your tick like that with a SSD). It’s that noticeable for sure!
Because of the single “core” nature of the Minecraft server we talked about it really can’t do anything else when it’s waiting for the storage to read / write these chunks to / from disk via random IO operations. Most of this time you are losing it’s just waiting for the storage device. It’s not because the Pi is too busy and can’t keep up (that looks very different in the timings, yours look fantastic other than the expensive saving / reading IO operations involving chunks). If the Minecraft server was designed to split things up into separate threads / processes like many modern servers are this wouldn’t be the case and it could be doing other things at the same time. Unfortunately with how the Java version was built and designed basically it can’t do anything else. Everything stops until that chunk is saved / read so people will notice blocks won’t break for a few seconds or however long it takes etc.
I think I agree with your assessment that it’s as good as it’s getting without a SSD with the exceptions that I mentioned here of you should give overclocking the SD card and probably even the core/CPU a try. The reason this can make a difference is you’re a generation back. It closes the gap in performance anywhere from a little bit to a moderate amount and honestly yours is so close that a little bit is all you need. Your TPS is completely under control other than the chunk saving which according to the “Lag” chart did spike over 100% during some of those saves (people will feel this, that’s a lag spike).
The RAM in the Pi 4 is much faster so you can’t catch up there but on core and even CPU it’s not that far behind for sure. This is more “nibbling around the edges” and all of that nibbling combined resulted in a smooth playable experience even on the Pi 3! I won’t say that the Pi 3 is indistinguishable from a PC hosting it (yours is going to be close I can already tell) but I would say that the Pi 4 on a SSD is.
Hopefully that helps and let us know how it turns out!
Oh. sorry I think my aggressive reduction in Xmx value – to below 1GB and mentioning me taking the card from my RPi3, has made you think it wasn’t the 2GB RPi4 I ‘ve been using when it was.
Between 980MB for the server, another 200MB-ish in the ramfs, 250MB-ish overhead for Java’s JVM, and another 300-350MB for the linux OS buffer/cache and needing to leave a small margin free of another 100MB or so for the buffer/cache to extend you actually get back to the 1900MB available.
As for the lag numbers, I was under the impression from a video I watch on timingsV2 that it is near impossible to eliminate the lag ticks entirely, but the importance was to keep the avg TPS at 20 and have the lag frames with a combination of minimal over-spend – by time taking more than 50ms – and to have a low percentage of them – and spaced out – over the range.
I was surprised you hadn’t interpreted the timings the way I had – as being improved for a RPi4 – and so I ran your excellent script freshly, again with the same world data and workload as I’d done for my own setup and did seem to get better performance, than my previous runs, and then from that, I reconfigured my own setup to use screen -dmS minecraft with the nogui option, and saw better performance in both setups, so I can only assume either something in the overnight update for Raspian aarch64 bulleyes fixed a bug in screen or ramfs, or my laziness to duplicate the world data, rather than share them with aliased symbolic links between setups, could be a discrepancy in an area of the micro-sd card that hampered my first timings comparison.
After doing a new comparison, and looking at the timings, both setups allocating 1240MB seemed very similar, and only on trying both with the view-distance – in the server.properties file – set to 16, did I then notice that the line trends, although very similar actually were resulting in more chunks being stored in memory on my config (10 versus 4, typically0 and when the orange loss of TPS sections happened, the green line didn’t ever drop from its 20 TPS average, whereas on the script setup the TPS loss was showing a dip in the green TPS line around these moments – like logging in, a “/save-all” or respawning from death – and then when I looked at the lag frame over-spends and the % number of laggy frames, that data reflected that observation of the lines, and those numbers are getting a 2 or more fold improvement in my comparison timings – unless I’m reading them wrong.
I whole heartedly agree, it is nimbling at the edges and even if the numbers implied more than a small improvement, again it could all be placebo benefits – or a regression – when tried in actual use with multiple client load.
Either way I’m pretty satisfied any more tinkering by me will be me, going in circles.
I’m very thankful for all your great input and producing your superb script that gave me great insight into how you optimised your paper.yml settings, along with the script clawing back about 250MB of RPi memory by using screen from linux in CLI mode.
I probably will get round to trying with an SSD – although seemed to remember my 2146Mhz overclock gets unstable with HDD and SSD storage last time I tried, so might go the RPi400 route as it has 4GB of ram for a 1GB ramfs allocation, and it is actually available in the UK, unlike the 4GB and 8GB RPi4 out of stock at MSRP. Normally I’d have spares of SSDs laying around for no-budget hobby projects, but a computer for each of nephews, and niece in recent months has wiped me out for now.
Hey Paul,
Whoops, sorry for the confusion on the Pi 3 vs Pi 4! Maybe that will come in handy for someone searching the comments later as people have asked that before and there’s some good tips in there for it. Overclocking on the Pi 4 is not nearly as “good” as the Pi 3 (the SD card port and I believe the RAM also do not overclock on the Pi 4).
You could definitely still do the general Pi 4 CPU overclock if you haven’t done that yet and the few other ones that are modifiable. It’s definitely an easy test and it costs nothing. Even if you set parameters that are too aggressive they can be reverted easily by putting the SD card into a PC and changing config.txt back basically. It’s best to have one of those little heatsink kits to do this but later firmware revisions won’t let the Pi 4 cook itself like the earlier revisions sometimes had a problem with.
That sounds about right for the JVM overhead. One thing that can impact it is 64 bit vs 32 bit. I’m not sure if not we ever talked about those but the 32 bit JVM tends to use a little bit less memory. Other factors can influence it such as what JVM you use. This probably isn’t worth chasing at this point but I figured I’d mention it for completeness.
So to explain my interpretation of the timings a little bit (I thought about going into detail about this but I had already wrote arguably too much) basically there are 2 different main views you will use. There is the “All” view which is the average over all ticks. You’ll notice your numbers are much better here. The default view it puts you in is the “Lag” view. This only shows ticks where there was lag / a problem.
Given that background there’s a few things I look for in the data. The first is I look for things that cause lag spikes. When the usage is near or over 100% (it will be dark red for these) it means that those operations were taking longer than the entire tick to complete. That’s a lag spike that the players will feel. It probably only happens every 5 minutes it so though for the world save tick for example (this is why my script turns it up to 15 minutes, it’s usually only a couple of a second spike even on really barebones setups without many resources).
However, the reason I said your timings look great is that the more important view is to use the “All” tab which is not the default. This is where everything looks great. Your TPS is pretty much at full with only a couple of times where you even had a little bump up in the chart. To sum it all up things look fantastic but it looks like you may have some occasional lag spikes people will “feel” during those expensive chunk saving / ticking operations. Frankly this can happen even on PCs (especially servers with old HDDs).
It’s definitely true they can’t be eliminated entirely. You can definitely get them under 100% though with a SSD but we aren’t talking about a huge lag spike still at over 100% (a BIG lag spike could be thousands of percentage points if it’s like 10-15+ seconds or something insane). I want to say my lag chart looks closer to highs of 30-40% of TPS on a NVMe SSD though and not anywhere near 100%. This is admittedly on an 8GB Pi though or a Compute Module 4. This is mostly just to give a reference of what kind of performance could possibly be achieved if you wanted to get an extremely expensive NVMe drive for it (wouldn’t really make sense to do unless you had one around).
I honestly think everything looks great. You don’t want to go in circles with it like you were saying. It’s all tradeoffs at the point and you’ve more than mastered the art! I thought you were the one that was unhappy with it to be honest with you so I think I may have misinterpreted that you may have wanted to go further still. You are basically right at the point I reached with the script where I was completely happy with it!
Obviously you can always throw more money or hardware at it but that wasn’t the point of it. I just thought it would be really cool and fun to run it on there and all the other guides / scripts out there were either horrible or never kept up to date. Since I was a hardware tech I had access to plenty of free SSDs that were being pulled out of destroyed / surplused computers.
The SSD would be the big upgrade you could do if you found that those occasional spikes were even noticeable or actually a problem. I totally understand about not having the spares around (they come and go, sometimes lots of RAM is surplused, sometimes lots of HDDs/SSDs and then sometimes there is droughts). I’m sure one will come back into your hands to give it a go at some point and it would be great to see a direct TPS comparison on the same Pi with a SSD and get a modern measurement of what the exact performance gains look like!
I do have the RPI 400 and it’s great! It’s the most powerful Pi you can get (it has the most MHz CPU). The CM4s are extremely powerful as well but that’s mostly due to the fact they have an IO board. This is basically a motherboard that they plug into so that the CM4 gains whatever capabilities the IO board has (including native PCIe 1x ports and other fun stuff). The cost overall is a little higher because of all the accessories you need to get that you don’t need for a regular Pi like the IO boards and things to connect them altogether but you can do some really cool stuff with them.
I’m grateful to you for the great conversation! I’m not sure if you could tell but I definitely enjoyed it as you asked me in a lot more detail than I think anyone ever has about some of these performance questions. There’s very few people out there that would be able to bear listening to me talking about the strange lengths I went to even getting this playable on those old Pis with so much less than even your 2GB Pi4 has (this should be a great setup).
It did gave me a chance to write down all of this too in one place outlining my “process” for creating and optimizing this script (using data and testing with the timings tool so we can know instead of suspect, there’s a lot of voodoo in old optimization guides still posted that does not bear out in the numbers). These techniques we’ve discussed could be used to optimize any Minecraft server. It would be the same process on the PC and Paper is still one of the best choices (some configuration required for best performance as you discovered).
People have done really impressive things with the script I definitely didn’t design it for or even imagine so I’m happy to share anything I know as a lot of it I learned from comments / conversations here just like this with others. My intention was for it to be an easy to modify base/template if that makes sense and people can run with it beyond anything I could design for / imagine.
Definitely feel free to stop by any time if you run into anything to bounce off me or find anything interesting you want to share!
Do you know if v1.17 has the needed fixes to protect against the log4j security flaw? I was under the impression that 1.18 was required. Any thoughts on this?
Hey CardinalFang,
I’m honestly not sure. I do not think so. I looked at Paper’s downloads and 1.17.1 is considered a “legacy” version that is no longer supported and they will not support according to papermc.io/legacy
That does not look very encouraging. They don’t appear to really be supporting 1.17 at all anymore. I’ve heard it’s extremely dangerous to run even if you “think” you’ve fixed it. It doesn’t appear to be getting patched anymore and if you submit an issue related to that version they say you will receive 0 support and it will be closed. That’s very harsh compared to the past and I imagine it’s because they want *nothing* to do with the log4j exploits.
Basically the exploits equate to not only your server getting pwned 100% but every single player on there also getting pwned 100%. It’s basically as bad as it gets.
If you had to run 1.17 for some reason you’d want to do the official Minecraft fixes but they are honestly complicated.
1.18: Upgrade to 1.18.1, if possible. If not, use the same approach as for 1.17.x:1.17: Add the following JVM arguments to your startup command line:
-Dlog4j2.formatMsgNoLookups=true
1.12-1.16.5: Download this file to the working directory where your server runs. Then add the following JVM arguments to your startup command line:
-Dlog4j.configurationFile=log4j2_112-116.xml
1.7-1.11.2: Download this file to the working directory where your server runs. Then add the following JVM arguments to your startup command line:
-Dlog4j.configurationFile=log4j2_17-111.xml
Versions below 1.7 are not affected
There’s the list of steps you’d need to do. You’d basically need to modify start.sh to include this JVM command line argument as well as download the files. You’d also have to be sure it’s working if you wanted to risk if it wasn’t that your server and everyone on it will get completely obliterated by hackers if you made a mistake. If you are comfortable trying to run the exploits against the server and testing it that is really the only way to be sure.
Because of the complexity of the fix (and really no easy way to tell it’s working other than patch it) I would say that I, like Paper, want nothing to do with telling anyone it’s safe to run those versions under any circumstances. The only exception to that would be if you wanted to do the manual fixes and run the exploits against your server to make sure they aren’t working.
My understanding is that there are all these fixes and plugins out there for this but I see just as many people saying they’re easily bypassed if all of those fixes aren’t done 100% correctly. It does not look like Paper is attempting to patch it to fix these though, and if they are they didn’t mention it anywhere and in fact made it sound like quite the opposite from their legacy section.
Basically the issue with these exploits is they are at a pretty low level (log4j). That’s why this same exploit can be used on thousands and thousands of different pieces of software. It’s an extremely devastating hack. That’s why you’re seeing people like me (someone who has published security research before and has some more upcoming CVEs to publish) really squirm with this one. It’s complicated to fix but more importantly you have to be *sure* because if you’re wrong it’s game over. Unlike being blown up by a creeper though this game over could result in your entire network being filled with malware / your identity being stolen / whatever else evil hackers are trying to do.
If this was a minor exploit I would have personally patched it and said it’s safe for all versions. It’s a lot more complicated with this one though because it’s so low-level. It’s very rare that an exploit this devastating and widespread is found (maybe once every few years?). You really have to get into the JVM and the libraries and start configuring them to fix this. Given that a lot of people may have multiple JVMs on their machine and other configuration differences this becomes impossible to make sure you’ve made every single setup 100% safe. I think that’s why Paper and others like me aren’t even trying. We can’t guarantee that will be safe no matter what we try to do unless you literally download the exploits and check it.
I just want you guys to stay as safe as possible and these fixes are anything but a sure thing. Everyone (including Mojang) has been saying the only true safe thing is to upgrade to the latest and I’ve pretty much only heard horror stories from people who didn’t. Hopefully that helps!
James –
Thanks for your quick response….but I remain confused. It is my understanding that your script uses v1.17 of Minecraft yet the only “safe’ version we should be running these days (because of the log4j exploit) is v1.18. Is this correct, or do I not understand the details of Minecraft versioning?
Hey CardinalFang36,
No problem at all, I’m happy to clarify. This is a very important issue so I’d rather have any questions about it asked and fully answer them for sure. It will help others who come and see this conversation who may have the same question as well!
So let me explain how this is fixed really quick here. The main official Minecraft security notice / fix post is here.
According to the official Minecraft post for versions 1.17 you would add -Dlog4j2.formatMsgNoLookups=true to your server launch line. You’d essentially put this in start.sh. It says you don’t need the .xml fixes for version 1.17 according to Microsoft / Mojang. If it’s older than that there are more fixes you need to do with xml files to be safe. I found another source with this information here.
It would be difficult to automate doing the correct fixes for each one of these old versions (there’s up to 3 different fixes you have to do depending on how old it is). I can’t be confident they are using a Java version that will even respect the log4j flag I’m passing to it. It would basically be guarantees / promises I can’t really make because I can’t be sure of everyone’s exact configuration (and when I’ve made assumptions like this in the past people always let me know hey we were using it that way your assumption was wrong)!
My version does not use 1.17 (although you may have that if you installed during that time) and I don’t recommend using it to be honest. The only exception is if you are comfortable testing the exploits against your server after doing the fixes for the version you want to run which is the only way to be sure it’s fixed (other than upgrading). The current version will be using version 1.18.2. If you are still on 1.17 (with a previous install) you may just need to run SetupMinecraft.sh again. This will update you to the latest version (1.18.2). It is safe to do this on an existing server (it’s designed for it) but it’s always a good idea to make sure you have your backups saved outside of the Minecraft backups folder before upgrading.
Hopefully that clears things up and let me know if you have any more questions!
My confusion was around the fact that this page starts with “Minecraft 1.17 (Caves and Cliffs Update) is here! ” so I thought the latest version your script would support was currently 1.17. I see now that it is using the latest “1.18.2”. I just did a clean install on my Raspberry Pi 4 (8GB) as the prior setup was on Ubuntu and I seem to be having “apt get update” issues with that…so used this as an opportunity to start with a fresh install overall. I was also concerned that if someone had exploited the log4j issue there is no way I would ever know.
I actually had thought that I had been hacked. I had not used the server for a while and found out it was down. When I investigated, I found the server was out of disk space. I panicked but upon investigation found that the daily backups of the server data was the cause of the disk space issue.
Hey CardinalFang,
That solves the mysteries! So the backup issue has actually been fixed. For most of 1.18 being out it has only kept the most recent 10 backups. I had the same thing happen to my server and added this in to prevent this so at least that shouldn’t happen again on the new version!
I’m glad you weren’t actually hacked. I think you should be all safe and secured now and the backups won’t be a problem with the new “pruning” system that keeps the most recent 10 backups. Cheers and have fun!
is there any way to install a snapshot version, for instance the new deep dark snapshot thats not in the launcher yet?
Hey Andre,
That’s a great question. I think the problem is mainly that you would need a link to the zip file that has the dedicated server for that snapshot.
Theoretically if we had this you absolutely could do that. Do you know of a public download somewhere? It might actually work to just extract that zip over your existing server (make sure you have backups before trying this). Theoretically that would work if there is a public download!
I tried using winscp to move the file for the server.jar file into the minecraft folder on the server for the newest deep dark snapshot and ran it like a normal server without your script and it worked