Revisiting Fixing Valheim Lag – Modifying Send/Receive Limits
Published: June 15, 2021 / Last Modified: December 3, 2023 / Gaming / 252 Comments
My Valheim Dedicated Server Castle
I’ve previously covered fixing Valheim’s dedicated server lag by modifying the dedicated server’s send receive limits. Since I wrote that article not long after launch though the game has been updated that the old method no longer applies. I had theorized in the comments of that article how to address this and today Sam R. confirmed that this method did indeed alleviate most congestion on the server in the places that the send/receive limits are known to bottleneck.
This updated guide will show some updated methods you can use to help reduce lag on your Valheim dedicated server!
Symptoms of lag on Valheim dedicated servers
The symptoms you are experiencing on the server can help narrow down what type of lag you are experiencing and what may be causing it. Lag caused by your CPU being too overloaded for example will have different symptoms than lag that is caused by your server hitting the maximum send/receive limit as an example.
By far the most common type of dedicated server lag seen in the early access Valheim release so far has been mainly due to the low send/receive limit if 64KB/s that is hard coded into the server. This means that when you do something like take your character a heavily populated area on the server it has to send all that information about all the buildings/players through an artificially limited very narrow 64KB/s pipe.
Symptoms of the send/receive limit type of lag include things like chests taking a very long time to open, sometimes 10-30 seconds or never opening while at the same time being able to see players walking around normally. This is because the other people’s X / Y / Z positions is a very small amount of data compared to all the items/names/attributes that are stored in the chest. The other players moving easily fits through the narrow pipe while the chest full of loot’s data chokes since it is much larger.
I’ll specifically lay out how to modify your server and increase this limit in the “Modifying your server’s send/receive limit” section below. Before that though let’s cover a more specific type of lag to watch out for (mostly by being avoided) related to terrain manipulation.
Avoid extensive terrain manipulation (for now)
Another commonly observed source of lag has to do with terrain manipulation. Some players have found that if they take a big chunk out of the ground in Valheim and make extensive terrain modifications that the area gets filled with separate “instances” of terrain for each modification people make. Think of the original ground as one big piece in the game files. Valheim has the very cool feature of letting you modify terrain by raising it/lower it/level it/etc. Each time you do this though the game has to store what you did as a new “instance” or a modification.
The way Valheim seems to work now is it renders that original piece and then the modifications afterward. Early reports suggest having a whole bunch of them in one spot seems to sometimes cause everyone’s in-game FPS on the server to drop as low as 40 when the clients normally are much much higher than that.
The best way to avoid this type of lag at the moment is to limit how much terrain modification you are doing in areas near your base. It is likely this will be addressed/eased in future patches. There should be some things the developer team can do to clean up/merge these instances or at least reduce their effect on the rendering process so it doesn’t drop the in-game FPS so dramatically.
At this time there is no options file for Valheim unfortunately. The only way to change it is to modify the server’s code and recompile the module that controls this.
Fortunately this is a lot easier than it sounds because the module we need to change is in .NET instead of something like C++ which would be much more difficult to do this with. I did not come up with this method, a very clever user on reddit (maximgame) shared it here and I want to give credit where credit is due!
Despite it not being too difficult these modifications are at your own risk. I haven’t experienced any problems so far but technically we are modifying the server’s code with the following procedure so understand that this isn’t an officially supported method and could have side effects or if you make a mistake could cause serious problems. Make sure you have backed up everything (your game world files especially) before you try anything!
Another important note is that it’s very likely you will have to make the changes again each update since the file we are going to modify is going to get replaced by updates. Until they add either a launch option or options file to officially change/control this property that is unfortunately unavoidable.
With all that being said if you understand the above and still feel comfortable let’s proceed!
Get dnSpy utility
We are going to use the utility dnSpy to modify the server code and recompile the module. The utility is free and is available here (download the Win64 build or dnSpy-net-win64.zip): GitHub dnSpy Official Page
Extract the archive and run the program dnSpy.exe from inside the folder you extracted. The dnSpy application will open.
Modifying server with dnSpy
We are going to go to the “File” menu in the top left and choose “Open” like this:
dnSpy – File -> Open Menu
Now you need to navigate to the Valheim dedicated server folder which is located in your “steamapps” folder. If you left all the options default when you installed it’s typically “C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed”.
We are looking for the file “assembly_valheim.dll” in that folder:
dnSpy – Selecting Valheim server assembly file
Select the file “assembly_valheim.dll” and click “Open” and the server’s assembly will load into dnSpy.
Once it has finished loading we are going to go to the “Edit” menu and click “Search Assemblies” like this:
dnSpy – Search Assemblies
For the search criteria enter “ZDOMan” like this:
dnSpy – Searching for ‘ZDOMan’
After the search completes you should see a list of results like the one above. Double click on one of these entries on the list and it should place you in the ZDOMan section of the assembly (seen on the left hand side of the screen):
dnSpy – ‘ZDOMan’ section of assembly
Perfect. We are now in the right place to make the changes. We just need to locate the line that has the send/receive limitation. To find that press Ctrl+F to bring up the find box (or choose “Edit” -> Find from the menu) and search for:
sendQueueSize
which should bring up the following lines:
dnSpy – sendQueueSize Variable
If you are having trouble finding the lines make sure you’re using Ctrl+F to search this time and not the “Search Assemblies” from earlier at the bottom of the screenshot, that’s a different type of search. Make sure you’ve disabled the case sensitive options (the Aa right below the box) or any other search filters. Your search query box should be in the top right of the screen like the above screenshot that has sendQueueSize in it.
Now right click on this line:
int num = 10240 - sendQueueSize
and click “Edit Class (C#)…”. Another window will open that will let you make changes to the file. We are going to modify ‘10240’ to a variable of our choosing. I made mine 30720 since after doing some conversion math that is the equivalent increase from my previous launch version guide that provided great performance and I kept them the same. If you followed my last guide this will give you the exact same performance as the old one did.
There are two instances of this variable. The second instance is 3 lines above the highlighted line we searched for and is:
if (!flush && sendQueueSize > 10240)
Modify both instances of 10240 to 30720.
Theoretically setting a limit too high could cause the server to swamp your internet connection if you have a large number of players on and there’s essentially no limit but there shouldn’t be any other effects of a very high number.
Once you’ve changed both numbers to the desired value we need to press the “Compile” button in the bottom right corner of this new window. This window will close and take you back to the previous one.
Saving the new assembly
It’s time to save our changes into a new .dll file. Choose “File” -> “Save Module…”:
dnSpy – Save Module as assembly_valheim_modded.dll
Change the file name to assembly_valheim_modded.dll. Don’t overwrite the original file just yet as we will want to take a backup and make sure that the server is closed before we do that. Press “OK” to write the new assembly_valheim_modded.dll file.
Backing up original file and replacing with our new one
Now it’s time to back up the original file and replace it with our new modded file. First make sure your dedicated server is closed otherwise the dll file will be in use. It is also a good time to make a backup of your world file just in case (see my Valheim Dedicated Server Backup Location / Guide for instructions)
Navigate in a file explorer window to your Valheim server directory where we saved the new assembly file (usually C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed).
First make a copy of the original assembly_valheim.dll file. You can just select the file and “copy” then paste a copy of it in that same folder and Windows will make a “valheim_assembly Copy(1).dll” file for you or you can back it up to a separate folder. Now remove the original file and rename your “valheim_assembly_modded.dll” file to the original file name of “valheim_assembly.dll”.
That’s it, you’ve now replaced the old assembly with the new one! Go ahead and start your server normally and the changes will have taken effect.
If you know of other Valheim lag issues that I didn’t cover here or are still having problems let me know in the comments and I will update this guide with other tips/tricks and try to answer any questions. It’s still a very new game and there isn’t very much out there for it yet. Have fun and enjoy!
252 thoughts on “Revisiting Fixing Valheim Lag – Modifying Send/Receive Limits”
David C Burkhart
Hey jameachambers!
I have attempted to follow this guide to the letter and I get to the point where it is time to “compile”, dnspy spits out a number of errors such as: cs1001, cs0246, cs0103, cs0012 on a huge number of lines.
some perliminary reading and looking at the errors themselves, it appears that a bunch of references are lost. I get these errors even if I don’t change anything and atempt to compile.
I am only just beginning to dabble in this stuff and am sure there is a magic switch somewhere I am not flipping before starting.
Maybe this will be seen in the RIGHT place this time XD
I only recall seeing this when someone tried to take a Linux DLL and compile it on Windows I believe (or vice versa). Even a stock Windows 10 install should be able to read all the dependencies / parse the code. Missing these references would indicate your OS install is corrupt (very corrupt) which seems unlikely but it’s possible.
The references it has should pretty much be all stuff built into Windows. I’m guessing that this is the wrong file / a file from a different OS. It is not typical to have problems with this piece of it. Do you have another computer you can try on to see if you get the same results?
I will have someone else look at it. I don’t see how my OS would be corrupt. I don’t generally get any windows errors and the computer is installed with a licensed OS.
Maybe when I get home from traveling I can try it on my wife’s computer.
It certainly could be mod related too. Have them try disabling the mods as a test and see if the lag goes away if you have the opportunity. Let us know what you find! I’m open to the possibility there’s more going on here for sure as other people seem to be encountering this all of the sudden as well. With several other people chiming in it seems like it’s not going to be anything like a OS or disk failure.
This mod definitely works as-is with vanilla so it seems like the only logical conclusion is something must be different, and that something seems to me to be the mods. Which mod is it? I’m not sure yet, but it seems like the only plausible explanation to have seasoned network admins / IT professionals all of the sudden all getting this after many months of not seeing anything like this!
If you are able to narrow down to the specific mod that is actually where you would want to start modding/etc. to get rid of this issue as it’s very likely due to poorly thought out / buggy code potentially in these add-ons since they are unofficial!
So! after some messing around with one of my server admin colleagues:
We discovered the reason why I was unable to compile the assembly after editing the class was because we were pulling the .dll out of its native directory, the references the error codes alluded to was it’s sibling files in that directory. Once we placed the assembly_valheim.dll back into its home, we were again able to make the changes prescribed in your article. We are all extremely new to .net stuff so we were ignorant. Now that it is edited, it can then be transferred to the dedicated gportal server.
Thanks alot for the follow-up.
This helped me alot because I did the same mistake to only copy the one file onto my PC in order to change the .NET code.
My understanding is that this happens if you try to edit the assembly_valheim.dll file in a folder by itself. You need to edit it in place in the valheim_data (or valheim_server_data if you’re editing the server dll)\managed folder, with all the other files in place or you get errors because it references those files.
Thanks for this guide, once we implemented this guide with a lot of mods installed our send and receive caps out at 150kb/s and when this happens we get massive desync.
After a bit of researching online with my brother we have discovered some other settings that are limiting the transfer speeds:
Note: we have to edit both the client and the server valheim_assembly.dll, as these are different make sure you select ‘close all’ in dnSpy before opening the next .dll
-Server dll: Search assembly for ‘zSteamSocket’ and find ‘RegisterGlobalCallbacks’ and edit the class.
-add an extra 0 for the number in ‘gchandle3’ (set to 1536000)
-add the following line just before ‘gchandle.Free();’
Great find! That does indeed look correct. It even matches some of the old values (15360, the number you’re increasing by 10). I’ll definitely perform this modification myself and add it to the article here (crediting you for submitting it). That’s great that the 150kb/s is verified as disappearing here.
I did a bit more testing, turns out I got something wrong in the server .dll config, I kept getting errors in the server console about the maximum buffer size being 524288 bytes. Anyway I need to do more testing but I restored both the client and server dll files and then I edited both of gchandle3 lines to:
I suspect that this is all we need to change in the dll files to get better a connection / sync (if I increase the 10240 number as above the average data sent to the server appears [for the same areas] to increase compared to not changing it). I didn’t have anyone to test this theory out with today, but I think it looks promising from the single player tests I’ve done with lots of instances loaded.
That is really interesting! I can tell you for sure this is the garbage collection code. If it’s trying to garbage collect more/less memory than it should then that could definitely explain some of the problems. I’ll check out this modification for myself soon as some guests I had just left so I should be able to get it tested and look through the code to figure out what is happening there within the next day or two here!
My initial theory is that you’re looking for the right things like:
That SteamNetworkingConfig_SendBufferSize variable is what you would want to change theoretically, I would search and find any other references to it as this seems like the garbage collection piece that cleans up after it.
Is indeed the speed cap (as reported by the F2 statistics, unsure if they stats page is actually accurate however), as if I raise it to something like 524288 the send speed can now exceed the 150kbs cap that appeared to be present before (and this is with no other changes to the .dll).
Now given this single change appears to uncap the speed, it stands to reason that the same change in the server .dll should do the same thing (although I haven’t been able to test this theory as I will need to have 3-4 other players online with me).
My speculation is that the server isn’t processing any of the data it is receiving from the clients, it just receives it and then sends it off to the clients that are within range of the client who sent it (basically like p2p just via a server). What I think is happening when fighting a boss like bonemass with 3-4 players, is that the data coming in from each client is exceeding the servers capacity to forward it on to each client… so if each client was sending 80kbs of data the server would need to send 240kbs back to each of the clients (3 * 80), because the send limit is 150kbs the events start queuing up until the server can get to them, and with a larger queue this is made even worse because there is potential for more out of sync data to be sent rather than be discarded.
With this in mind I think that for servers that the player base has low speed uploads (DSL etc) you may want to reduce the queue on both the server and the client, but for those servers where the players have decent upload speed I think increasing the queue (maybe only x2-x4) and lifting the send speed limit. Not sure if what I am saying is true, I am not super familiar with C# I am just speculating based on observation.
It looks like in C# this is creating a “pinned” object that is not touched by the garbage collector until the handle to the object is completely free’d and released. So this does look like the right place after reading more about what it’s doing here. This should be the allocation of the Send buffer.
There’s a couple of things to consider here about why this might be happening. The first one is what happens if the receiving client’s receive buffer is too small to receive the amount of data that is in the server’s send buffer? Would this data be lost / dropped? Or is it resent somehow?
If you are trying to send them 153600 and their receive buffer can only hold 15360 these packets may be getting dropped and resent making things overall more slow/laggy. That would be 80-90% packet loss with that 10x value theoretically. For important packets these would be resent but there would be a delay. This may seem why lower numbers are better.
In theory what is happening is that the closer the number you’re putting in there is to 15360 the less packets are being lost by the client. If you set it to twice as much they would theoretically have up to 50% packet loss depending on how full the buffer is (anything past 15360 would be getting cut off/lost in this theoretical scenario).
If all clients were properly modified to the exact correct value this wouldn’t happen but I’m guessing from what you’re describing some variation of this is absolutely happening. Unfortunately the “receive” buffer is not going to be allocated in the same place on the client as this “sending” buffer most likely and would correspond to a different Steam_Networking variable since for every connection both sides will have a separate send buffer and a receive buffer for their end of the connection.
Just a few more things to think about but I definitely think this is indeed part of the send/receive equation (there may even be other parts as well we haven’t found yet limiting/restricting things in other ways). Each piece is like lifting another “gate” that is restricting the flow of traffic. Some of these gates may be too difficult to change or adjust or may have side effects such as needing a client modification for every player’s client to work properly.
My original method in the post is actually changing some of the restrictions around how the send/receive limits are managed (such as when it decides to drop data or try to send it anyway which sometimes still makes it through) but it is not actually changing the literal send/receive limits (at least not the “revisiting” method, the original post did but Valheim’s code around this was more primitive at launch than now).
This newer method you’re investigating is *literally* changing the send limit / send buffer size on the server side and that I think is where some of these problems may be coming from (especially if buffer sizes on client/server are not matching). There should also be a *server* receive buffer too (this is the send buffer) so there’s even more to full picture than this part as well. It’s possible that the send limit location and receive limit location are the same on both the client and server code but there should be another side of the coin here for the “receive” buffer.
Like you said sometimes it is GOOD to restrict the flow of this traffic. If your connection can’t handle it then it will be worse as the traffic will get dropped/resent/congested/etc. Other times these values could use some tuning to get optimal performance (varies by connection) which is why it would be really nice if these were all options for the server that were adjustable. Great discoveries!
Thanks for explaining that, that definitely appears to match what I am observing.
So i did some testing with 2 others just now (3 of us in total) fighting bonemass… which is a boss that has never been smooth. On both the server and the client .dlls I changed the queue size to 15360 (15kb) and the sendratemin/max to 524288 (512kb) and then distributed the .dlls to the other clients and the fight went flawlessly, there were lots of slimes and AOE animations going off everywhere and we never saw any signs of desync and we all could agree where the boss was throughout the fight. I also had the F2 menu up the whole fight and the receive rate would occasionally hit ~160 to 170kbs for a couple of seconds and then settle out around 130 mostly.
Hopefully this weekend we can try it again with a couple more players, but the results so far look promising.
GregR
If you want to share your modified dll’s, I’d be happy to jump in and help you test. I’m running Epic Loot 0.8.4 and my character is just about ready to fight Bonemass. My dedicated server has been having issues running more than 2 people with Epic Loot, so I’d love to get a fix for that.
patk88
Hi Greg,
After a bit more testing the settings we found that work best for us is to set the send queue to the size recommended in this article (30720) on both the server and the clients, and the sendratemin/max to 524288. This has by far given us the best results yet.
For your reference all of our players have at least 50/20 mbps internet connections and the game is running very well now with 3-4 players in a single area.
Nyanha
Hi patk88
I’m bad at this.
Can you explain more clearly?
You just need to change the value of gchandle3 to 30720. And where do you change the sendratemin/max entry?
GregR
If I may… I think he’s saying that you need to follow the instructions in the article above to set “sendQueueSize” to 30720 on server and client, and then set “GCHandle.Alloc” to 524288 on both server and client, as well.
I’m not sure about the GCHandle.Alloc piece, my understanding is that anyone who has tried that has had problems. Theoretically the values between sendQueueSize and the actual buffer garbage allocation would need to match. If they don’t you’re going to have packet loss or worse because there won’t be enough room in the buffer to hold the data.
My personal suggestion would be to stick with the article one. You don’t literally need to change the buffer sizes. The original person that did this changed them back because they were experiencing massive packet loss (makes sense). If you change the send/receive buffer on one ALL clients will need to be modified. I really don’t recommend this piece of the mod from the comments as we were more trying to examine it and learn together but the GCHandle.Alloc piece has never “worked” or fixed anything from what I’ve seen (quite the opposite)! I’ll still leave it all available if people want to experiment with those methods though.
Just know that nobody has really got that working at all and it’s a separate (unnecessary and ineffective, maybe even counterproductive so far) way to do it! I’ve never done the second mod personally and am just helping them try to decipher through the code but frankly since it requires every client to be modified or they will experience packet loss I don’t really recommend messing with the garbage collection / send receive literal buffer!
Pretty much if the mod that is covered in the article didn’t largely eliminate the lag on your server you should be looking elsewhere at this point. You’re not going to be able to modify your way out of it if you’re still experiencing lag even after loosening the code restrictions on when data is skipped/dropped. It’s not a very demanding application (the server) relatively speaking but having an old slow HDD and things like that are going to have a negative impact. Your server/computer might need a reimage, a SSD upgrade from a HDD, a router reboot or network connection upgrade, something like that. Hopefully that helps!
GregR
I thought that was what Patk88 was saying he’s tried and achieved great results on? He said “After a bit more testing the settings we found that work best for us is to set the send queue to the size recommended in this article (30720) on both the server and the clients, and the sendratemin/max to 524288. This has by far given us the best results yet.”. Granted, maybe I misinterpreted “sendratemin/max” as “GCHandle.Alloc”, because I wasn’t 100% sure that’s what Pat was referring to, but based on an earlier post, I thought it most likely. We previously set our server’s queue size to 30720 as stated in the article, but hadn’t modded our client .dll’s. We got slightly improved performance, but still had desync bad enough to not be playable with more than 2 players. Strangely, it doesn’t appear to be a latency issue for us, two of us are on the same LAN as the dedicated server, with a gigabit connection to the internet, and the third player is on a separate gigabit connection. I think that the server simply can’t handle all the extra data associated with Epic Loot. Anyway, I made changes to the dll’s as I indicated above, I’ll try that out this evening and post results.
Your interpretation may be right as well! My interpretation was right at least initially but after telling him that everything needs to match up it does sound like he may have found it but I’m not entirely clear if he meant he rolled back the changes and then it was fine after that or if he meant that after adjusting it to that specific variable on all clients (but not the default variable) the buffers matched and it worked. Nobody else has ever chimed in on it so far here and not even I’ve tried it.
I personally honestly have very little interest in modifying all of the game clients alongside the server which you would have to do with something like this. There’s easier solutions than that for sure and frankly if you’re getting in the business of distributing/modding clients you can do way, way cooler stuff than that.
If you try this let me know what you find for sure as I’d love to clear this up! I usually would have 4-5 players on with just the mod from this article and didn’t experience any lag after that. That’s what I meant by saying you may not be able to mod your way out of this one. Even a non-dedicated server just running as one of the other player’s game clients shouldn’t start lagging with 2-3ish and it would indicate there’s something else wrong (almost always network / individual PC related).
Is there anything else running on the server at all? There’s no way you are hitting the send/receive limits with 2-3 people. It’s really only a problem at scale like at least 4-5+ and the normal mod should alleviate it enough to get up to the player cap. If you ran a packet capture I bet it’s not even close to using 30KB or even 10KB of buffer for these packets so I wouldn’t expect this to have much of an impact at all. You shouldn’t be lagging to even a noticeable extent with that amount of players on. I have a 1 gigabit fiber connection though with extremely low pings so your mileage may vary. Again, this would point to it not being something in the Valheim code you can modify away. You need to fix your setup/infrastructure to really solve something like this.
When the server has 5-8 people on all of them will lag at once. It is a different and specific type of lag than what is being described here / the mod in the article addresses. It will not cure any other types of lag or even help basically because the type of lag we are fixing is a consequence of the game’s design and having lag with 3 people on (even with the mod) is not that at all. It’s not bumping up against the send/receive limits in that case for some technical reason and is coming from something else. If you tried hosting it on a newer/other computer / different connection it wouldn’t do it.
I’d just say that hardware and software-wise no it should not be lagging with at all with 2 or even 3 unless this is really old, outdated hardware like a HDD instead of SSD, other things are running on the server using all the resources, or the server’s OS needs a reimage / cleanup. It would not point to anything being wrong with Valheim at all at that point and that is *not* normal lag to get for 2-3 people! It’s definitely not a Valheim code problem though or anything to do with the send/receive limits.
Regardless of how you decide to proceed I wanted to write this up as 99% of the people going down this road of trying multiple different mods including ones that require modifying all the clients are barking up the wrong tree. They’ll never solve it thinking that way because they’ve ruled out touching the computer’s hardware / connection / even reinstalling their OS at all before even starting their investigation and that is where most of these types of issues are found (especially on the extreme end).
They’re lost down the rabbit hole trying to modify the software when the actual issue is the network / the PC / the OS install / etc. so if you’re considering modding every client to try to get rid of lag it’s time to recognize that you’re out in the weeds and you need to question your assumptions about what / where the problem is. It’s a sign of strength / wisdom to recognize this and adjust course and an essential job/survival skill for IT tech work. I do this all the time so I’m speaking from experience here!
Unfortunately from my IT jobs over the years it’s almost certainly going to be the most annoying / difficult / painful / potentially expensive parts of your infrastructure to work with that are causing the slowdowns with this kind of latency/lag. Sometimes they just need a reboot. Other times you need upgrades if you want to seriously host from that location.
It’s seriously worth investigating these types of causes if you have a spare router / cable / modem / etc. you can try hosting on temporarily just to rule things out/narrow things down. It’s not at all obvious when networking equipment is malfunctioning a lot of the time but if you start swapping things / testing things out it can start to become pretty clear where the problem is coming from.
There’s no Valheim mod that will make up for a flaky router, or a <50 mbit connection you're trying to host multiple people on, etc. even if this doesn't end up being true for your individual case. It will be true for almost everyone. Why would not only your server, but all of your clients/players need this special mod to get 5 people on there without extreme lag but the other servers are (mostly) fine? That doesn't make sense. It's definitely hardware / connection related when it's this extreme.
Definitely let us know what you find if you try the comments-based mod anyway!
GregR
I get what you’re saying. I’m in IT also, have been for 20+ years. I’ve already done all of the low-level stuff to try to fix the issue. Swapped physical servers, rebooted routers, added RAM, reinstalled dedicated server software, etc. Nothing seems to help. I’m on a fiber 1gbps connection, and so is our remote player, so bandwidth is definitely not the issue.
The thing is, we DIDN’T have desync issues with vanilla (unmodded) Valheim. We were able to play through the whole game with 6 people on the server playing together. I’m almost certain the issue (and it’s widely reported) is with the Epic Loot mod for Valheim. We’re trying to do a second play through with that mod installed because it adds new content, which the Iron Gate devs aren’t doing very quickly. I’m almost certain that the issues are caused by several players with lots of “enhanced” gear, which the game apparently has a hard time keeping track of.
Anyway, I’m just hoping that the changes Pat suggested will work for us. I’m going to try to ight, I’ll post back with results. Thanks for hosting this discussion, either way!
Thanks for the context, that definitely helps, I agree with your conclusion here. I’ve heard talk about the Epic Loot mod desync bug here as well. It may be worth running a session without it just to see if that immediately fixes the lag too during your testing later.
I had a feeling you had your bases covered here technically (I left myself some outs for that because of some of the terms you used that seemed more technical than I’d expect) but I wanted to take the chance to throw up a caution sign here as most people haven’t worked in the industry like we have! Anybody who has all of their bases covered technically (and they know it) will disregard my warnings and know that it’s time for drastic action. I definitely didn’t mean it as an attack or anything personal but rather wrote that keeping in mind that a lot of people are going to follow in the footsteps of what they read (at least historically speaking that has been true) so I took it more as an opportunity to do my due diligence here and make sure people are on the right track and understand they are entering the “advanced” zone going this far and should have good reasons for doing so.
I look forward to your results if you’re willing to share them with us, it sounds like we couldn’t have asked for a more experienced tester for this modification as it is hot off the presses and not tested by even myself yet. The modding all of the clients is still a pain but if the result is worth it then in some cases it may indeed be worth it. I can imagine large groups of players playing together frequently on the same server would accept having to do this or just distributing a team zip or something of the modded file (or even fancier)!
Zarien
Hey GregR, I think you’re on the right track. We’re running into the same issue (fellow IT Manager/Networking Specialist) and this has been driving me nuts. No issues on our vanilla unmodded server, tons of issues when we added the mods recently.
I’ve made the original changes that james mentioned and saw no significant improvements when we added more than 2 players. I’ll be really interested in seeing your results and if you had any improvements tonight. I’m pulling my hair out looking for a way to improve this, because I know it’s not my hardware/1gbps fiber line either, and the headless client itself is barely touching my insanely overkill hardware.
Please let me know if you we’re able to get this to work!
Yeah I’m not sure how this could help in that situation. I do a lot of work with Minecraft as well and it’s well known that installing mods slows down your server / lowers your TPS or “ticks per second” that the server is running at.
What we do in that situation is remove all of the mods and enable them one by one. Have everyone log in and test and find out which mod *specifically* is causing the lag. Maybe it will be several combining together but it’s usually just one bad one.
Epic Loot has come up about a half dozen times here at least. I’m not sure if that’s just because it’s super popular and everyone has it or if that is the actual culprit that is causing the lag but you will need to disable them and turn them on one by one to find out which one it is typically.
What is your CPU usage / network usage like during these lag spikes? How much data are you actually sending during these periods? Does the CPU spike / peg during those times? How about memory usage? Have you run a continuous ping -t during gameplay during the server to watch for spikes in latency during lag spikes in game? Does everyone lag at once or is it one or two people lagging separately / much worse than the others?
What information do you guys have that is pointing to the TCP Send/Receive limits in this case? All information to me seems to be clearly pointing to the mods here. Epic Loot mod seems like an obvious guess, but I wouldn’t be surprised if we’re thrown a curveball and that it ends up being something else or several other things.
This article has been up a while and this hadn’t really been coming up at all until really recently. I’m guessing the huge spike in people having this problem are just because there are a lot more mods and people are installing them a lot more regularly. I really don’t think the TCP stack is the answer here though!
Zarien
It’s been confirmed this is stemming from Epic Loot in the forums/post discussion for the mod recently. I’m just looking for a modification, even if required for both client and server, that will allow for the desyncing to not occur despite the overloading of information coming from the recently poor gear characterization data from the mod, because most of us are revisiting the game for the flavor it brings to the table.
Thanks for filling me in! I’ve been holding out for the Hearth and Home update but even in that there are no new zones so I completely understand the burgeoning mod scene.
Have you attempt to decompile / look at the code for this mod by chance? I bet it’s not nearly as complicated as the Valheim game code and even that is .NET so it’s pretty readable. If you know anyone with a software background you could actually run this mod through a debugger and find out what is so slow in it and disable / modify it.
Any idea what language we’re up against or how much code we’re talking about here? Oh wait, it’s open source.
I guess I should ask if you guys are running the latest branch from GitHub (not necessarily the one from the downloads site) unless you get it from GitHub anyway. This should make this a *lot* easier to figure out though! My guess just off the top of my head is that it’s going to be something to do with this “Implement syncing known items and recipes between clients and server.” that I see from 4 months ago. That sounds like a bad idea. If it’s possible to shut off / turn off some of this stuff I’m guessing all of the lag is coming from some noble goal but practically unwise feature in this mod!
GregR
We’re using 0.8.4, the latest version on Nexus and also on Github. Epic Loot really adds a ton of replayability to the game. If you haven’t tried it out, it’s definitely worth a shot.
I did notice that there’s an issue reported on the Github page that would seem to identify a root cause (or at least a contributing factor) for the desync issues. Could be a clue for those of you who code!
GregR
Sadly, my testing didn’t happen tonight, life got in the way. Hoping for better luck tomorrow.
As for what makes me suspect it’s the TCP stack, I’m kinda just grasping at straws at this point. The desync makes me think that the net code is getting overwhelmed by too many objects, and I’m thinking that could be alleviated by opening things up. I’m not a coder, I’m just hoping this works!
No worries, that’s what I was trying to determine but Zarien and David both kind of filled me in and this is for sure a “thing”. The investigation is ongoing so don’t worry if you don’t have time to check it out for a while.
It’s really getting interesting so we’ll have to see what this ends up being exactly! Mods seems like the most likely explanation (especially now that I’m reading about some of the features they tried to put in the Epic Loot mod that sync between the client and server, yikes) but I’m open to it being something else since a lot of people are having this. Take care!
GregR
OK, so, I still didn’t get a chance to test fully. My wife’s machine decided to have a boot issue that I now have to sort before she can play. In better news, however, both the server and the one client who did manage to play last night had modded dll’s (SendQueueSize AND GCHandle.Alloc) in place, and the player said he didn’t have any issues, everything seemed stable. He was able to fight and got some loot drops. I’m hoping we can test with more clients tonight, but no promises.
Thanks for the update! Those sound initially promising that someone had a good experience playing through with these settings.
GregR
We finally got to try it out tonight. Seems to work really well. We fought a ton of Draugr, Fuling, Skeletons, etc. Three of us in one area, all with lots of epic/legendary gear, no desync. I’ll continue testing, but it sure looks like this fixes the problem!
Wow, those are some fantastic results! Thanks for following up. This seems to be a potential option/solution here!
patk88
Hello Guys,
We have also been running since the last post I made about the queue size and sendratemin/max fine with 3-4 people playing in the same area.
The extra data does indeed seem to be caused by EpicLoot… which is a mod that we just can’t do without.
I did a bit more googling and it seems that a while back someone made a network compression mod for valheim a few versions ago which could alleviate much of the extra data.
Unfortunately it no longer works with the current version, if you are interested the link to it is here.
Also for anyone wondering how we manage clients, we are just using onedrive and a script to keep everyone’s client up to date (sync the directory).
This sounds like you’re really on the right track here. It would make sense that they hit a “wall” with the compression as that is quite a bit more tricky and is basically an additional game system someone has to not only create but insert into someone else’s game. Not easy!
I honestly don’t see where this much extra data could be coming from without it being a bug / unintentional but I’m not an expert in the plugin and haven’t installed it. You guys know better than I do what the capabilities of it are but this strikes me as more of a bug/unintentional thing where something is happening that isn’t supposed to. There’s no way they added enough extra packets to create this intense of lag.
What is it doing streaming Netflix in the background?! I guess it’s possible if they added events that fire a whole lot of times per second and generate new packets that are sent to everyone every time or something like that. It would definitely be poor design if that is the case. Some people that have chimed in on this though saying they also have the lag are seriously experienced IT personnel working with enterprise grade hardware on enterprise grade networks. They lag all the same which points more to a bug to me. If it was just “more data” and raised the requirements we would expect to see somewhat linear scaling and that the people with excellent networks/hardware might be okay and people with less bandwidth/power might be hurting. That is not the case with this from everyone’s reports/data.
My guess is that this is unintended / a bug for those reasons. I definitely need to get my Valheim stuff fired up and test some of this. A network trace even with Wireshark when running this mod might make it pretty obvious if there’s some packets repeating / spamming that don’t belong theoretically (at least to enough of an extent to cause this much lag almost universally for people running the mod). If it isn’t immediately obvious then you could take a capture with and without the mod enabled and compare the two and see how much the volume of data is changing etc.
If the volume of the data isn’t substantially changing between enabling/disabling the mod then that would point to the mod is actually lagging/stuttering/choking the server in CPU/memory and not in network/bandwidth. This is also a strong possibility and attaching a debugger/profiler may reveal where the slow code is in this case. It’s .NET so universalish stuff should be okay (dnSpy might even be able to attach, I can’t remember and would have to check). This is how a developer would usually debug their app. dotTrace would probably do it but it’s a 30 day trial and there are a bunch of alternatives as well.
Thanks for continuing to follow up and investigate this. I’m really impressed with what you guys have found and almost feel behind the curve now since you guys have just ran with this so well!
GregR
What I’ve read (and again, I’m not a programmer) is that the Epic Loot mod is syncing information between all clients to keep track of non-standard information like item enchantments and enhancements, and that that is creating a bunch of additional traffic that isn’t necessary when you’re just playing vanilla Valheim without the Epic Loot mod.
Thanks for the context! I think you’re probably right that this is happening, but as someone who has developed/published multiplayer games before I can tell you it’s a (possibly fatal) design flaw for sure if it isn’t a bug. I suspect this is the case from what everyone has told me.
If done correctly this mod shouldn’t add any significant amount of data even if it was changing/adding 10x more than it currently is. Minecraft tracks way more information than this about players/items/etc. and they’re absolutely tiny files of basically text. Tons of games/mods track far more information than this about the players and the items they’re holding and a lot of those games have player caps of way more than 8 on at a time. It’s definitely not that this mod is just doing so much and adding so much data.
The amount of extra stats/features this mod is adding is a pittance compared to how much data/stats are tracked in a lot (probably most honestly) of your other favorite games, and many of them allow more than 8 players! Yet they don’t lag when ran on enterprise hardware/networks (nor does vanilla Valheim) but this mod does and it definitely can’t be explained/rationalized/justified by what the mod is doing and the extra attributes/systems it tracks. If you added the mod’s features to vanilla Valheim directly (like if the developers added them in officially) I guarantee the lag / network traffic wouldn’t change at all. It’s a design/implementation issue of the mod itself.
The problem would be the design of the add-on and how it is tracking and distributing/sending that data. If it’s constantly sending a ton of packets back and forth between *ALL* clients in the game (whether they are on the screen/map or not) or anything like that then yeah that is not going to scale. Not even to 8 people.
Maybe they had to do it this way because most servers are non-dedicated/co-op servers and rely on the clients a lot more than a dedicated server model has to. If that’s the case they probably need a “dedicated server” specific mod that goes to a more client/server oriented model instead of blasting these packets to every single other player in the game for every single little action.
Doing this the “right way” as a mod is really really hard but not impossible. I suspect that they are working on how to redesign this because honestly if it was a bug it would have been fixed by now. This will require almost a complete rewrite or a *major* overhaul to change something like this which I’m sure is why this continues to be an issue!
Stepan
Good afternoon! I’ve been reading your comments for the third day now, and I’ve been trying to stabilize my server for about a week. I’m not a programmer, I just want to play a game with my friends without desynchronization. Please, could you send me both files (dll for the server and dll for the client) by email I would be immensely grateful to you, my life turned into hell while I was looking for a solution to this problem, but I never managed to fix everything!
It’s pretty easy to do the guide and I strongly recommend it! Distributing binaries wouldn’t make a lot of sense for this as every tiny patch would require new binaries to be built. It only takes a minute or two to do the mod. If it has been making your life a living hell surely it’s worth giving a try!
It’s going to be way, way more work to try to get people to do this for you and send it to you than to just do it and you’re going to have to find someone who is willing to do it for you every single patch/bugfix. The software is free and it only takes a couple of minutes. You won’t be “programming” anything and basically just change a couple of numbers! I’d strongly recommend giving it a try for sure. I don’t think even inexperienced people have had much trouble following the guide!
King
Hi Pat. I will try this tonight with 5 or 7 players in same area.
GregR
Were you able to try this out? If so, how did it work for you?
King
Hi. I think combining james and patk ideas changing SendQueueSize and GCHandle.Alloc to their specified values helped. data rcv increased upto 250. but…
With mods such as epic loot (this mod is in demand, maybe there are others causing issue), playing with 6 players that scrambles in 1 area, desync is inevitable at some point. host and clients keep on changing, instances merging (it’s not fun to stay in 1 area to maintain host), etc.
the fact that not all has good latency. It’s a hassle, determining verbally with others who is host and if they can portal in and out the area so others can be host, whichever has good ping result.
Btw, we are combination of wired/wireless and fiber/4g/cable. mostly it is the cable guy causing issue and sometimes 4g folks. They are my friends so I can’t kick. haha
Sorry for non technical feedback but I hope it helps. probably if you have situation like me, follow the fix here. then avoid using mods with demanding packets, data sent/rcv or whatever. Also bookmark this page.
Thanks, this is a good summary. That sounds about right and is the reason that the additional mod here in the comments could help. This previously wasn’t really necessary but with heavy mod use things seemed to be strained enough that this is coming up a lot more.
Thanks for the non-technical translation, I get lost in the technical weeds here a lot so it’s always appreciated!
GregR
What mods are you guys using? We use Epic Loot and CLLC, and we have been having huge desync issues with just three players. I’m desperate to find a fix, I love the game, and this combo of mods makes it worth playing through again after having beaten it on vanilla.
This fixed the issue in my server, both gchandle3 and sendqueuesize changes did the trick. Now it’s up to 305kb/s per player, which probably explains the desync when it was limited to 150kb/s.
I’m also using a lot of mods but, as reported in Epic Loot forums by other players, that mod alone is the one causing the problems. Someone said that when up to 6 players there is still desync happening, maybe try using a higher value in sendqueuesize if you see the receive speed close to the limit (some of you guys who know better maybe can correct me)
Thank you all for taking the time to share this info, it helped me and my friends a lot and now we can continue to enjoy this game. The mods really add tons of new gameplay features, we are using Epic Valheim v3.0 mods pack if anyone interested.
I followed James and pat instruction on a dedicated server. I don’t see server error logs, but I can only see lands and mountains. no trees, creatures, etc.
It sounds like there is an issue with that (patk88 posted a follow up) and I haven’t checked it out yet. You may want to hold off on that second part until it’s fixed but the one from my article shouldn’t cause any trouble!
Yes I ran into this issue as well, the thing I initially posted works only on the client .dll (turns out when I attempted to edit the server .dll i ended up searching the already open client .dll and didnt end up changing anything). For some reason when you change the server .dll it doesn’t send any data to the client (hence the land with no trees or structures).
I am still investigating down this route as I can see in the F2 display that when I change the client .dll the send speeds go from being capped at 150kbs to seemingly not having a cap (most I have seen so far is about 290kbs). I will post back here once I know more, this may take a few days because it is challenging to organise 2 to 3 other people to do some testing around work and home commitments.
One of the things I am wondering about at the moment is that increasing the queue size too much may result in too many past actions queuing up once the client hits the send cap, which could be causing the desync to get worse as it will have quite a large backlog of past events to send to the server and the newest events won’t get sent until the queue clears out…. this is only speculation based on observation though I’m not sure if that is what is happening.
And for anyone wondering I running Valheim with a lot of mods (probably 20-30 mods), with Valheim+ and Epic loot being the main ones.
Same here. Please make sure you are using latest epic loot as older versions are causing what it looks like desync. really appreciate you trying to fix it along with James. thank you so much guys
The latest (0.8.4) version of Epic Loot still has desync issues. A guy on the Nexusmods forums for the mod had this to say:
“All out of sync from the terrible logic of syncing things between the player. All players are constantly exchanging with all packages with their things. This is necessary to highlight enchanted items on the player.
This logic clogs up all traffic. If usually ~ 20 kb, then with a mod ~ 80 kb per second. 1 The player weighs 0.1 mb.
Such a huge weight is achieved by the same terrible logic of Extended Items Framework. Tens of kilobytes of garbage are simply recorded in the field “CreatorName”.”
I don’t know if that’s accurate or not, but it certainly seems to jive with what we’ve been seeing, in that 1-2 people with lots of “enhanced” gear on can be in an area without issues, but the moment the third person shows up, it’s desync city. I’m hoping that some netcode optimization like what’s being discussed here will help.
See my reply to your other message which has our current settings, we have all these mods installed and running with 3-4 players in a single area… the data receive rate does sometimes get into the 300kbps per client when there are a lot of monsters running around though (We have a physical server with on 1000/50 connection, so you may want to keep this in mind depending on where your server is being hosted).
So our group has been dealing with a lot of bad lag. 50+ mods but it really kicked in after adding Monsternomicon. Going into new areas or getting a lot of mobs in the area at the same time would cause stuttering, teleporting, or invisible mobs. It was fine alone and when my friends played together they were fine. It would only fail when I joined. Their pings are 15-40ms and mine is 80-90 due to distance. It really seemed that this difference in ping time was the issue because even one other player and me on the server would cause it to lag. But 2 or 3 of them together was no issue.
I first tried setting some of the mods back. I tried premade work arounds that I now know are outdated. I found this walk through and tried the buffer extension above. No luck. But then after rereading this cap change from Patk88 I finally gave it a whirl. I added it to the server and the clients and zero issues now. Even mods like the seed totem that would show a delay when hitting the totem to pop the vegetables is gone. We hit the server heavy using everything we had at once and no hiccups.
Absolutely amazing Patk88! Bravo!
Thank you!
While I am no stranger to programming, it is not my line of work. I would suggest making a guide like the queue changes to make it easier for others to make this modification. It appears stable and without issues.
What a fantastic outcome! It’s very strange that raising the buffer size has such a dramatic impact. That’s… so easy. No game / mod I’ve ever worked on would have been that easy to fix lag in!
It honestly makes even less sense that the buffer size would impact your ping. Like that is just not how pings work and they’re tiny ICMP packets. I’m guessing what is changing is the way that Valheim measures and displays the ping. It must be getting bottlenecked without this change and it’s showing up in the in-game ping. I bet if you kept a command prompt window open with a ping -t yourserverip the ping wouldn’t change even when it’s lagging in the game meaning it’s a “Valheim ping measurement lag” thing and your “real” ping from a network level is not likely changing here (especially if the send/receive buffers are the bottleneck, literally has nothing to do with ping times).
The only explanation I can come up with is that the mod is so badly programmed (especially at the network level) that the buffers are literally bursting at the seams and breaking down the entire network engine to the point where even in-game pings are spiking etc. because this type of a change should *never* have this dramatic of an impact unless there’s serious design issues at play.
I’m definitely not opposed to writing up a guide for how to do this modification but let’s be clear that this is to fix essentially broken mods (which is normally the developer’s job). The game is fine. Most other mods also appear to be fine (except obsolete and outdated ones etc). It’s only the “Epic Loot” mod and a few key others that are not fine. I wouldn’t use them if there’s *any* alternative to it at all (even if it takes a half dozen mods or more better programmed mods to make up for all the functionality) because it seems like garbage from what people are having to do just to not have lag. I’m assuming there either are no alternatives or it would take a dozen mods to make up for it or something like that but some of these mods are in an unacceptable technical state.
Why don’t they just change the buffer size for you if this is necessary for it to even work when it’s already a massive game-mod making thousands of changes? Have any of you approached the developer or opened a GitHub issue? Why is this mod (and a few others) getting this free pass while we have to reprogram their mod / the game itself for them while they’re not being pressured to fix/improve it? You all have literally proven it’s the mods, which mods, and now have even proven how to fix it (or how to work around it at the very least).
I don’t see a single issue regarding the lag on on the first issues page: (actually I see one from all the way back in June, and it was never answered and pretty far down the page, and no one replied and backed him up which means the developer will assume they have network/hardware issues or at least I would without other voices echoing it). Like does this developer even know this is going on to this extent? I doubt it. There’s seemingly far more posts about it on this page than the official project’s GitHub page! This really looks like one of the cases where everyone has assumed the developer is getting swamped with issues about this and in reality nobody actually bothered to try to contact them / submit an issue. I was expecting 50% of the issues on the page to be lag related from how much activity we’ve had here about it and in reality there’s almost nothing on there.
I’m sure the features are cool, but this network mod is *extreme* (from a technical standpoint) and making up for very very bad mod behavior. If you absolutely have to have this mod then definitely this additional change is worth considering! I know you mentioned some other mods here too and I focused on the “Epic Loot” one but all of this would apply to those mods as well. I should say that with 50 mods though (wow, the last time I logged on 50 mods didn’t exist in total!) maybe you really are hitting the send/receive limits because that’s a lot of additional packets/data theoretically and each mod has it’s own overhead so it’s not as efficient as if these were all “official” features etc. and some mod developers are much better/more efficient programmers than others.
If you are running 50 mods on the server I would just go ahead and do this mod for sure rather than chase down individual developers but for some of these really popular/important ones I highly encourage people to start posting some issues because from what I’m seeing even the *big* Epic Loot mod is getting almost zero feedback about the lag they are causing and these other developers have probably never even been contacted about it before from the look of it. A lot of these could be fixed/improved I’m sure if the developers took a hard look at it! Fixing a few of the bad apples would probably let you get it to 100 mods without lag until you have to chase down the worst offenders again. This send/receive limits change is just raising the ceiling but those mods are still eating up all your bandwidth which will limit your total mod capacity at some point. It would be good to get them fixed/optimized even if you’re doing the mod because it gives you more “breathing room” and capacity for additional mods!
Thanks for leaving this feedback. This modification sounds like a winner so don’t take my message the wrong way. I’m more disappointed in the developers as a fellow developer and really think they need to clean this stuff up/improve it. This networking code is so sloppy you literally couldn’t have got away with it back when I developed games regularly a good 10-15+ years ago. People didn’t have the connections/bandwidth to handle turning it up and you had to do it the right way and at some point you’re going to run into this even today if the mods aren’t improved (they are setting your ceiling for you with the inefficiencies).
With this type of a model you’re only as strong as your weakest client. Which players have the least bandwidth on their connections? They’re the ones that will get hit first as you continue to turn up these limits and start reaching the limits of their connections. Because of the P2P client-to-client heavier model used in Valheim once one person starts lagging everyone will feel/notice some effects to varying degrees as clients start “disagreeing” on what is happening in the game etc. and I’ve seen some STRANGE behavior from this.
For an example of strange behavior from these disagreements apparently elites are spawned by the clients based on what biome you are in and one time I was with a friend right along the Mountains and the Dark forest biome borders and an elite spawned. My friend spawned the elite as a troll because on his screen we were in the dark forest but on my screen we were still in the mountains and spawned a drake. Except my drake was bugged and he OBLITERATED me with a single cold ball (with a full cold resist set on) while on my friend’s screen he was fighting a troll! When I logged out and back in I also saw a troll and my bugged unobtainium-powered-cold-ball one-shot-wonder Drake was gone leaving the troll my friend had always seen from the very beginning in his place.
It would probably honestly be easier for everyone to just make an official(ish) send/receive mod based on these discoveries. Maybe I’ll do it or if one of you who has participated/investigated wants to run with it then you absolutely could. I might be able to whip up a one-click send/receive patching utility as well. For people running 50 mods though it really shouldn’t be a big deal to install a send/receive mod on top of the other 50 and would probably make this easier overall!
I agree it’s an issue with the mods themselves and for the ones we like the best there are no other options. This modification has continued to work seamlessly. I did see a couple times the server ram fill up all the way. It’s rare and I restart it when it happens. It only happens with the mods active. I imagine there is also a memory leak somewhere.
Great find, memory leaks would explain a lot! Bugs and memory leaks happen all the time (as every gamer knows) even in supposedly highly polished games, so it’s not difficult to imagine that modding your code into someone else’s game makes these things even more likely to happen for sure.
After I modded the DLL the game no longer starts. It works with the original DLL. The error message I see is:
Jul 23 13:08:32 valheim valheim_server.x86_64[2772]: (null)/.steam/sdk64/steamclient.so: cannot open shared object file: No such file or directory
Jul 23 13:08:32 valheim valheim_server.x86_64[2772]: [S_API] SteamAPI_Init(): Sys_LoadModule failed to load: (null)/.steam/sdk64/steamclient.so
Jul 23 13:08:32 valheim valheim_server.x86_64[2772]: [Steamworks.NET] SteamAPI_Init() failed. Refer to Valve's documentation or the comment above this line for more information.
Searching for steamclient.so:
locate steamclient.so
/home/steam/.local/share/Steam/steamcmd/linux32/steamclient.so
/home/steam/.local/share/Steam/steamcmd/linux64/steamclient.so
/home/steam/Valheim/linux64/steamclient.so
/home/steam/Valheim/steamclient.so
Switched back to original DLL and it worked again. Running Ubuntu 21.04. Did the modifications on my Windows PC and then uploaded the DLL and set the original owner and file rights (steam:steam / 755).
I think I may be able to help, fortunately Linux should work and it has been done recently! The original DLL shouldn’t have anything to do with the Steam client. I take it you aren’t launching the dedicated server with a batch file on Linux but Steam is barely used at all for anything but authentication with it so if it’s just opened in the background you can launch the server directly and it will get whatever it needs from it (if you launch it without it open it will try to launch it to authenticate). Otherwise the server is just a black console window. I don’t even launch the dedicated server using Steam on Windows and I wouldn’t on Linux either as you can’t set any options that way (it should be pretty easy to modify the batch file examples for Windows to a bash .sh script) and this may be worth trying because getting a crash in Steam itself doesn’t make a lot of sense here.
I don’t think you’re even getting as far as the last person who tried this (also Ubuntu, but LTS 18.04) before they got a crash (but not in Steam, in the Valheim dedicated server) and they followed up and said they reinstalled everything from scratch and then the modification “just worked” after that. All of their output was from the Valheim dedicated server (as expected, this is the only application that uses that valheim dll) and not Steam.
The file should be almost identical to the original. It should be like two bytes different and a nearly identical size. Have you compared the sizes? You could try using ILSpy and doing the modification on Linux but it doesn’t make any sense for this Valheim mod to be crashing Steam itself and we’ve never seen this before. The other people on Linux had fairly standard output (from the Valheim dedicated server, the thing we’re actually modding, NOT Steam at all) and it’s difficult to explain why they got theirs working from a reinstall other than probably 100 different very minor configuration differences that could explain it. Did you try setting the value for the mod to something really unusually high? You may want to try a lower value for the mod if you went absolutely bonkers as that could be an OS difference between Windows and Linux.
Did you take the original file from Linux and then modify it or did you modify a Windows version of the DLL and upload that to Linux? Definitely make sure you start with an original Linux file. I’m not sure if they’re different for Valheim specifically since it’s written in .NET (there may still be some differences, Steam games/servers have to be specifically supported for Linux, compatibility with Mono stuff although that’s less of an issue these days, etc.) but the Steam ones are very extensively different linking to all kinds of Linux shared libraries so I would definitely use the original Linux DLL and not one from Windows to avoid running into anything strange (it should still be okay to modify on Windows, just copy it from Linux first).
Make sure you do a chown on top of the chmod so the right users own the files (I know you already mentioned the chmod but wanted to add the take ownership part as well as this could still be a simple permissions error but due to ownership).
ls -al
then replace youruser/group with the ones from the other files listed in the directory using this command:
sudo chown youruser:yourgroup valheimdllfile
now ls -al should show the same user / group / permissions (far left column) for all files in the directory.
double check that the permissions match exactly too -- yours may be right in most situations but make sure all the +xs, the group permissions all match etc.
I don’t want to speculate further because this doesn’t make any sense. The Valheim DLL we are modifying is *not* used by Steam. The only way this makes sense is if it actually *is* a permissions/ownership error. In this case maybe Steam is choking on validating the permissions/file ownership before launch (in a very, very ungraceful way) but it shouldn’t actually be using this file yet otherwise if that makes sense. It doesn’t use any code in the DLL but it may check before launching if it owns the files which could explain this. The ownership is probably root:root or youruser:youruser or something like that after reuploading it and it just needs to be set to the same owner/group as the other files because Windows doesn’t have/understand these types of file attributes/permissions (it has it’s own similar but different ways to do them) so they will be lost in this process.
This has been done successfully on Ubuntu Linux no more than 2 weeks ago and looks like permissions still to me from your output (and the old DLL fixing it). The more I think about it the more I think it has to be ownership/permissions as literally nothing else can explain this behavior that I can think of. The errors are not even happening in the right application they should be theoretically. The good news is it should be an easy fix. Hopefully all of that helps and let us know what you find!
i’m just curious, since it’s not ever explicitly touched on in this guide or on the original reddit post, but what does the 10240 actually refer to? rather, how does that translate to the kbps that most people refer to?
Great question! The answer is tricky but I think we can grapple with it here.
So the first thing to say is that the old version worked a lot more like you’re saying where there was a strict kbps limit. The new version is kind of a hybrid of that system. It has a queue and that 10240 number is the queue size.
In the old version when this queue was full everything would start lagging. For a long time. You may not be able to open a chest for a whole minute for example. In this new version they added a “flush” flag where they can push through “critical” data and when the queue is full it actually DROPS your data. The routine just exits if the queue is bigger than the maximum queue size.
Most updates like this can be safely dropped (for example, it’s constantly sending your position to the server, if a few of those get dropped many many more will still get through). The ones that are too important to drop now use the “flush” flag and they will always be sent.
Ok, so with that explained we can attempt to grapple with the question. The 10240 number is the size in bytes of your TCP buffer. It’s basically a container and it can hold a maximum of 10240 bytes or 10.24 KB. Basically your send/receive buffer only holds 10KB of data. Once it has more data than that in there everything else is dropped unless it’s a critical update passed with the “flush” flag.
Because this is a two tiered system now this makes it impossible to say the exact KB/sec that this breaks down to because their “flush” command bypasses the limit so it’s more of a guideline in the new version. Increasing the size of this buffer basically gives you much more bandwidth to send “generalized” data that isn’t sent with the “flush” flag before you start dropping packets. You can feel when this happens because certain things you do will be ignored. The classic one used to be the chests. You would be able to see everyone moving but you could be trying to open a chest for more than 60 seconds while you can still see everyone run around.
In the new system I believe they made opening chests “flush” so that doesn’t happen as much anymore (and it was the most infamous send/receive limit problem at launch, nobody could open their chests in busy areas) but there’s still certain types of data that aren’t flushed. When your server is under high load you can actually tell which types of packets are being “flushed” and which ones are being dropped.
When packets are dropped your client has to “guess” a lot more on where things are. They will extrapolate where a monster will be for example and “guess” if the packets are being dropped. Then when the packets come through the monster might have to be moved if the guess was dramatically off from the server. Once the monsters position does come through you may see some jarring things as the client tries to figure out how to get the monster where it’s actually supposed to be vs. where it incorrectly guessed it was going to be. This works both ways too. The server and other clients also have to guess about YOU if you are the one dropping data so they’ll see you in slightly wrong positions as well. If you pay extra close attention you’ll see the tricks the game uses to try and move people/monsters back where they should be gently without it looking bad but if the position is wrong enough it will insta-teleport them and not even try.
So basically the best way to think of this new system is a “buffer” and the bigger the buffer is the more breathing room you will have under heavy load scenarios before packets start getting dropped due to these limits in the code. The bigger your buffer the less this situation will arise where players are dropping packets and “guessing” where things should be. The smaller the buffer the more packets will get dropped and the more your player’s clients will be “guessing” and “interpreting” player/monster positions and they won’t always be right on their predictions unfortunately. Hopefully that helps, sorry it’s tricky to explain and more complicated than it was at launch!
oh my gosh, thank you so much for that thoughtful and in-depth response! i was able to edit my server to sam’s 102400 below, and it seems to have alleviated much of the lag we experienced when we had 6 people online at once. i did experience the chest lag a couple times during our recent session, though that may have been attributable to some internet issues a couple of us were having at the time. i’m also thinking about compiling and sending an editted .dll to each user to see if that helps – since some of the users on the original reddit post had also tried it, and especially considering the next boss we have to tackle as a group is bonemass. any thoughts/recommendations in that regard?
That’s fantastic, I’m glad the giant explanation ended up making enough sense that you were able to get something out of it! It’s a complicated topic for sure.
Your initial results sound right on the money here. It makes a great deal of difference but definitely isn’t a 100% silver bullet or solution for all situations or types of lag. It does make enough of a difference to make 6 people in a zone go from a nearly unplayable mess to pretty decently playable with some glitches here and there. Definitely an improvement worth the time because in my opinion having lag/bandwidth issues during these types of battles is not only immersion breaking but can approach game ruining levels depending on how bad things go as a result of it. Especially in Valheim where death is relatively unforgiving!
Okay, now your question about the client side mod in most situations it won’t make a difference on a dedicated server. This is because when you’re in peer to peer mode or the non-dedicated server mode someone literally is the server so the reason the mod works in those situations is the same it works for the dedicated server.
Now I said most situations, so what kind of situations might it make a difference in? Actually, funny enough, the most likely place it could make a difference is a giant fight with lots of minion spawns that are attacking everyone in the group with a lot of annoying AoE tick per second damage.
To my horror, damage in Valheim, like even taking damage, everything to do with it actually, seems to be 100% client side. Sometimes if there’s a giant 2 star nuking a stuck player over and over it will finally kill them but it takes a WHILE and I’m not sure if the server is finally calculating they should have been dead 5 hits ago or if packets finally got through or what is happening exactly. What is not happening though is the server is definitely not calculating damage and killing people, their clients are, and you can tell when they lag and they don’t die when they should have been dead 10 times over basically.
Because damage is client side in a situation like a Bonemass fight everyone’s clients are going to be sending *many* more updates per second / frame than they would just walking through the forest even with a couple of other people. Think of all the arrows flying around, damage from projectiles and all the poison slowly ticking away, all these things require the client to send an update to the server saying “hey this toxic puddle doing x ticks per second damaged me”, “hey this arrow hit”, “this healing potion was popped so show everyone the status effect” just to kind of give some less abstract examples of the client-side updates that could potentially hit that default 10,240 buffer.
In normal situations this would almost never happen but it could make a difference here in something like a Bonemass fight which has some of the most minion action and poison/toxic cloud damage in the entire game. There’s very few places it could potentially help more in theory or where there are more weird updates flying around during a fight that wouldn’t necessarily be present in other fights / normal fights.
Hopefully that helps and let us know if you try it if you saw the same (or any) improvement from modding the client side. The answer I’ve got from people in the past (like in comments on the article from before this, this is the revisiting with the fixed version that was changed after launch) is that it helps but it’s a pretty marginal improvement in some situations vs. doing it on the server which has a strong, noticeable improvement across the board pretty much!
It works with the unmodified DLL but not with the modified DLL? The file should literally be like 2 bytes different from the original. If you put back the original file will it load? It definitely shouldn’t.
Did you modify the value to something crazy like way beyond what was used in the guide? Did you set it to something crazy like 1000000000 or -1 or something way outside a normal integer? Have you tried running this server on Windows by chance as a test?
We’re really not changing much here. Just one number twice. There shouldn’t really be any opportunity for things to be crashing from what we did unless the values are very strange. We changed two integer values and nothing else and if you followed the guide the code would literally just be comparing a slightly different number.
How did you even make this modification for Linux? Windows must have obviously been involved here somewhere to use dnSpy. Did you actually pull the file off the Linux machine to mod it or did you use a copy of the Windows dedicated server and then upload that modified DLL to a Linux system? You will definitely need to modify the Linux version of the file and not mix and match them. That is actually something crazy enough to cause something weird like this now that I think of it.
Can you show us your modification? I wouldn’t be surprised if restoring a backup fixes it and your whole server is crashed. It’s quite common which is why I emphasize the backups so much! The DLL literally shouldn’t make a difference here unless something went really wrong with the modification and I think there’s a good chance if you restore the old DLL it’s going to crash in the same way. It doesn’t make sense and something had to have gone wrong along the way different about your process vs. a typical Windows setup here.
I had the same problem as werner. The problem was modding the client DLL instead of the dedicated server DLL. They are different, and only the server one can work in dedicated server mode. Starting with the dedicated server Managed directory instead of the Valheim client directory fixed it.
Thanks for chiming in, that’s very useful information. It’s also extremely easy to do on accident / assume they are the same. The paths / subfolders to end up getting to the client and server look remarkably similar and the .dll files are the exact same name (but NOT identical).
This should help others out as well, thanks again!
I’m currently trying this method. Don’t seems to make a difference for now though. Maybe my values are not high enough, I don’t know. It’s like in the 200000 for the client.
It may be possible you’re dealing with latency/lag that isn’t related to the send/receive limits as well. Is it everyone or one person?
One of my favorite tests is to have them open a command window and type “ping -t yourserver”. Have them leave this window in the background (or throw it on another screen if it’s a multiscreen setup). This command will ping the server over and over every 1 second.
When people are experiencing lag related to the network link/connection you will see the ping response times spike during lag periods. In that case the send/receive limits won’t help and a connection diagnosis/upgrade may be in order (sometimes as simple as a router reboot). If the send/receive limits are restraining the connection (or something else is wrong, more on that in a moment) the ping response times will never change as the network link is fine. If the ping times never change that would point to other causes for sure.
It could also be something unusual about the server / structures / terrain like if there is a massive amount of terrain modifications, giant castles the size of entire valleys or other things like that on the server or some other source of lag created from within the server (sometimes due to bugs as well). To test for that you would just temporarily start a new server (as a test) and have all the same people jump on and try doing some things. If no lag is experienced you can narrow it down to something inside the server data files (definitely this can and has happened for sure, early access etc).
Hopefully that helps and let us know what you find!
I am currently running 102400 as the “magic number” and it appears stable, but more multiplayer testing is needed.
A few other server admins are testing this out as well (you can find them over at Valheim Modding in the #user-assistance channel). We are all grateful for your help!
That’s awesome! I’m curious what you will finally settle on for the best final number, that’s really interesting! Theoretically it may depend on how much bandwidth they have but I’m guessing there is going to be a sweet spot and that you’re probably already pretty close.
Thanks for the Discord link as well, that should help anyone else who is also looking for Valheim help!
Hey jameachambers!
I have attempted to follow this guide to the letter and I get to the point where it is time to “compile”, dnspy spits out a number of errors such as: cs1001, cs0246, cs0103, cs0012 on a huge number of lines.
some perliminary reading and looking at the errors themselves, it appears that a bunch of references are lost. I get these errors even if I don’t change anything and atempt to compile.
I am only just beginning to dabble in this stuff and am sure there is a magic switch somewhere I am not flipping before starting.
Maybe this will be seen in the RIGHT place this time XD
I only recall seeing this when someone tried to take a Linux DLL and compile it on Windows I believe (or vice versa). Even a stock Windows 10 install should be able to read all the dependencies / parse the code. Missing these references would indicate your OS install is corrupt (very corrupt) which seems unlikely but it’s possible.
The references it has should pretty much be all stuff built into Windows. I’m guessing that this is the wrong file / a file from a different OS. It is not typical to have problems with this piece of it. Do you have another computer you can try on to see if you get the same results?
I will have someone else look at it. I don’t see how my OS would be corrupt. I don’t generally get any windows errors and the computer is installed with a licensed OS.
Maybe when I get home from traveling I can try it on my wife’s computer.
Thanks for the answer back!
Hey David,
It certainly could be mod related too. Have them try disabling the mods as a test and see if the lag goes away if you have the opportunity. Let us know what you find! I’m open to the possibility there’s more going on here for sure as other people seem to be encountering this all of the sudden as well. With several other people chiming in it seems like it’s not going to be anything like a OS or disk failure.
This mod definitely works as-is with vanilla so it seems like the only logical conclusion is something must be different, and that something seems to me to be the mods. Which mod is it? I’m not sure yet, but it seems like the only plausible explanation to have seasoned network admins / IT professionals all of the sudden all getting this after many months of not seeing anything like this!
If you are able to narrow down to the specific mod that is actually where you would want to start modding/etc. to get rid of this issue as it’s very likely due to poorly thought out / buggy code potentially in these add-ons since they are unofficial!
So! after some messing around with one of my server admin colleagues:
We discovered the reason why I was unable to compile the assembly after editing the class was because we were pulling the .dll out of its native directory, the references the error codes alluded to was it’s sibling files in that directory. Once we placed the assembly_valheim.dll back into its home, we were again able to make the changes prescribed in your article. We are all extremely new to .net stuff so we were ignorant. Now that it is edited, it can then be transferred to the dedicated gportal server.
Thanks for the interaction!
Hey David,
Thanks for the follow-up! Don’t sweat it, I didn’t even consider that it was because they were “relative” references to be honest with you!
Thanks again for leaving this as I’m sure it will help others following along who may save some time with this!
Thanks alot for the follow-up.
This helped me alot because I did the same mistake to only copy the one file onto my PC in order to change the .NET code.
My understanding is that this happens if you try to edit the assembly_valheim.dll file in a folder by itself. You need to edit it in place in the valheim_data (or valheim_server_data if you’re editing the server dll)\managed folder, with all the other files in place or you get errors because it references those files.
just saw this! Thanks for the answer back. it seems all I had to do is read further XD
Hi James,
Thanks for this guide, once we implemented this guide with a lot of mods installed our send and receive caps out at 150kb/s and when this happens we get massive desync.
After a bit of researching online with my brother we have discovered some other settings that are limiting the transfer speeds:
Note: we have to edit both the client and the server valheim_assembly.dll, as these are different make sure you select ‘close all’ in dnSpy before opening the next .dll
-Server dll: Search assembly for ‘zSteamSocket’ and find ‘RegisterGlobalCallbacks’ and edit the class.
-add an extra 0 for the number in ‘gchandle3’ (set to 1536000)
-add the following line just before ‘gchandle.Free();’
SteamGameServerNetworkingUtils.SetConfigValue(ESteamNetworkingConfigValue.k_ESteamNetworkingConfig_SendBufferSize, ESteamNetworkingConfigScope.k_ESteamNetworkingConfig_Global, IntPtr.Zero, ESteamNetworkingConfigDataType.k_ESteamNetworkingConfig_Int32, gchandle3.AddrOfPinnedObject());-Now compile, and save the dll and select ‘Close All’
-Now open the client dll and do the same thing but the line will be this instead:
SteamNetworkingUtils.SetConfigValue(ESteamNetworkingConfigValue.k_ESteamNetworkingConfig_SendBufferSize, ESteamNetworkingConfigScope.k_ESteamNetworkingConfig_Global, IntPtr.Zero, ESteamNetworkingConfigDataType.k_ESteamNetworkingConfig_Int32, gchandle3.AddrOfPinnedObject());Once we did this we no longer had a cap of 150kb/s
Hey patk88,
Great find! That does indeed look correct. It even matches some of the old values (15360, the number you’re increasing by 10). I’ll definitely perform this modification myself and add it to the article here (crediting you for submitting it). That’s great that the 150kb/s is verified as disappearing here.
Thanks again!
Good Evening James,
I did a bit more testing, turns out I got something wrong in the server .dll config, I kept getting errors in the server console about the maximum buffer size being 524288 bytes. Anyway I need to do more testing but I restored both the client and server dll files and then I edited both of gchandle3 lines to:
GCHandle gchandle3 = GCHandle.Alloc(524288, GCHandleType.Pinned);
I suspect that this is all we need to change in the dll files to get better a connection / sync (if I increase the 10240 number as above the average data sent to the server appears [for the same areas] to increase compared to not changing it). I didn’t have anyone to test this theory out with today, but I think it looks promising from the single player tests I’ve done with lots of instances loaded.
Hey patk88,
That is really interesting! I can tell you for sure this is the garbage collection code. If it’s trying to garbage collect more/less memory than it should then that could definitely explain some of the problems. I’ll check out this modification for myself soon as some guests I had just left so I should be able to get it tested and look through the code to figure out what is happening there within the next day or two here!
My initial theory is that you’re looking for the right things like:
SteamGameServerNetworkingUtils.SetConfigValue(ESteamNetworkingConfigValue.k_ESteamNetworkingConfig_SendBufferSizeThat SteamNetworkingConfig_SendBufferSize variable is what you would want to change theoretically, I would search and find any other references to it as this seems like the garbage collection piece that cleans up after it.
Hi James,
I have done a bit more solo testing, from my tests the line:
GCHandle gchandle3 = GCHandle.Alloc(153600, GCHandleType.Pinned);
Is indeed the speed cap (as reported by the F2 statistics, unsure if they stats page is actually accurate however), as if I raise it to something like 524288 the send speed can now exceed the 150kbs cap that appeared to be present before (and this is with no other changes to the .dll).
Now given this single change appears to uncap the speed, it stands to reason that the same change in the server .dll should do the same thing (although I haven’t been able to test this theory as I will need to have 3-4 other players online with me).
My speculation is that the server isn’t processing any of the data it is receiving from the clients, it just receives it and then sends it off to the clients that are within range of the client who sent it (basically like p2p just via a server). What I think is happening when fighting a boss like bonemass with 3-4 players, is that the data coming in from each client is exceeding the servers capacity to forward it on to each client… so if each client was sending 80kbs of data the server would need to send 240kbs back to each of the clients (3 * 80), because the send limit is 150kbs the events start queuing up until the server can get to them, and with a larger queue this is made even worse because there is potential for more out of sync data to be sent rather than be discarded.
With this in mind I think that for servers that the player base has low speed uploads (DSL etc) you may want to reduce the queue on both the server and the client, but for those servers where the players have decent upload speed I think increasing the queue (maybe only x2-x4) and lifting the send speed limit. Not sure if what I am saying is true, I am not super familiar with C# I am just speculating based on observation.
Cheers.
Great followup! I think you are correct. I read some more about what this code is doing here.
It looks like in C# this is creating a “pinned” object that is not touched by the garbage collector until the handle to the object is completely free’d and released. So this does look like the right place after reading more about what it’s doing here. This should be the allocation of the Send buffer.
There’s a couple of things to consider here about why this might be happening. The first one is what happens if the receiving client’s receive buffer is too small to receive the amount of data that is in the server’s send buffer? Would this data be lost / dropped? Or is it resent somehow?
If you are trying to send them 153600 and their receive buffer can only hold 15360 these packets may be getting dropped and resent making things overall more slow/laggy. That would be 80-90% packet loss with that 10x value theoretically. For important packets these would be resent but there would be a delay. This may seem why lower numbers are better.
In theory what is happening is that the closer the number you’re putting in there is to 15360 the less packets are being lost by the client. If you set it to twice as much they would theoretically have up to 50% packet loss depending on how full the buffer is (anything past 15360 would be getting cut off/lost in this theoretical scenario).
If all clients were properly modified to the exact correct value this wouldn’t happen but I’m guessing from what you’re describing some variation of this is absolutely happening. Unfortunately the “receive” buffer is not going to be allocated in the same place on the client as this “sending” buffer most likely and would correspond to a different Steam_Networking variable since for every connection both sides will have a separate send buffer and a receive buffer for their end of the connection.
Just a few more things to think about but I definitely think this is indeed part of the send/receive equation (there may even be other parts as well we haven’t found yet limiting/restricting things in other ways). Each piece is like lifting another “gate” that is restricting the flow of traffic. Some of these gates may be too difficult to change or adjust or may have side effects such as needing a client modification for every player’s client to work properly.
My original method in the post is actually changing some of the restrictions around how the send/receive limits are managed (such as when it decides to drop data or try to send it anyway which sometimes still makes it through) but it is not actually changing the literal send/receive limits (at least not the “revisiting” method, the original post did but Valheim’s code around this was more primitive at launch than now).
This newer method you’re investigating is *literally* changing the send limit / send buffer size on the server side and that I think is where some of these problems may be coming from (especially if buffer sizes on client/server are not matching). There should also be a *server* receive buffer too (this is the send buffer) so there’s even more to full picture than this part as well. It’s possible that the send limit location and receive limit location are the same on both the client and server code but there should be another side of the coin here for the “receive” buffer.
Like you said sometimes it is GOOD to restrict the flow of this traffic. If your connection can’t handle it then it will be worse as the traffic will get dropped/resent/congested/etc. Other times these values could use some tuning to get optimal performance (varies by connection) which is why it would be really nice if these were all options for the server that were adjustable. Great discoveries!
Hi James,
Thanks for explaining that, that definitely appears to match what I am observing.
So i did some testing with 2 others just now (3 of us in total) fighting bonemass… which is a boss that has never been smooth. On both the server and the client .dlls I changed the queue size to 15360 (15kb) and the sendratemin/max to 524288 (512kb) and then distributed the .dlls to the other clients and the fight went flawlessly, there were lots of slimes and AOE animations going off everywhere and we never saw any signs of desync and we all could agree where the boss was throughout the fight. I also had the F2 menu up the whole fight and the receive rate would occasionally hit ~160 to 170kbs for a couple of seconds and then settle out around 130 mostly.
Hopefully this weekend we can try it again with a couple more players, but the results so far look promising.
If you want to share your modified dll’s, I’d be happy to jump in and help you test. I’m running Epic Loot 0.8.4 and my character is just about ready to fight Bonemass. My dedicated server has been having issues running more than 2 people with Epic Loot, so I’d love to get a fix for that.
Hi Greg,
After a bit more testing the settings we found that work best for us is to set the send queue to the size recommended in this article (30720) on both the server and the clients, and the sendratemin/max to 524288. This has by far given us the best results yet.
After a bit more googling I found a reddit post from way back that gave a good analysis of what the updated code is doing:
For your reference all of our players have at least 50/20 mbps internet connections and the game is running very well now with 3-4 players in a single area.
Hi patk88
I’m bad at this.
Can you explain more clearly?
You just need to change the value of gchandle3 to 30720. And where do you change the sendratemin/max entry?
If I may… I think he’s saying that you need to follow the instructions in the article above to set “sendQueueSize” to 30720 on server and client, and then set “GCHandle.Alloc” to 524288 on both server and client, as well.
I’m not sure about the GCHandle.Alloc piece, my understanding is that anyone who has tried that has had problems. Theoretically the values between sendQueueSize and the actual buffer garbage allocation would need to match. If they don’t you’re going to have packet loss or worse because there won’t be enough room in the buffer to hold the data.
My personal suggestion would be to stick with the article one. You don’t literally need to change the buffer sizes. The original person that did this changed them back because they were experiencing massive packet loss (makes sense). If you change the send/receive buffer on one ALL clients will need to be modified. I really don’t recommend this piece of the mod from the comments as we were more trying to examine it and learn together but the GCHandle.Alloc piece has never “worked” or fixed anything from what I’ve seen (quite the opposite)! I’ll still leave it all available if people want to experiment with those methods though.
Just know that nobody has really got that working at all and it’s a separate (unnecessary and ineffective, maybe even counterproductive so far) way to do it! I’ve never done the second mod personally and am just helping them try to decipher through the code but frankly since it requires every client to be modified or they will experience packet loss I don’t really recommend messing with the garbage collection / send receive literal buffer!
Pretty much if the mod that is covered in the article didn’t largely eliminate the lag on your server you should be looking elsewhere at this point. You’re not going to be able to modify your way out of it if you’re still experiencing lag even after loosening the code restrictions on when data is skipped/dropped. It’s not a very demanding application (the server) relatively speaking but having an old slow HDD and things like that are going to have a negative impact. Your server/computer might need a reimage, a SSD upgrade from a HDD, a router reboot or network connection upgrade, something like that. Hopefully that helps!
I thought that was what Patk88 was saying he’s tried and achieved great results on? He said “After a bit more testing the settings we found that work best for us is to set the send queue to the size recommended in this article (30720) on both the server and the clients, and the sendratemin/max to 524288. This has by far given us the best results yet.”. Granted, maybe I misinterpreted “sendratemin/max” as “GCHandle.Alloc”, because I wasn’t 100% sure that’s what Pat was referring to, but based on an earlier post, I thought it most likely. We previously set our server’s queue size to 30720 as stated in the article, but hadn’t modded our client .dll’s. We got slightly improved performance, but still had desync bad enough to not be playable with more than 2 players. Strangely, it doesn’t appear to be a latency issue for us, two of us are on the same LAN as the dedicated server, with a gigabit connection to the internet, and the third player is on a separate gigabit connection. I think that the server simply can’t handle all the extra data associated with Epic Loot. Anyway, I made changes to the dll’s as I indicated above, I’ll try that out this evening and post results.
Hey Greg,
Your interpretation may be right as well! My interpretation was right at least initially but after telling him that everything needs to match up it does sound like he may have found it but I’m not entirely clear if he meant he rolled back the changes and then it was fine after that or if he meant that after adjusting it to that specific variable on all clients (but not the default variable) the buffers matched and it worked. Nobody else has ever chimed in on it so far here and not even I’ve tried it.
I personally honestly have very little interest in modifying all of the game clients alongside the server which you would have to do with something like this. There’s easier solutions than that for sure and frankly if you’re getting in the business of distributing/modding clients you can do way, way cooler stuff than that.
If you try this let me know what you find for sure as I’d love to clear this up! I usually would have 4-5 players on with just the mod from this article and didn’t experience any lag after that. That’s what I meant by saying you may not be able to mod your way out of this one. Even a non-dedicated server just running as one of the other player’s game clients shouldn’t start lagging with 2-3ish and it would indicate there’s something else wrong (almost always network / individual PC related).
Is there anything else running on the server at all? There’s no way you are hitting the send/receive limits with 2-3 people. It’s really only a problem at scale like at least 4-5+ and the normal mod should alleviate it enough to get up to the player cap. If you ran a packet capture I bet it’s not even close to using 30KB or even 10KB of buffer for these packets so I wouldn’t expect this to have much of an impact at all. You shouldn’t be lagging to even a noticeable extent with that amount of players on. I have a 1 gigabit fiber connection though with extremely low pings so your mileage may vary. Again, this would point to it not being something in the Valheim code you can modify away. You need to fix your setup/infrastructure to really solve something like this.
When the server has 5-8 people on all of them will lag at once. It is a different and specific type of lag than what is being described here / the mod in the article addresses. It will not cure any other types of lag or even help basically because the type of lag we are fixing is a consequence of the game’s design and having lag with 3 people on (even with the mod) is not that at all. It’s not bumping up against the send/receive limits in that case for some technical reason and is coming from something else. If you tried hosting it on a newer/other computer / different connection it wouldn’t do it.
I’d just say that hardware and software-wise no it should not be lagging with at all with 2 or even 3 unless this is really old, outdated hardware like a HDD instead of SSD, other things are running on the server using all the resources, or the server’s OS needs a reimage / cleanup. It would not point to anything being wrong with Valheim at all at that point and that is *not* normal lag to get for 2-3 people! It’s definitely not a Valheim code problem though or anything to do with the send/receive limits.
Regardless of how you decide to proceed I wanted to write this up as 99% of the people going down this road of trying multiple different mods including ones that require modifying all the clients are barking up the wrong tree. They’ll never solve it thinking that way because they’ve ruled out touching the computer’s hardware / connection / even reinstalling their OS at all before even starting their investigation and that is where most of these types of issues are found (especially on the extreme end).
They’re lost down the rabbit hole trying to modify the software when the actual issue is the network / the PC / the OS install / etc. so if you’re considering modding every client to try to get rid of lag it’s time to recognize that you’re out in the weeds and you need to question your assumptions about what / where the problem is. It’s a sign of strength / wisdom to recognize this and adjust course and an essential job/survival skill for IT tech work. I do this all the time so I’m speaking from experience here!
Unfortunately from my IT jobs over the years it’s almost certainly going to be the most annoying / difficult / painful / potentially expensive parts of your infrastructure to work with that are causing the slowdowns with this kind of latency/lag. Sometimes they just need a reboot. Other times you need upgrades if you want to seriously host from that location.
It’s seriously worth investigating these types of causes if you have a spare router / cable / modem / etc. you can try hosting on temporarily just to rule things out/narrow things down. It’s not at all obvious when networking equipment is malfunctioning a lot of the time but if you start swapping things / testing things out it can start to become pretty clear where the problem is coming from.
There’s no Valheim mod that will make up for a flaky router, or a <50 mbit connection you're trying to host multiple people on, etc. even if this doesn't end up being true for your individual case. It will be true for almost everyone. Why would not only your server, but all of your clients/players need this special mod to get 5 people on there without extreme lag but the other servers are (mostly) fine? That doesn't make sense. It's definitely hardware / connection related when it's this extreme.
Definitely let us know what you find if you try the comments-based mod anyway!
I get what you’re saying. I’m in IT also, have been for 20+ years. I’ve already done all of the low-level stuff to try to fix the issue. Swapped physical servers, rebooted routers, added RAM, reinstalled dedicated server software, etc. Nothing seems to help. I’m on a fiber 1gbps connection, and so is our remote player, so bandwidth is definitely not the issue.
The thing is, we DIDN’T have desync issues with vanilla (unmodded) Valheim. We were able to play through the whole game with 6 people on the server playing together. I’m almost certain the issue (and it’s widely reported) is with the Epic Loot mod for Valheim. We’re trying to do a second play through with that mod installed because it adds new content, which the Iron Gate devs aren’t doing very quickly. I’m almost certain that the issues are caused by several players with lots of “enhanced” gear, which the game apparently has a hard time keeping track of.
Anyway, I’m just hoping that the changes Pat suggested will work for us. I’m going to try to ight, I’ll post back with results. Thanks for hosting this discussion, either way!
Hey Greg,
Thanks for the context, that definitely helps, I agree with your conclusion here. I’ve heard talk about the Epic Loot mod desync bug here as well. It may be worth running a session without it just to see if that immediately fixes the lag too during your testing later.
I had a feeling you had your bases covered here technically (I left myself some outs for that because of some of the terms you used that seemed more technical than I’d expect) but I wanted to take the chance to throw up a caution sign here as most people haven’t worked in the industry like we have! Anybody who has all of their bases covered technically (and they know it) will disregard my warnings and know that it’s time for drastic action. I definitely didn’t mean it as an attack or anything personal but rather wrote that keeping in mind that a lot of people are going to follow in the footsteps of what they read (at least historically speaking that has been true) so I took it more as an opportunity to do my due diligence here and make sure people are on the right track and understand they are entering the “advanced” zone going this far and should have good reasons for doing so.
I look forward to your results if you’re willing to share them with us, it sounds like we couldn’t have asked for a more experienced tester for this modification as it is hot off the presses and not tested by even myself yet. The modding all of the clients is still a pain but if the result is worth it then in some cases it may indeed be worth it. I can imagine large groups of players playing together frequently on the same server would accept having to do this or just distributing a team zip or something of the modded file (or even fancier)!
Hey GregR, I think you’re on the right track. We’re running into the same issue (fellow IT Manager/Networking Specialist) and this has been driving me nuts. No issues on our vanilla unmodded server, tons of issues when we added the mods recently.
I’ve made the original changes that james mentioned and saw no significant improvements when we added more than 2 players. I’ll be really interested in seeing your results and if you had any improvements tonight. I’m pulling my hair out looking for a way to improve this, because I know it’s not my hardware/1gbps fiber line either, and the headless client itself is barely touching my insanely overkill hardware.
Please let me know if you we’re able to get this to work!
Yeah I’m not sure how this could help in that situation. I do a lot of work with Minecraft as well and it’s well known that installing mods slows down your server / lowers your TPS or “ticks per second” that the server is running at.
What we do in that situation is remove all of the mods and enable them one by one. Have everyone log in and test and find out which mod *specifically* is causing the lag. Maybe it will be several combining together but it’s usually just one bad one.
Epic Loot has come up about a half dozen times here at least. I’m not sure if that’s just because it’s super popular and everyone has it or if that is the actual culprit that is causing the lag but you will need to disable them and turn them on one by one to find out which one it is typically.
What is your CPU usage / network usage like during these lag spikes? How much data are you actually sending during these periods? Does the CPU spike / peg during those times? How about memory usage? Have you run a continuous ping -t during gameplay during the server to watch for spikes in latency during lag spikes in game? Does everyone lag at once or is it one or two people lagging separately / much worse than the others?
What information do you guys have that is pointing to the TCP Send/Receive limits in this case? All information to me seems to be clearly pointing to the mods here. Epic Loot mod seems like an obvious guess, but I wouldn’t be surprised if we’re thrown a curveball and that it ends up being something else or several other things.
This article has been up a while and this hadn’t really been coming up at all until really recently. I’m guessing the huge spike in people having this problem are just because there are a lot more mods and people are installing them a lot more regularly. I really don’t think the TCP stack is the answer here though!
It’s been confirmed this is stemming from Epic Loot in the forums/post discussion for the mod recently. I’m just looking for a modification, even if required for both client and server, that will allow for the desyncing to not occur despite the overloading of information coming from the recently poor gear characterization data from the mod, because most of us are revisiting the game for the flavor it brings to the table.
Thanks for filling me in! I’ve been holding out for the Hearth and Home update but even in that there are no new zones so I completely understand the burgeoning mod scene.
Have you attempt to decompile / look at the code for this mod by chance? I bet it’s not nearly as complicated as the Valheim game code and even that is .NET so it’s pretty readable. If you know anyone with a software background you could actually run this mod through a debugger and find out what is so slow in it and disable / modify it.
Any idea what language we’re up against or how much code we’re talking about here? Oh wait, it’s open source.
I guess I should ask if you guys are running the latest branch from GitHub (not necessarily the one from the downloads site) unless you get it from GitHub anyway. This should make this a *lot* easier to figure out though! My guess just off the top of my head is that it’s going to be something to do with this “Implement syncing known items and recipes between clients and server.” that I see from 4 months ago. That sounds like a bad idea. If it’s possible to shut off / turn off some of this stuff I’m guessing all of the lag is coming from some noble goal but practically unwise feature in this mod!
We’re using 0.8.4, the latest version on Nexus and also on Github. Epic Loot really adds a ton of replayability to the game. If you haven’t tried it out, it’s definitely worth a shot.
I did notice that there’s an issue reported on the Github page that would seem to identify a root cause (or at least a contributing factor) for the desync issues. Could be a clue for those of you who code!
Sadly, my testing didn’t happen tonight, life got in the way. Hoping for better luck tomorrow.
As for what makes me suspect it’s the TCP stack, I’m kinda just grasping at straws at this point. The desync makes me think that the net code is getting overwhelmed by too many objects, and I’m thinking that could be alleviated by opening things up. I’m not a coder, I’m just hoping this works!
Hey Greg,
No worries, that’s what I was trying to determine but Zarien and David both kind of filled me in and this is for sure a “thing”. The investigation is ongoing so don’t worry if you don’t have time to check it out for a while.
It’s really getting interesting so we’ll have to see what this ends up being exactly! Mods seems like the most likely explanation (especially now that I’m reading about some of the features they tried to put in the Epic Loot mod that sync between the client and server, yikes) but I’m open to it being something else since a lot of people are having this. Take care!
OK, so, I still didn’t get a chance to test fully. My wife’s machine decided to have a boot issue that I now have to sort before she can play. In better news, however, both the server and the one client who did manage to play last night had modded dll’s (SendQueueSize AND GCHandle.Alloc) in place, and the player said he didn’t have any issues, everything seemed stable. He was able to fight and got some loot drops. I’m hoping we can test with more clients tonight, but no promises.
Hey Greg,
Thanks for the update! Those sound initially promising that someone had a good experience playing through with these settings.
We finally got to try it out tonight. Seems to work really well. We fought a ton of Draugr, Fuling, Skeletons, etc. Three of us in one area, all with lots of epic/legendary gear, no desync. I’ll continue testing, but it sure looks like this fixes the problem!
Hey Greg,
Wow, those are some fantastic results! Thanks for following up. This seems to be a potential option/solution here!
Hello Guys,
We have also been running since the last post I made about the queue size and sendratemin/max fine with 3-4 people playing in the same area.
The extra data does indeed seem to be caused by EpicLoot… which is a mod that we just can’t do without.
I did a bit more googling and it seems that a while back someone made a network compression mod for valheim a few versions ago which could alleviate much of the extra data.
Unfortunately it no longer works with the current version, if you are interested the link to it is here.
Also for anyone wondering how we manage clients, we are just using onedrive and a script to keep everyone’s client up to date (sync the directory).
This sounds like you’re really on the right track here. It would make sense that they hit a “wall” with the compression as that is quite a bit more tricky and is basically an additional game system someone has to not only create but insert into someone else’s game. Not easy!
I honestly don’t see where this much extra data could be coming from without it being a bug / unintentional but I’m not an expert in the plugin and haven’t installed it. You guys know better than I do what the capabilities of it are but this strikes me as more of a bug/unintentional thing where something is happening that isn’t supposed to. There’s no way they added enough extra packets to create this intense of lag.
What is it doing streaming Netflix in the background?! I guess it’s possible if they added events that fire a whole lot of times per second and generate new packets that are sent to everyone every time or something like that. It would definitely be poor design if that is the case. Some people that have chimed in on this though saying they also have the lag are seriously experienced IT personnel working with enterprise grade hardware on enterprise grade networks. They lag all the same which points more to a bug to me. If it was just “more data” and raised the requirements we would expect to see somewhat linear scaling and that the people with excellent networks/hardware might be okay and people with less bandwidth/power might be hurting. That is not the case with this from everyone’s reports/data.
My guess is that this is unintended / a bug for those reasons. I definitely need to get my Valheim stuff fired up and test some of this. A network trace even with Wireshark when running this mod might make it pretty obvious if there’s some packets repeating / spamming that don’t belong theoretically (at least to enough of an extent to cause this much lag almost universally for people running the mod). If it isn’t immediately obvious then you could take a capture with and without the mod enabled and compare the two and see how much the volume of data is changing etc.
If the volume of the data isn’t substantially changing between enabling/disabling the mod then that would point to the mod is actually lagging/stuttering/choking the server in CPU/memory and not in network/bandwidth. This is also a strong possibility and attaching a debugger/profiler may reveal where the slow code is in this case. It’s .NET so universalish stuff should be okay (dnSpy might even be able to attach, I can’t remember and would have to check). This is how a developer would usually debug their app. dotTrace would probably do it but it’s a 30 day trial and there are a bunch of alternatives as well.
Thanks for continuing to follow up and investigate this. I’m really impressed with what you guys have found and almost feel behind the curve now since you guys have just ran with this so well!
What I’ve read (and again, I’m not a programmer) is that the Epic Loot mod is syncing information between all clients to keep track of non-standard information like item enchantments and enhancements, and that that is creating a bunch of additional traffic that isn’t necessary when you’re just playing vanilla Valheim without the Epic Loot mod.
Hey Greg,
Thanks for the context! I think you’re probably right that this is happening, but as someone who has developed/published multiplayer games before I can tell you it’s a (possibly fatal) design flaw for sure if it isn’t a bug. I suspect this is the case from what everyone has told me.
If done correctly this mod shouldn’t add any significant amount of data even if it was changing/adding 10x more than it currently is. Minecraft tracks way more information than this about players/items/etc. and they’re absolutely tiny files of basically text. Tons of games/mods track far more information than this about the players and the items they’re holding and a lot of those games have player caps of way more than 8 on at a time. It’s definitely not that this mod is just doing so much and adding so much data.
The amount of extra stats/features this mod is adding is a pittance compared to how much data/stats are tracked in a lot (probably most honestly) of your other favorite games, and many of them allow more than 8 players! Yet they don’t lag when ran on enterprise hardware/networks (nor does vanilla Valheim) but this mod does and it definitely can’t be explained/rationalized/justified by what the mod is doing and the extra attributes/systems it tracks. If you added the mod’s features to vanilla Valheim directly (like if the developers added them in officially) I guarantee the lag / network traffic wouldn’t change at all. It’s a design/implementation issue of the mod itself.
The problem would be the design of the add-on and how it is tracking and distributing/sending that data. If it’s constantly sending a ton of packets back and forth between *ALL* clients in the game (whether they are on the screen/map or not) or anything like that then yeah that is not going to scale. Not even to 8 people.
Maybe they had to do it this way because most servers are non-dedicated/co-op servers and rely on the clients a lot more than a dedicated server model has to. If that’s the case they probably need a “dedicated server” specific mod that goes to a more client/server oriented model instead of blasting these packets to every single other player in the game for every single little action.
Doing this the “right way” as a mod is really really hard but not impossible. I suspect that they are working on how to redesign this because honestly if it was a bug it would have been fixed by now. This will require almost a complete rewrite or a *major* overhaul to change something like this which I’m sure is why this continues to be an issue!
Good afternoon! I’ve been reading your comments for the third day now, and I’ve been trying to stabilize my server for about a week. I’m not a programmer, I just want to play a game with my friends without desynchronization. Please, could you send me both files (dll for the server and dll for the client) by email I would be immensely grateful to you, my life turned into hell while I was looking for a solution to this problem, but I never managed to fix everything!
Hey Stephan,
It’s pretty easy to do the guide and I strongly recommend it! Distributing binaries wouldn’t make a lot of sense for this as every tiny patch would require new binaries to be built. It only takes a minute or two to do the mod. If it has been making your life a living hell surely it’s worth giving a try!
It’s going to be way, way more work to try to get people to do this for you and send it to you than to just do it and you’re going to have to find someone who is willing to do it for you every single patch/bugfix. The software is free and it only takes a couple of minutes. You won’t be “programming” anything and basically just change a couple of numbers! I’d strongly recommend giving it a try for sure. I don’t think even inexperienced people have had much trouble following the guide!
Hi Pat. I will try this tonight with 5 or 7 players in same area.
Were you able to try this out? If so, how did it work for you?
Hi. I think combining james and patk ideas changing SendQueueSize and GCHandle.Alloc to their specified values helped. data rcv increased upto 250. but…
With mods such as epic loot (this mod is in demand, maybe there are others causing issue), playing with 6 players that scrambles in 1 area, desync is inevitable at some point. host and clients keep on changing, instances merging (it’s not fun to stay in 1 area to maintain host), etc.
the fact that not all has good latency. It’s a hassle, determining verbally with others who is host and if they can portal in and out the area so others can be host, whichever has good ping result.
Btw, we are combination of wired/wireless and fiber/4g/cable. mostly it is the cable guy causing issue and sometimes 4g folks. They are my friends so I can’t kick. haha
Sorry for non technical feedback but I hope it helps. probably if you have situation like me, follow the fix here. then avoid using mods with demanding packets, data sent/rcv or whatever. Also bookmark this page.
Hey King,
Thanks, this is a good summary. That sounds about right and is the reason that the additional mod here in the comments could help. This previously wasn’t really necessary but with heavy mod use things seemed to be strained enough that this is coming up a lot more.
Thanks for the non-technical translation, I get lost in the technical weeds here a lot so it’s always appreciated!
What mods are you guys using? We use Epic Loot and CLLC, and we have been having huge desync issues with just three players. I’m desperate to find a fix, I love the game, and this combo of mods makes it worth playing through again after having beaten it on vanilla.
This fixed the issue in my server, both gchandle3 and sendqueuesize changes did the trick. Now it’s up to 305kb/s per player, which probably explains the desync when it was limited to 150kb/s.
I’m also using a lot of mods but, as reported in Epic Loot forums by other players, that mod alone is the one causing the problems. Someone said that when up to 6 players there is still desync happening, maybe try using a higher value in sendqueuesize if you see the receive speed close to the limit (some of you guys who know better maybe can correct me)
Thank you all for taking the time to share this info, it helped me and my friends a lot and now we can continue to enjoy this game. The mods really add tons of new gameplay features, we are using Epic Valheim v3.0 mods pack if anyone interested.
I followed James and pat instruction on a dedicated server. I don’t see server error logs, but I can only see lands and mountains. no trees, creatures, etc.
Hey King,
It sounds like there is an issue with that (patk88 posted a follow up) and I haven’t checked it out yet. You may want to hold off on that second part until it’s fixed but the one from my article shouldn’t cause any trouble!
Visiting this page often. Thank you
Thanks for the kind words!
Hi King,
Yes I ran into this issue as well, the thing I initially posted works only on the client .dll (turns out when I attempted to edit the server .dll i ended up searching the already open client .dll and didnt end up changing anything). For some reason when you change the server .dll it doesn’t send any data to the client (hence the land with no trees or structures).
I am still investigating down this route as I can see in the F2 display that when I change the client .dll the send speeds go from being capped at 150kbs to seemingly not having a cap (most I have seen so far is about 290kbs). I will post back here once I know more, this may take a few days because it is challenging to organise 2 to 3 other people to do some testing around work and home commitments.
One of the things I am wondering about at the moment is that increasing the queue size too much may result in too many past actions queuing up once the client hits the send cap, which could be causing the desync to get worse as it will have quite a large backlog of past events to send to the server and the newest events won’t get sent until the queue clears out…. this is only speculation based on observation though I’m not sure if that is what is happening.
And for anyone wondering I running Valheim with a lot of mods (probably 20-30 mods), with Valheim+ and Epic loot being the main ones.
Same here. Please make sure you are using latest epic loot as older versions are causing what it looks like desync. really appreciate you trying to fix it along with James. thank you so much guys
The latest (0.8.4) version of Epic Loot still has desync issues. A guy on the Nexusmods forums for the mod had this to say:
“All out of sync from the terrible logic of syncing things between the player. All players are constantly exchanging with all packages with their things. This is necessary to highlight enchanted items on the player.
This logic clogs up all traffic. If usually ~ 20 kb, then with a mod ~ 80 kb per second. 1 The player weighs 0.1 mb.
Such a huge weight is achieved by the same terrible logic of Extended Items Framework. Tens of kilobytes of garbage are simply recorded in the field “CreatorName”.”
I don’t know if that’s accurate or not, but it certainly seems to jive with what we’ve been seeing, in that 1-2 people with lots of “enhanced” gear on can be in an area without issues, but the moment the third person shows up, it’s desync city. I’m hoping that some netcode optimization like what’s being discussed here will help.
Hi Greg,
See my reply to your other message which has our current settings, we have all these mods installed and running with 3-4 players in a single area… the data receive rate does sometimes get into the 300kbps per client when there are a lot of monsters running around though (We have a physical server with on 1000/50 connection, so you may want to keep this in mind depending on where your server is being hosted).
So our group has been dealing with a lot of bad lag. 50+ mods but it really kicked in after adding Monsternomicon. Going into new areas or getting a lot of mobs in the area at the same time would cause stuttering, teleporting, or invisible mobs. It was fine alone and when my friends played together they were fine. It would only fail when I joined. Their pings are 15-40ms and mine is 80-90 due to distance. It really seemed that this difference in ping time was the issue because even one other player and me on the server would cause it to lag. But 2 or 3 of them together was no issue.
I first tried setting some of the mods back. I tried premade work arounds that I now know are outdated. I found this walk through and tried the buffer extension above. No luck. But then after rereading this cap change from Patk88 I finally gave it a whirl. I added it to the server and the clients and zero issues now. Even mods like the seed totem that would show a delay when hitting the totem to pop the vegetables is gone. We hit the server heavy using everything we had at once and no hiccups.
Absolutely amazing Patk88! Bravo!
Thank you!
While I am no stranger to programming, it is not my line of work. I would suggest making a guide like the queue changes to make it easier for others to make this modification. It appears stable and without issues.
Hey Norby,
What a fantastic outcome! It’s very strange that raising the buffer size has such a dramatic impact. That’s… so easy. No game / mod I’ve ever worked on would have been that easy to fix lag in!
It honestly makes even less sense that the buffer size would impact your ping. Like that is just not how pings work and they’re tiny ICMP packets. I’m guessing what is changing is the way that Valheim measures and displays the ping. It must be getting bottlenecked without this change and it’s showing up in the in-game ping. I bet if you kept a command prompt window open with a ping -t yourserverip the ping wouldn’t change even when it’s lagging in the game meaning it’s a “Valheim ping measurement lag” thing and your “real” ping from a network level is not likely changing here (especially if the send/receive buffers are the bottleneck, literally has nothing to do with ping times).
The only explanation I can come up with is that the mod is so badly programmed (especially at the network level) that the buffers are literally bursting at the seams and breaking down the entire network engine to the point where even in-game pings are spiking etc. because this type of a change should *never* have this dramatic of an impact unless there’s serious design issues at play.
I’m definitely not opposed to writing up a guide for how to do this modification but let’s be clear that this is to fix essentially broken mods (which is normally the developer’s job). The game is fine. Most other mods also appear to be fine (except obsolete and outdated ones etc). It’s only the “Epic Loot” mod and a few key others that are not fine. I wouldn’t use them if there’s *any* alternative to it at all (even if it takes a half dozen mods or more better programmed mods to make up for all the functionality) because it seems like garbage from what people are having to do just to not have lag. I’m assuming there either are no alternatives or it would take a dozen mods to make up for it or something like that but some of these mods are in an unacceptable technical state.
Why don’t they just change the buffer size for you if this is necessary for it to even work when it’s already a massive game-mod making thousands of changes? Have any of you approached the developer or opened a GitHub issue? Why is this mod (and a few others) getting this free pass while we have to reprogram their mod / the game itself for them while they’re not being pressured to fix/improve it? You all have literally proven it’s the mods, which mods, and now have even proven how to fix it (or how to work around it at the very least).
I don’t see a single issue regarding the lag on on the first issues page: (actually I see one from all the way back in June, and it was never answered and pretty far down the page, and no one replied and backed him up which means the developer will assume they have network/hardware issues or at least I would without other voices echoing it). Like does this developer even know this is going on to this extent? I doubt it. There’s seemingly far more posts about it on this page than the official project’s GitHub page! This really looks like one of the cases where everyone has assumed the developer is getting swamped with issues about this and in reality nobody actually bothered to try to contact them / submit an issue. I was expecting 50% of the issues on the page to be lag related from how much activity we’ve had here about it and in reality there’s almost nothing on there.
I’m sure the features are cool, but this network mod is *extreme* (from a technical standpoint) and making up for very very bad mod behavior. If you absolutely have to have this mod then definitely this additional change is worth considering! I know you mentioned some other mods here too and I focused on the “Epic Loot” one but all of this would apply to those mods as well. I should say that with 50 mods though (wow, the last time I logged on 50 mods didn’t exist in total!) maybe you really are hitting the send/receive limits because that’s a lot of additional packets/data theoretically and each mod has it’s own overhead so it’s not as efficient as if these were all “official” features etc. and some mod developers are much better/more efficient programmers than others.
If you are running 50 mods on the server I would just go ahead and do this mod for sure rather than chase down individual developers but for some of these really popular/important ones I highly encourage people to start posting some issues because from what I’m seeing even the *big* Epic Loot mod is getting almost zero feedback about the lag they are causing and these other developers have probably never even been contacted about it before from the look of it. A lot of these could be fixed/improved I’m sure if the developers took a hard look at it! Fixing a few of the bad apples would probably let you get it to 100 mods without lag until you have to chase down the worst offenders again. This send/receive limits change is just raising the ceiling but those mods are still eating up all your bandwidth which will limit your total mod capacity at some point. It would be good to get them fixed/optimized even if you’re doing the mod because it gives you more “breathing room” and capacity for additional mods!
Thanks for leaving this feedback. This modification sounds like a winner so don’t take my message the wrong way. I’m more disappointed in the developers as a fellow developer and really think they need to clean this stuff up/improve it. This networking code is so sloppy you literally couldn’t have got away with it back when I developed games regularly a good 10-15+ years ago. People didn’t have the connections/bandwidth to handle turning it up and you had to do it the right way and at some point you’re going to run into this even today if the mods aren’t improved (they are setting your ceiling for you with the inefficiencies).
With this type of a model you’re only as strong as your weakest client. Which players have the least bandwidth on their connections? They’re the ones that will get hit first as you continue to turn up these limits and start reaching the limits of their connections. Because of the P2P client-to-client heavier model used in Valheim once one person starts lagging everyone will feel/notice some effects to varying degrees as clients start “disagreeing” on what is happening in the game etc. and I’ve seen some STRANGE behavior from this.
For an example of strange behavior from these disagreements apparently elites are spawned by the clients based on what biome you are in and one time I was with a friend right along the Mountains and the Dark forest biome borders and an elite spawned. My friend spawned the elite as a troll because on his screen we were in the dark forest but on my screen we were still in the mountains and spawned a drake. Except my drake was bugged and he OBLITERATED me with a single cold ball (with a full cold resist set on) while on my friend’s screen he was fighting a troll! When I logged out and back in I also saw a troll and my bugged unobtainium-powered-cold-ball one-shot-wonder Drake was gone leaving the troll my friend had always seen from the very beginning in his place.
It would probably honestly be easier for everyone to just make an official(ish) send/receive mod based on these discoveries. Maybe I’ll do it or if one of you who has participated/investigated wants to run with it then you absolutely could. I might be able to whip up a one-click send/receive patching utility as well. For people running 50 mods though it really shouldn’t be a big deal to install a send/receive mod on top of the other 50 and would probably make this easier overall!
I agree it’s an issue with the mods themselves and for the ones we like the best there are no other options. This modification has continued to work seamlessly. I did see a couple times the server ram fill up all the way. It’s rare and I restart it when it happens. It only happens with the mods active. I imagine there is also a memory leak somewhere.
Hey Norby,
Great find, memory leaks would explain a lot! Bugs and memory leaks happen all the time (as every gamer knows) even in supposedly highly polished games, so it’s not difficult to imagine that modding your code into someone else’s game makes these things even more likely to happen for sure.
I can’t find this field, but I’ve also never used DNspy and this is one of the most complicated interfaces I’ve ever used.
After I modded the DLL the game no longer starts. It works with the original DLL. The error message I see is:
Jul 23 13:08:32 valheim valheim_server.x86_64[2772]: (null)/.steam/sdk64/steamclient.so: cannot open shared object file: No such file or directoryJul 23 13:08:32 valheim valheim_server.x86_64[2772]: [S_API] SteamAPI_Init(): Sys_LoadModule failed to load: (null)/.steam/sdk64/steamclient.so
Jul 23 13:08:32 valheim valheim_server.x86_64[2772]: [Steamworks.NET] SteamAPI_Init() failed. Refer to Valve's documentation or the comment above this line for more information.
Searching for steamclient.so:
locate steamclient.so
/home/steam/.local/share/Steam/steamcmd/linux32/steamclient.so
/home/steam/.local/share/Steam/steamcmd/linux64/steamclient.so
/home/steam/Valheim/linux64/steamclient.so
/home/steam/Valheim/steamclient.so
Switched back to original DLL and it worked again. Running Ubuntu 21.04. Did the modifications on my Windows PC and then uploaded the DLL and set the original owner and file rights (steam:steam / 755).
Any ideas?
Hey Tomm,
I think I may be able to help, fortunately Linux should work and it has been done recently! The original DLL shouldn’t have anything to do with the Steam client. I take it you aren’t launching the dedicated server with a batch file on Linux but Steam is barely used at all for anything but authentication with it so if it’s just opened in the background you can launch the server directly and it will get whatever it needs from it (if you launch it without it open it will try to launch it to authenticate). Otherwise the server is just a black console window. I don’t even launch the dedicated server using Steam on Windows and I wouldn’t on Linux either as you can’t set any options that way (it should be pretty easy to modify the batch file examples for Windows to a bash .sh script) and this may be worth trying because getting a crash in Steam itself doesn’t make a lot of sense here.
I don’t think you’re even getting as far as the last person who tried this (also Ubuntu, but LTS 18.04) before they got a crash (but not in Steam, in the Valheim dedicated server) and they followed up and said they reinstalled everything from scratch and then the modification “just worked” after that. All of their output was from the Valheim dedicated server (as expected, this is the only application that uses that valheim dll) and not Steam.
The file should be almost identical to the original. It should be like two bytes different and a nearly identical size. Have you compared the sizes? You could try using ILSpy and doing the modification on Linux but it doesn’t make any sense for this Valheim mod to be crashing Steam itself and we’ve never seen this before. The other people on Linux had fairly standard output (from the Valheim dedicated server, the thing we’re actually modding, NOT Steam at all) and it’s difficult to explain why they got theirs working from a reinstall other than probably 100 different very minor configuration differences that could explain it. Did you try setting the value for the mod to something really unusually high? You may want to try a lower value for the mod if you went absolutely bonkers as that could be an OS difference between Windows and Linux.
Did you take the original file from Linux and then modify it or did you modify a Windows version of the DLL and upload that to Linux? Definitely make sure you start with an original Linux file. I’m not sure if they’re different for Valheim specifically since it’s written in .NET (there may still be some differences, Steam games/servers have to be specifically supported for Linux, compatibility with Mono stuff although that’s less of an issue these days, etc.) but the Steam ones are very extensively different linking to all kinds of Linux shared libraries so I would definitely use the original Linux DLL and not one from Windows to avoid running into anything strange (it should still be okay to modify on Windows, just copy it from Linux first).
Make sure you do a chown on top of the chmod so the right users own the files (I know you already mentioned the chmod but wanted to add the take ownership part as well as this could still be a simple permissions error but due to ownership).
ls -al
then replace youruser/group with the ones from the other files listed in the directory using this command:
sudo chown youruser:yourgroup valheimdllfile
now ls -al should show the same user / group / permissions (far left column) for all files in the directory.
double check that the permissions match exactly too -- yours may be right in most situations but make sure all the +xs, the group permissions all match etc.
ubuntu@ip-172-26-x-x:~$ ls -altotal 1430804
drwxr-xr-x 6 ubuntu ubuntu 4096 Jul 18 18:20 .
drwxr-xr-x 3 root root 4096 Jul 15 20:11 ..
-rw------- 1 ubuntu ubuntu 4746 Jul 23 00:56 .bash_history
I don’t want to speculate further because this doesn’t make any sense. The Valheim DLL we are modifying is *not* used by Steam. The only way this makes sense is if it actually *is* a permissions/ownership error. In this case maybe Steam is choking on validating the permissions/file ownership before launch (in a very, very ungraceful way) but it shouldn’t actually be using this file yet otherwise if that makes sense. It doesn’t use any code in the DLL but it may check before launching if it owns the files which could explain this. The ownership is probably root:root or youruser:youruser or something like that after reuploading it and it just needs to be set to the same owner/group as the other files because Windows doesn’t have/understand these types of file attributes/permissions (it has it’s own similar but different ways to do them) so they will be lost in this process.
This has been done successfully on Ubuntu Linux no more than 2 weeks ago and looks like permissions still to me from your output (and the old DLL fixing it). The more I think about it the more I think it has to be ownership/permissions as literally nothing else can explain this behavior that I can think of. The errors are not even happening in the right application they should be theoretically. The good news is it should be an easy fix. Hopefully all of that helps and let us know what you find!
i’m just curious, since it’s not ever explicitly touched on in this guide or on the original reddit post, but what does the 10240 actually refer to? rather, how does that translate to the kbps that most people refer to?
Hey rufi0,
Great question! The answer is tricky but I think we can grapple with it here.
So the first thing to say is that the old version worked a lot more like you’re saying where there was a strict kbps limit. The new version is kind of a hybrid of that system. It has a queue and that 10240 number is the queue size.
In the old version when this queue was full everything would start lagging. For a long time. You may not be able to open a chest for a whole minute for example. In this new version they added a “flush” flag where they can push through “critical” data and when the queue is full it actually DROPS your data. The routine just exits if the queue is bigger than the maximum queue size.
Most updates like this can be safely dropped (for example, it’s constantly sending your position to the server, if a few of those get dropped many many more will still get through). The ones that are too important to drop now use the “flush” flag and they will always be sent.
Ok, so with that explained we can attempt to grapple with the question. The 10240 number is the size in bytes of your TCP buffer. It’s basically a container and it can hold a maximum of 10240 bytes or 10.24 KB. Basically your send/receive buffer only holds 10KB of data. Once it has more data than that in there everything else is dropped unless it’s a critical update passed with the “flush” flag.
Because this is a two tiered system now this makes it impossible to say the exact KB/sec that this breaks down to because their “flush” command bypasses the limit so it’s more of a guideline in the new version. Increasing the size of this buffer basically gives you much more bandwidth to send “generalized” data that isn’t sent with the “flush” flag before you start dropping packets. You can feel when this happens because certain things you do will be ignored. The classic one used to be the chests. You would be able to see everyone moving but you could be trying to open a chest for more than 60 seconds while you can still see everyone run around.
In the new system I believe they made opening chests “flush” so that doesn’t happen as much anymore (and it was the most infamous send/receive limit problem at launch, nobody could open their chests in busy areas) but there’s still certain types of data that aren’t flushed. When your server is under high load you can actually tell which types of packets are being “flushed” and which ones are being dropped.
When packets are dropped your client has to “guess” a lot more on where things are. They will extrapolate where a monster will be for example and “guess” if the packets are being dropped. Then when the packets come through the monster might have to be moved if the guess was dramatically off from the server. Once the monsters position does come through you may see some jarring things as the client tries to figure out how to get the monster where it’s actually supposed to be vs. where it incorrectly guessed it was going to be. This works both ways too. The server and other clients also have to guess about YOU if you are the one dropping data so they’ll see you in slightly wrong positions as well. If you pay extra close attention you’ll see the tricks the game uses to try and move people/monsters back where they should be gently without it looking bad but if the position is wrong enough it will insta-teleport them and not even try.
So basically the best way to think of this new system is a “buffer” and the bigger the buffer is the more breathing room you will have under heavy load scenarios before packets start getting dropped due to these limits in the code. The bigger your buffer the less this situation will arise where players are dropping packets and “guessing” where things should be. The smaller the buffer the more packets will get dropped and the more your player’s clients will be “guessing” and “interpreting” player/monster positions and they won’t always be right on their predictions unfortunately. Hopefully that helps, sorry it’s tricky to explain and more complicated than it was at launch!
oh my gosh, thank you so much for that thoughtful and in-depth response! i was able to edit my server to sam’s 102400 below, and it seems to have alleviated much of the lag we experienced when we had 6 people online at once. i did experience the chest lag a couple times during our recent session, though that may have been attributable to some internet issues a couple of us were having at the time. i’m also thinking about compiling and sending an editted .dll to each user to see if that helps – since some of the users on the original reddit post had also tried it, and especially considering the next boss we have to tackle as a group is bonemass. any thoughts/recommendations in that regard?
Hey rufi0,
That’s fantastic, I’m glad the giant explanation ended up making enough sense that you were able to get something out of it! It’s a complicated topic for sure.
Your initial results sound right on the money here. It makes a great deal of difference but definitely isn’t a 100% silver bullet or solution for all situations or types of lag. It does make enough of a difference to make 6 people in a zone go from a nearly unplayable mess to pretty decently playable with some glitches here and there. Definitely an improvement worth the time because in my opinion having lag/bandwidth issues during these types of battles is not only immersion breaking but can approach game ruining levels depending on how bad things go as a result of it. Especially in Valheim where death is relatively unforgiving!
Okay, now your question about the client side mod in most situations it won’t make a difference
on a dedicated server. This is because when you’re in peer to peer mode or the non-dedicated server mode someone literally is the server so the reason the mod works in those situations is the same it works for the dedicated server.Now I said most situations, so what kind of situations might it make a difference in? Actually, funny enough, the most likely place it could make a difference is a giant fight with lots of minion spawns that are attacking everyone in the group with a lot of annoying AoE tick per second damage.
To my horror, damage in Valheim, like even taking damage, everything to do with it actually, seems to be 100% client side. Sometimes if there’s a giant 2 star nuking a stuck player over and over it will finally kill them but it takes a WHILE and I’m not sure if the server is finally calculating they should have been dead 5 hits ago or if packets finally got through or what is happening exactly. What is not happening though is the server is definitely not calculating damage and killing people, their clients are, and you can tell when they lag and they don’t die when they should have been dead 10 times over basically.
Because damage is client side in a situation like a Bonemass fight everyone’s clients are going to be sending *many* more updates per second / frame than they would just walking through the forest even with a couple of other people. Think of all the arrows flying around, damage from projectiles and all the poison slowly ticking away, all these things require the client to send an update to the server saying “hey this toxic puddle doing x ticks per second damaged me”, “hey this arrow hit”, “this healing potion was popped so show everyone the status effect” just to kind of give some less abstract examples of the client-side updates that could potentially hit that default 10,240 buffer.
In normal situations this would almost never happen but it could make a difference here in something like a Bonemass fight which has some of the most minion action and poison/toxic cloud damage in the entire game. There’s very few places it could potentially help more in theory or where there are more weird updates flying around during a fight that wouldn’t necessarily be present in other fights / normal fights.
Hopefully that helps and let us know if you try it if you saw the same (or any) improvement from modding the client side. The answer I’ve got from people in the past (like in comments on the article from before this, this is the revisiting with the fixed version that was changed after launch) is that it helps but it’s a pretty marginal improvement in some situations vs. doing it on the server which has a strong, noticeable improvement across the board pretty much!
‘When i put the modified dll in my server it does not start. I’m running my server on docker on a linux 18.04.
I’m getting that on server log:
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 39)[PrefabCreator] Added recipe: Recipe_FreezeGlandDrakeTrophy
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 39)
Unloading 92 unused Assets to reduce memory usage. Loaded Objects now: 92036.
Total: 139.312885 ms (FindLiveObjects: 5.078719 ms CreateObjectMapping: 15.522041 ms MarkObjects: 117.656501 ms DeleteObjects: 1.048291 ms)
Setting up 2 worker threads for Enlighten.
Thread -> id: 7fd30f7fe700 -> priority: 1
Thread -> id: 7fd30effd700 -> priority: 1
07/09/2021 10:36:34: Steam manager on destroy
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 39)
It works with the unmodified DLL but not with the modified DLL? The file should literally be like 2 bytes different from the original. If you put back the original file will it load? It definitely shouldn’t.
Did you modify the value to something crazy like way beyond what was used in the guide? Did you set it to something crazy like 1000000000 or -1 or something way outside a normal integer? Have you tried running this server on Windows by chance as a test?
We’re really not changing much here. Just one number twice. There shouldn’t really be any opportunity for things to be crashing from what we did unless the values are very strange. We changed two integer values and nothing else and if you followed the guide the code would literally just be comparing a slightly different number.
How did you even make this modification for Linux? Windows must have obviously been involved here somewhere to use dnSpy. Did you actually pull the file off the Linux machine to mod it or did you use a copy of the Windows dedicated server and then upload that modified DLL to a Linux system? You will definitely need to modify the Linux version of the file and not mix and match them. That is actually something crazy enough to cause something weird like this now that I think of it.
Can you show us your modification? I wouldn’t be surprised if restoring a backup fixes it and your whole server is crashed. It’s quite common which is why I emphasize the backups so much! The DLL literally shouldn’t make a difference here unless something went really wrong with the modification and I think there’s a good chance if you restore the old DLL it’s going to crash in the same way. It doesn’t make sense and something had to have gone wrong along the way different about your process vs. a typical Windows setup here.
So, i reinstalled everything and tried again, it worked. Now i will see if it can help us.
Thank you for the attention and for the article!
No problem at all, glad you got it working, thanks for letting us know what worked!
I had the same problem as werner. The problem was modding the client DLL instead of the dedicated server DLL. They are different, and only the server one can work in dedicated server mode. Starting with the dedicated server Managed directory instead of the Valheim client directory fixed it.
Hey David,
Thanks for chiming in, that’s very useful information. It’s also extremely easy to do on accident / assume they are the same. The paths / subfolders to end up getting to the client and server look remarkably similar and the .dll files are the exact same name (but NOT identical).
This should help others out as well, thanks again!
I’m currently trying this method. Don’t seems to make a difference for now though. Maybe my values are not high enough, I don’t know. It’s like in the 200000 for the client.
Hey Frank,
It may be possible you’re dealing with latency/lag that isn’t related to the send/receive limits as well. Is it everyone or one person?
One of my favorite tests is to have them open a command window and type “ping -t yourserver”. Have them leave this window in the background (or throw it on another screen if it’s a multiscreen setup). This command will ping the server over and over every 1 second.
When people are experiencing lag related to the network link/connection you will see the ping response times spike during lag periods. In that case the send/receive limits won’t help and a connection diagnosis/upgrade may be in order (sometimes as simple as a router reboot). If the send/receive limits are restraining the connection (or something else is wrong, more on that in a moment) the ping response times will never change as the network link is fine. If the ping times never change that would point to other causes for sure.
It could also be something unusual about the server / structures / terrain like if there is a massive amount of terrain modifications, giant castles the size of entire valleys or other things like that on the server or some other source of lag created from within the server (sometimes due to bugs as well). To test for that you would just temporarily start a new server (as a test) and have all the same people jump on and try doing some things. If no lag is experienced you can narrow it down to something inside the server data files (definitely this can and has happened for sure, early access etc).
Hopefully that helps and let us know what you find!
You are too cool!
I am currently running 102400 as the “magic number” and it appears stable, but more multiplayer testing is needed.
A few other server admins are testing this out as well (you can find them over at Valheim Modding in the #user-assistance channel). We are all grateful for your help!
Hey Sam,
That’s awesome! I’m curious what you will finally settle on for the best final number, that’s really interesting! Theoretically it may depend on how much bandwidth they have but I’m guessing there is going to be a sweet spot and that you’re probably already pretty close.
Thanks for the Discord link as well, that should help anyone else who is also looking for Valheim help!