Revisiting Fixing Valheim Lag – Modifying Send/Receive Limits
Published: June 15, 2021 / Last Modified: December 3, 2023 / Gaming / 232 Comments
My Valheim Dedicated Server Castle
I’ve previously covered fixing Valheim’s dedicated server lag by modifying the dedicated server’s send receive limits. Since I wrote that article not long after launch though the game has been updated that the old method no longer applies. I had theorized in the comments of that article how to address this and today Sam R. confirmed that this method did indeed alleviate most congestion on the server in the places that the send/receive limits are known to bottleneck.
This updated guide will show some updated methods you can use to help reduce lag on your Valheim dedicated server!
Symptoms of lag on Valheim dedicated servers
The symptoms you are experiencing on the server can help narrow down what type of lag you are experiencing and what may be causing it. Lag caused by your CPU being too overloaded for example will have different symptoms than lag that is caused by your server hitting the maximum send/receive limit as an example.
By far the most common type of dedicated server lag seen in the early access Valheim release so far has been mainly due to the low send/receive limit if 64KB/s that is hard coded into the server. This means that when you do something like take your character a heavily populated area on the server it has to send all that information about all the buildings/players through an artificially limited very narrow 64KB/s pipe.
Symptoms of the send/receive limit type of lag include things like chests taking a very long time to open, sometimes 10-30 seconds or never opening while at the same time being able to see players walking around normally. This is because the other people’s X / Y / Z positions is a very small amount of data compared to all the items/names/attributes that are stored in the chest. The other players moving easily fits through the narrow pipe while the chest full of loot’s data chokes since it is much larger.
I’ll specifically lay out how to modify your server and increase this limit in the “Modifying your server’s send/receive limit” section below. Before that though let’s cover a more specific type of lag to watch out for (mostly by being avoided) related to terrain manipulation.
Avoid extensive terrain manipulation (for now)
Another commonly observed source of lag has to do with terrain manipulation. Some players have found that if they take a big chunk out of the ground in Valheim and make extensive terrain modifications that the area gets filled with separate “instances” of terrain for each modification people make. Think of the original ground as one big piece in the game files. Valheim has the very cool feature of letting you modify terrain by raising it/lower it/level it/etc. Each time you do this though the game has to store what you did as a new “instance” or a modification.
The way Valheim seems to work now is it renders that original piece and then the modifications afterward. Early reports suggest having a whole bunch of them in one spot seems to sometimes cause everyone’s in-game FPS on the server to drop as low as 40 when the clients normally are much much higher than that.
The best way to avoid this type of lag at the moment is to limit how much terrain modification you are doing in areas near your base. It is likely this will be addressed/eased in future patches. There should be some things the developer team can do to clean up/merge these instances or at least reduce their effect on the rendering process so it doesn’t drop the in-game FPS so dramatically.
At this time there is no options file for Valheim unfortunately. The only way to change it is to modify the server’s code and recompile the module that controls this.
Fortunately this is a lot easier than it sounds because the module we need to change is in .NET instead of something like C++ which would be much more difficult to do this with. I did not come up with this method, a very clever user on reddit (maximgame) shared it here and I want to give credit where credit is due!
Despite it not being too difficult these modifications are at your own risk. I haven’t experienced any problems so far but technically we are modifying the server’s code with the following procedure so understand that this isn’t an officially supported method and could have side effects or if you make a mistake could cause serious problems. Make sure you have backed up everything (your game world files especially) before you try anything!
Another important note is that it’s very likely you will have to make the changes again each update since the file we are going to modify is going to get replaced by updates. Until they add either a launch option or options file to officially change/control this property that is unfortunately unavoidable.
With all that being said if you understand the above and still feel comfortable let’s proceed!
Get dnSpy utility
We are going to use the utility dnSpy to modify the server code and recompile the module. The utility is free and is available here (download the Win64 build or dnSpy-net-win64.zip): GitHub dnSpy Official Page
Extract the archive and run the program dnSpy.exe from inside the folder you extracted. The dnSpy application will open.
Modifying server with dnSpy
We are going to go to the “File” menu in the top left and choose “Open” like this:
dnSpy – File -> Open Menu
Now you need to navigate to the Valheim dedicated server folder which is located in your “steamapps” folder. If you left all the options default when you installed it’s typically “C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed”.
We are looking for the file “assembly_valheim.dll” in that folder:
dnSpy – Selecting Valheim server assembly file
Select the file “assembly_valheim.dll” and click “Open” and the server’s assembly will load into dnSpy.
Once it has finished loading we are going to go to the “Edit” menu and click “Search Assemblies” like this:
dnSpy – Search Assemblies
For the search criteria enter “ZDOMan” like this:
dnSpy – Searching for ‘ZDOMan’
After the search completes you should see a list of results like the one above. Double click on one of these entries on the list and it should place you in the ZDOMan section of the assembly (seen on the left hand side of the screen):
dnSpy – ‘ZDOMan’ section of assembly
Perfect. We are now in the right place to make the changes. We just need to locate the line that has the send/receive limitation. To find that press Ctrl+F to bring up the find box (or choose “Edit” -> Find from the menu) and search for:
sendQueueSize
which should bring up the following lines:
dnSpy – sendQueueSize Variable
If you are having trouble finding the lines make sure you’re using Ctrl+F to search this time and not the “Search Assemblies” from earlier at the bottom of the screenshot, that’s a different type of search. Make sure you’ve disabled the case sensitive options (the Aa right below the box) or any other search filters. Your search query box should be in the top right of the screen like the above screenshot that has sendQueueSize in it.
Now right click on this line:
int num = 10240 - sendQueueSize
and click “Edit Class (C#)…”. Another window will open that will let you make changes to the file. We are going to modify ‘10240’ to a variable of our choosing. I made mine 30720 since after doing some conversion math that is the equivalent increase from my previous launch version guide that provided great performance and I kept them the same. If you followed my last guide this will give you the exact same performance as the old one did.
There are two instances of this variable. The second instance is 3 lines above the highlighted line we searched for and is:
if (!flush && sendQueueSize > 10240)
Modify both instances of 10240 to 30720.
Theoretically setting a limit too high could cause the server to swamp your internet connection if you have a large number of players on and there’s essentially no limit but there shouldn’t be any other effects of a very high number.
Once you’ve changed both numbers to the desired value we need to press the “Compile” button in the bottom right corner of this new window. This window will close and take you back to the previous one.
Saving the new assembly
It’s time to save our changes into a new .dll file. Choose “File” -> “Save Module…”:
dnSpy – Save Module as assembly_valheim_modded.dll
Change the file name to assembly_valheim_modded.dll. Don’t overwrite the original file just yet as we will want to take a backup and make sure that the server is closed before we do that. Press “OK” to write the new assembly_valheim_modded.dll file.
Backing up original file and replacing with our new one
Now it’s time to back up the original file and replace it with our new modded file. First make sure your dedicated server is closed otherwise the dll file will be in use. It is also a good time to make a backup of your world file just in case (see my Valheim Dedicated Server Backup Location / Guide for instructions)
Navigate in a file explorer window to your Valheim server directory where we saved the new assembly file (usually C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed).
First make a copy of the original assembly_valheim.dll file. You can just select the file and “copy” then paste a copy of it in that same folder and Windows will make a “valheim_assembly Copy(1).dll” file for you or you can back it up to a separate folder. Now remove the original file and rename your “valheim_assembly_modded.dll” file to the original file name of “valheim_assembly.dll”.
That’s it, you’ve now replaced the old assembly with the new one! Go ahead and start your server normally and the changes will have taken effect.
If you know of other Valheim lag issues that I didn’t cover here or are still having problems let me know in the comments and I will update this guide with other tips/tricks and try to answer any questions. It’s still a very new game and there isn’t very much out there for it yet. Have fun and enjoy!
Two short question:
At the moment I’m running a dedicated Server with 5-7 players (mostly we play in one area) / no mods.
With the same players we played a year ago on a dedicated server also again via Gportal, but we all note much more lag than last year.
1. First of all: is this whole fix still a thing after the Patch 0.211.9 dropped on October 11, 2022 in which the developers stated exact this issue that “majority of the changes include stabilising connection issues, as well as fixing send/receive limitations in multiplayer.”
2. I tried your solution (only the server-sided) and it seems to help a bit, especially in the base and with chests as described, but I’m not 100% sure, because I don’t get the results as I expected overall. Still opponents disappear or sink into the ground when 3 people are in one area. 3 Players should be no problem, never was. Haven’t discovered this problem last year with the exact same configuration and the same people playing.
When I read all the comments here and from modders on Github, I realized that a client can quickly become a bottleneck for lag. So I checked, but everyones connection is fine/game runs fine.
So the bottom line is that the game has become more worse in terms of multiplayer dedicated server, even if they made a patch to fix exactly these things? Is this true?
PS: Saidly, I have no documentation about my last year Gportal server. Maybe they got more worse since last year, but it would be cool to be sure about my second question, so i can eliminate the gamecode itself before spending more money or get another host. Thanks in advance.
Welcome! So theoretically if they fixed the send/receive limitations that would be the same fix as this article. It’s always been really complicated though and it has changed a few times. This method is actually different than the original method and it sounds like that patch from October 11th most likely implemented these changes.
I can’t find very much online about people’s experience with the patch specifically related to this issue. Are you using a fresh map by chance or are you using your original map? There’s a lot of terrain stuff in older maps that can cause massive lag. It may be worth trying with a clean server (even if it’s just as a test) to see if the lag might be related to your server files (something we’ve seen before).
All data on the Valheim servers is sent between client <-> server.
If a single player is alone in one area, all creatures and objects are controlled and synchronized with the server through this client.
Even if a second or third player joins the area, the first client will still be the one doing the controlling and synchronizing. Therefore, the connection of the first player is crucial. If this player has a bad connection, the other joining clients will notice this as well. These are the lags and desync-problems some players are experiencing, even if their internet is fast and runs without any problems.
We are sure, that the game devs will find a solution for this problem.
This notice is still live on their site. That may have just not been removed after October 11th if it was actually fixed but it also may mean that this can still be be an issue. It seems like either there hasn’t really been a change or that maybe they haven’t updated it since the patch (but your experience seems to suggest there may still be issues).
In fact the issue they are describing is different than send/receive limits. It is related because the send/receive limits were artificially low (as in lower than the amount of data the game is expected to send under normal operation with only a handful of people on) so they could cause this condition on otherwise perfect connections. That’s just a factor though. If people’s connections are actually lagging I don’t think they redesigned the multiplayer logic for this game and that it still works this way.
It would be a major overhaul to completely change their multiplayer system entirely. This just sounds like a tweak (just like my original mod and this were, slight tweaks). The tweaks can help but only a different multiplayer server model would solve the issue entirely as this is such a P2P-driven server model (which perplexes me to this day why they would use an O.G. original Xbox P2P-style multiplayer system). In my opinion they need to overhaul this system and not tweak it.
It still suffers from the issue of it will be as laggy as the slowest player’s connection (which I don’t believe has changed, even though the limits are higher and not as artificially low). People can still have a low-bandwidth/flaky connection and everyone else has to wait for that connection to sync with the server and their clients due to Valheim’s multiplayer model. I would describe it like Xbox and PS4 multiplayer gaming where someone else’s console is literally hosting the matches (and when they leave everyone has to migrate, that kind of system).
Valheim’s system is only a step above that since the dedicated server is actually in charge of storing the map and items (at least in chests and things like that). It’s not fully peer-to-peer but it’s inappropriately close still to this day. It’s silly that it works this way when we have an actual dedicated server. The Valheim dedicated server is really just acting as a P2P connection coordinator (and storing the map and items in chests) when really we would like it to act more like a traditional dedicated server where it does everything and player connections have nothing to do with the server-wide lag level.
I know you checked everyone’s clients and that nothing came up unusual and everything looked okay. That’s why I suggested trying a new server as a test and a few other tests because it’s not a 100% certainty it’s one of your players causing it. I just wanted to explain why when this happens it’s one of the first places we want to look. The Valheim multiplayer code is still very P2P driven to the extent that the server is not actually moving enemies around or anything like that and it is all controlled by people’s clients (believe it or not, in 2022 still).
In other words yes, it might actually be worse. If all they did was implement my send/receive limits fix in the multiplayer code we have been doing that for a long time and it never solved the problem entirely because of all the reasons I’ve outlined here (it’s the system itself). Since new features have been added since that time and the core system itself is the same it’s possible (even likely) that there’s more data being exchanged between the clients/server and the same underlying issues (which would in fact be worse on average). We saw this a lot when people started installing too many mods. You’d hit a point where the P2P-style networking would start bottlenecking due to the extra data from the mods flying around.
Hello James,
Thanks for the quick and insightful answer regarding the code mechanics.
To be honest: I’m very perplexed how fast and simple it’s possible to get an detailed, clear, targeted answer without any sham substance in between, providing more information than I got from anyone on the official discord in a whole year.
I’m apparently just not used to that kind of service anymore or that someone is seriously taking action about your problems, but I highly appreciate this. Thank you very much.
To answer your question we started on a fresh installation and with new characters. Should it become unplayable we maybe restart and I will try another host, but for the moment and after your fix it’s “more likeable” and we passed the 2nd boss already.
It just caught our attention that the “overall experience” is worse than last year.
But after all information I got now, my best bet is the same as yours and I think the game itself has an increased data transfer compared to last year with all features added which totally makes sense, but on the other hand causes in combination with P2P driven game code an overall worse experience. Don’t get me wrong – it’s not unplayable after all, but it was unfavorable last year with up to 4 or more players and now it is even less advantageous.
If you are 3 people in one area and another one joins us, you can feel an instant drop of stability in the connection regardless what you configured (or if you let someone with ftth connection join first / or if you exclude the one with the “worst” ping – makes no difference to this effect). Last year it was not noticeable to the same extent. 4 players now is like playing with 6-7 players last year if I have to compare it. But what you said makes absolute sense to me.
“I’ve been waiting for Mistlands so I have not tried this update yet. It can’t be too much longer before Mistlands hopefully.”
Very optimistic. I think they hold it till the X-Box release. Maybe it takes so long because they fix exactly these issues we talk about behind the doors for more sensitive console players.
But this is wishful thinking – in reality, I don’t think they will rewrite the code to this degree.
Thanks for all informations and keep up your good work! It’s rare these days.
Thanks so much for that kind compliment, that is very much appreciated! I’d imagine it comes from me being a L3 hardware tech before I started doing the blog full time as it was literally my job to help people like this so it’s almost hard-wired into me.
So if you guys have already started fresh recently I don’t think this will have any impact. Supposedly a lot of the earlier terrain modification issues that caused lag have been mostly fixed (my understanding is it combines and cleans up those meshes now and previously it was a giant mess). There were also numerous bugs in the map files that have been fixed over time to the extent that simply starting a new map you will be much less likely to encounter problems (sometimes weird ones you won’t find any information about anywhere).
I think you are right on the money here on your understanding of the problem. There’s slightly more data flying around related to the new features added since the big launch rush that so many of us played. You can actually train yourself to see/feel this in-game and it sounds like your senses have become attuned to this. You’re noticing the difference/strain when a 4th person jumps into your area.
All of these clients are communicating with each other and their clients are essentially negotiating what the monsters are doing and a whole number of other things. The problem with this model is that it doesn’t scale. You can literally feel the weight of it crushing your network performance for each individual person that joins the active area.
If you have 2 other people in the area then your client is negotiating/talking with 2 other clients. That’s not too crazy but when you have 5 people in the area all talking with each other that is 25 connections (plus 5 to the dedicated server itself) and most things in the game will only be as fast as the weakest link of any of those 25 connections. That’s what I mean by it not scaling. Adding additional players does not increase the network load in a linear way since each person has to P2P communicate with everyone else that came before them (and everyone in the area gets another person they have to communicate with in addition to everyone they already were).
Maybe all of your players have a great connection to the dedicated server itself. Do they all have a great connection to each other though? Every single player to every single other player? That will be the weakest link then and the single-cored nature of the game engine (but not as bad as release, some things are multithreaded now like I want to say opening/closing chests specifically was one they patched a while ago) means it will all wait for that link to sync up and resolve whatever it’s trying to do basically. It’s pretty hard to test this too vs. testing everyone’s connection to the dedicated server itself. If it wasn’t for the P2P model in this then the connection to the dedicated server would be the end of the story but the P2P means everyone’s connections matter not only to the server but to each other.
You may very well be right about Mistlands! It’s shocking to me that this still isn’t out yet. Early on they were hiring a bunch of people because they originally were only a team of 4 when they released the game. The whole studio was 4 people. They have a lot more than that now though (and they’ve had time to train them) so I’m not sure why the content train isn’t showing up yet still.
Thanks again for your reply, take care!
Kush
2 years ago
Writing assembly_valheim_modded.dll…
Access error: Access to the path 'C:\WINDOWS\system32\assembly_valheim_modded.dll' is denied.
Getting this error, would appreciate it a lot if someone helps (:
It looks like you’re trying to save the Valheim dll into the C:\Windows\System32 folder. I would just try saving it to the desktop or somewhere like that as the system32 folder is a protected system location and is basically giving you permission errors. If you save it somewhere else it should go!
Thanks a ton for linking to this. I know I had asked earlier if anyone had contacted the dev about this specific issue. You gave a lot of technical details and it looks like it’s going somewhere. Great work, you may have made the ultimate difference here and are actually going to fix this (for everyone) after months and months of this being an issue!
Looks like it is patching the two methods using harmony just like we have been doing manually, should work in the same manner… just make sure the host and server are configured the same.
Does it work? and if yes, how to do it.. i downloaded the NetworkFix in thunderstore but there is no readme or howto anywhere. Does it increase those limits as sonn as i launch the game with it? Is it permament? Do i have to configure those values somehow?
1010101110
2 years ago
this doesnt seem to be the case for me, this is on a modded server but without changes to sendqueue or socket parameters:
Without the changes to the send queue size why would it change? How would it change? How would more data be flowing if it was the same size?
How would the clients know to send you more data? They don’t basically, and if you don’t modify the clients they will be sending the wrong size queue and we have seen what happens in this situation (nothing good).
You need to modify all clients as well if you’re going to change the send/receive window size to the extent it will show on a graph instead of just loosening things up on the server. If they don’t match between server / clients it will be worse than not doing it all as there will be packet loss. Patk covers this really well if you want the graphs to show more throughput!
It isn’t really a throughput issue though. It kind of is but it’s artificial constriction and actually has nothing at all to do with your connection or any player’s connection. Every single one of your player’s connections has enough bandwidth to send 1000x the data Valheim is sending. It’s actually really complicated why it happens but if I were to try to sum it up Valheim doesn’t leave a lot of extra room in the packets for “extra” stuff from mods. You hit a point where the networking code is no longer behaving efficiently and basically lags/chokes itself to death (on purpose, it’s trying to limit the flow of data because it doesn’t understand why there’s this much of it, it doesn’t know about your mods). The conditions are created artificially basically. Valheim wasn’t designed to transfer that much data so their code / numbers don’t facilitate it and things start going very wrong even though it’s honestly not a lot of bandwidth. Run a bandwidth monitor on your connection and you will see it’s nowhere near saturated.
It’s essentially a design issue that Valheim coded hard limits that fit just their needs but don’t adapt / can’t be changed if you are doing anything like running additional mods. If they even gave us an option to have a larger window in the options none of this would be necessary at all (like many multiplayer games do).
The mod from this article loosens these restrictions (it raises the standards before packets start to be dropped). Will you see it in the graph? No, because we didn’t change the send/receive limits for the in article mod, we only changed the criteria before the server starts dropping packets if the queue is already a certain size (meaning after my change it will try to send those packets instead of drop them and resend them again later). There’s no way for more data bits to physically flow in a way that will show on the graphs though unless you actually change the window size on the server and all clients.
Does it help just doing the server mod from the article? Definitely. It helps because it isn’t a connection-related throughput issue that is causing the lag, it’s a design issue and this alleviates some of that poor design. The reason you won’t see a change on those throughput graphs though is that they aren’t measuring what we changed. Before the mod those packets would have been dropped and resent. What does this look like? It would just look like your packets being more “full” and closer to the limit next time. The packets will just keep getting dropped until there’s enough free extra space left over on another packet that it can fit.
This is the vicious cycle of lag that starts that you can’t really get yourself out of and is kind of a death spiral. The article mod greatly increases the size of the sendQueueSize before it will try dropping *anything* and will just ram them through. There’s limits to this though and people running 50+ mods will hit the TCP window size 100% of the time and have no choice but to do the client-based mod as well or it will be unplayable. There won’t be any packets with extra free space to piggy back on ever basically and that is when you will reach the breaking point.
It will not measure though on your graph because what you are measuring isn’t the actual problem / cause of the lag but just another side effect/manifestation of the underlying reality that Valheim’s networking code wasn’t designed for the amount of data you’re sending over it. To observe this effect on a connection monitor what you would see is that the size of the Valheim packets would be creeping closer and closer to 15000ish bytes. They will never cross that because there’s nowhere for more bytes to go, that is the limit. If you want to break that limit, you have to change that limit on all clients. Being close to 100% full on each packet would indicate poor health. There should be empty space and there won’t be when these conditions start.
Another more extreme way to alleviate it is to literally change the send/receive window size in all clients / the server. If you want to literally change the limits so they will show up on a graph you will be REQUIRED to modify every single game client every single patch but it’s possible and well covered here! It has to be that way. There’s no way to just do this on the server side because your clients won’t know about the change. Hopefully that helps!
i do see sometimes players will “desync” from the server. i’m guessing they are lagging and their queue gets so full that it just is perma full and drops new packets.
so in that case increasing the queue size might help them recover. but still it will hit a limit eventually. i don’t want to fill up their memory.
That sounds about right. The article modification will get you to that point and then patk’s modification will let you raise that ceiling further. You are right though, at a certain point a lot of other resource constraints come into play but you could probably get away with raising the send/receive limits multiple times.
It sounds like some progress is being made on getting the code running the mods improved. Dayaleth just posted an update on a thread showing the dev looking at it that looks promising!
AlistairChiss
2 years ago
No matter how much I increase the value it appears that my data send and receive are capped at 150kbps. This definitely appears to be helping with stability but at least for me it doesn’t seem to actually be increasing the throughput. My group and I are seeing 150kbps as caps for send and receive even when using a value of 512000. We are using Valheim+ and EpicLoot both.
That sounds exactly right. If you want to change the #s that show in the actual then check out patk’s mod here in the comments (pretty popular for servers using a lot of mods). It requires modifying all game clients as well for each player otherwise their send/receive window size wouldn’t match (not good, we’ve seen it before during testing, lots of packet loss and things are overall slightly worse) but with everyone having it done correctly those #s will show as higher in those windows.
Basically this mod used to modify the send/receive limits when Valheim first came out but the code was changed requiring a rewrite and this new version is more like tweaking their pressure release valve to let more through before packets start getting dropped. It’s kind of a misnomer though because they changed how the Valheim code worked so the version in the article doesn’t technically do that anymore (as it requires a mod on all clients in this version) but you can still literally modify the send/receive limits (it’s just a little bit more in depth in the current version). Hopefully that helps!
LordAg
2 years ago
Thought I’d add a weird foible I’ve come across since the H & H update and have not been able to figure out at all. I’m going to detail what we observed and see if anyone else has ideas, because we are stumped. We had made the changes successfully and they worked fine for a while, but lately we’ve been having an odd issue with the Spawning Stones and the area around it. I’ll try to describe it as best as I can. The first person that logs in to the stones loads them fine, but anyone else that loads into the stones can’t see them or any structures. They can’t see it UNTIL the person the it was initially loaded for leaves the area. The person that can see the terrain can log out, teleport, or even run away, but once they are out of range, the terrain appears for everyone that couldn’t see it. This seems to be an issue only with the Spawn Stones, as we’ve tried to replicate this in other areas and it doesn’t happen. It also seems to happen strictly on logging out and logging in. Once that portion of the world has loaded in for someone, it remains loaded for them whether they teleport away or run away.
It also happens even if you don’t log in near the stones, but someone is currently at the stones when you login. We had a teleporter set up from one island a significant way away back to the Spawn Stones. If someone was already at the stones when you logged in, even though you weren’t anywhere near the stones, if you tried to take the portal back to the stones it would not work despite showing that it was connected. Basically the portal didn’t “exist” on the other side for you. You COULD destroy the portal, rebuild it with the same name, and THEN it would work, but if you went through, ONLY the portal would exist within the area of the stones and the nearby terrain would not load in, again, until the person it was initially loaded for left the area. I believe it’s because the object was interacted with by linking the portals. We noticed that when the world wasn’t loaded in for someone who logged in, if a person interacted with one of the objects that had not spawned in, it would appear. For example: if I loaded into the spawn stones unable to see anything because someone else had been in the area when logging in, objects that THEY interacted with would appear, such as a door opening or a tree being hit with an axe. Enemies did not disappear, strictly the stones and surrounding landscape.
Now, we did try several things to attempt fixing it: pulled out mods one by one, reverified the files then added the modded assembly_valheim.dll files back in. The thing that seemed to work was using vanilla, unmodded assembly_valheim.dll files. I know it’s a bit long, but I just could not figure our exactly what was going on.
I have observed similar weirdness randomly near the spawn stones also, for us it is more like a delay in the server sending the data to the client (the rec rate is usually sitting around the 0.1 – 0.5kbps mark), after 5 – 10 seconds on approach to the area all the trees / structures suddenly load in with a big burst of incoming data. The spawn area is the only place we have seen this behaviour so we haven’t been too concerned getting to the bottom of it (may even be related to a mod rather than the changes listed here).
Fast reply, ty. Yea, we can play around it pretty easily. The frustration was just trying to figure it out. It’s like an annoying hangnail: a minor inconvenience but one that can be played around. At least now we know for sure it’s the spawn and can mostly avoid building around it or having to much traffic around it.
I have found that this issue doesn’t seem to present if you take a portal to the spawn stones, it only appears for me if you approach the stones via walk/running.
wait what? this is the thread I was searching for. We built our base around said stones and some people noticed that terrain changes around that area wont load in or half the houses disappear. But once the host leaves that area it return to normal, only in rare cases a roof actually breaks. So we tried 3 different network mods with all kinds of settings and observed the same thing every time.
Can you list your mods so we can find a possible culprid? Or can anyine try Analyze what is happening here?
Two short question:
At the moment I’m running a dedicated Server with 5-7 players (mostly we play in one area) / no mods.
With the same players we played a year ago on a dedicated server also again via Gportal, but we all note much more lag than last year.
1. First of all: is this whole fix still a thing after the Patch 0.211.9 dropped on October 11, 2022 in which the developers stated exact this issue that “majority of the changes include stabilising connection issues, as well as fixing send/receive limitations in multiplayer.”
2. I tried your solution (only the server-sided) and it seems to help a bit, especially in the base and with chests as described, but I’m not 100% sure, because I don’t get the results as I expected overall. Still opponents disappear or sink into the ground when 3 people are in one area. 3 Players should be no problem, never was. Haven’t discovered this problem last year with the exact same configuration and the same people playing.
When I read all the comments here and from modders on Github, I realized that a client can quickly become a bottleneck for lag. So I checked, but everyones connection is fine/game runs fine.
So the bottom line is that the game has become more worse in terms of multiplayer dedicated server, even if they made a patch to fix exactly these things? Is this true?
PS: Saidly, I have no documentation about my last year Gportal server. Maybe they got more worse since last year, but it would be cool to be sure about my second question, so i can eliminate the gamecode itself before spending more money or get another host. Thanks in advance.
Hey Gesper,
Welcome! So theoretically if they fixed the send/receive limitations that would be the same fix as this article. It’s always been really complicated though and it has changed a few times. This method is actually different than the original method and it sounds like that patch from October 11th most likely implemented these changes.
I can’t find very much online about people’s experience with the patch specifically related to this issue. Are you using a fresh map by chance or are you using your original map? There’s a lot of terrain stuff in older maps that can cause massive lag. It may be worth trying with a clean server (even if it’s just as a test) to see if the lag might be related to your server files (something we’ve seen before).
I’ve been waiting for Mistlands so I have not tried this update yet. It can’t be too much longer before Mistlands hopefully. As far as Gportal specifically I was able to find this:
This notice is still live on their site. That may have just not been removed after October 11th if it was actually fixed but it also may mean that this can still be be an issue. It seems like either there hasn’t really been a change or that maybe they haven’t updated it since the patch (but your experience seems to suggest there may still be issues).
In fact the issue they are describing is different than send/receive limits. It is related because the send/receive limits were artificially low (as in lower than the amount of data the game is expected to send under normal operation with only a handful of people on) so they could cause this condition on otherwise perfect connections. That’s just a factor though. If people’s connections are actually lagging I don’t think they redesigned the multiplayer logic for this game and that it still works this way.
It would be a major overhaul to completely change their multiplayer system entirely. This just sounds like a tweak (just like my original mod and this were, slight tweaks). The tweaks can help but only a different multiplayer server model would solve the issue entirely as this is such a P2P-driven server model (which perplexes me to this day why they would use an O.G. original Xbox P2P-style multiplayer system). In my opinion they need to overhaul this system and not tweak it.
It still suffers from the issue of it will be as laggy as the slowest player’s connection (which I don’t believe has changed, even though the limits are higher and not as artificially low). People can still have a low-bandwidth/flaky connection and everyone else has to wait for that connection to sync with the server and their clients due to Valheim’s multiplayer model. I would describe it like Xbox and PS4 multiplayer gaming where someone else’s console is literally hosting the matches (and when they leave everyone has to migrate, that kind of system).
Valheim’s system is only a step above that since the dedicated server is actually in charge of storing the map and items (at least in chests and things like that). It’s not fully peer-to-peer but it’s inappropriately close still to this day. It’s silly that it works this way when we have an actual dedicated server. The Valheim dedicated server is really just acting as a P2P connection coordinator (and storing the map and items in chests) when really we would like it to act more like a traditional dedicated server where it does everything and player connections have nothing to do with the server-wide lag level.
I know you checked everyone’s clients and that nothing came up unusual and everything looked okay. That’s why I suggested trying a new server as a test and a few other tests because it’s not a 100% certainty it’s one of your players causing it. I just wanted to explain why when this happens it’s one of the first places we want to look. The Valheim multiplayer code is still very P2P driven to the extent that the server is not actually moving enemies around or anything like that and it is all controlled by people’s clients (believe it or not, in 2022 still).
In other words yes, it might actually be worse. If all they did was implement my send/receive limits fix in the multiplayer code we have been doing that for a long time and it never solved the problem entirely because of all the reasons I’ve outlined here (it’s the system itself). Since new features have been added since that time and the core system itself is the same it’s possible (even likely) that there’s more data being exchanged between the clients/server and the same underlying issues (which would in fact be worse on average). We saw this a lot when people started installing too many mods. You’d hit a point where the P2P-style networking would start bottlenecking due to the extra data from the mods flying around.
Hopefully that helps!
Hello James,
Thanks for the quick and insightful answer regarding the code mechanics.
To be honest: I’m very perplexed how fast and simple it’s possible to get an detailed, clear, targeted answer without any sham substance in between, providing more information than I got from anyone on the official discord in a whole year.
I’m apparently just not used to that kind of service anymore or that someone is seriously taking action about your problems, but I highly appreciate this. Thank you very much.
To answer your question we started on a fresh installation and with new characters. Should it become unplayable we maybe restart and I will try another host, but for the moment and after your fix it’s “more likeable” and we passed the 2nd boss already.
It just caught our attention that the “overall experience” is worse than last year.
But after all information I got now, my best bet is the same as yours and I think the game itself has an increased data transfer compared to last year with all features added which totally makes sense, but on the other hand causes in combination with P2P driven game code an overall worse experience. Don’t get me wrong – it’s not unplayable after all, but it was unfavorable last year with up to 4 or more players and now it is even less advantageous.
If you are 3 people in one area and another one joins us, you can feel an instant drop of stability in the connection regardless what you configured (or if you let someone with ftth connection join first / or if you exclude the one with the “worst” ping – makes no difference to this effect). Last year it was not noticeable to the same extent. 4 players now is like playing with 6-7 players last year if I have to compare it. But what you said makes absolute sense to me.
“I’ve been waiting for Mistlands so I have not tried this update yet. It can’t be too much longer before Mistlands hopefully.”
Very optimistic. I think they hold it till the X-Box release. Maybe it takes so long because they fix exactly these issues we talk about behind the doors for more sensitive console players.
But this is wishful thinking – in reality, I don’t think they will rewrite the code to this degree.
Thanks for all informations and keep up your good work! It’s rare these days.
Hey Gesper,
Thanks so much for that kind compliment, that is very much appreciated! I’d imagine it comes from me being a L3 hardware tech before I started doing the blog full time as it was literally my job to help people like this so it’s almost hard-wired into me.
So if you guys have already started fresh recently I don’t think this will have any impact. Supposedly a lot of the earlier terrain modification issues that caused lag have been mostly fixed (my understanding is it combines and cleans up those meshes now and previously it was a giant mess). There were also numerous bugs in the map files that have been fixed over time to the extent that simply starting a new map you will be much less likely to encounter problems (sometimes weird ones you won’t find any information about anywhere).
I think you are right on the money here on your understanding of the problem. There’s slightly more data flying around related to the new features added since the big launch rush that so many of us played. You can actually train yourself to see/feel this in-game and it sounds like your senses have become attuned to this. You’re noticing the difference/strain when a 4th person jumps into your area.
All of these clients are communicating with each other and their clients are essentially negotiating what the monsters are doing and a whole number of other things. The problem with this model is that it doesn’t scale. You can literally feel the weight of it crushing your network performance for each individual person that joins the active area.
If you have 2 other people in the area then your client is negotiating/talking with 2 other clients. That’s not too crazy but when you have 5 people in the area all talking with each other that is 25 connections (plus 5 to the dedicated server itself) and most things in the game will only be as fast as the weakest link of any of those 25 connections. That’s what I mean by it not scaling. Adding additional players does not increase the network load in a linear way since each person has to P2P communicate with everyone else that came before them (and everyone in the area gets another person they have to communicate with in addition to everyone they already were).
Maybe all of your players have a great connection to the dedicated server itself. Do they all have a great connection to each other though? Every single player to every single other player? That will be the weakest link then and the single-cored nature of the game engine (but not as bad as release, some things are multithreaded now like I want to say opening/closing chests specifically was one they patched a while ago) means it will all wait for that link to sync up and resolve whatever it’s trying to do basically. It’s pretty hard to test this too vs. testing everyone’s connection to the dedicated server itself. If it wasn’t for the P2P model in this then the connection to the dedicated server would be the end of the story but the P2P means everyone’s connections matter not only to the server but to each other.
You may very well be right about Mistlands! It’s shocking to me that this still isn’t out yet. Early on they were hiring a bunch of people because they originally were only a team of 4 when they released the game. The whole studio was 4 people. They have a lot more than that now though (and they’ve had time to train them) so I’m not sure why the content train isn’t showing up yet still.
Thanks again for your reply, take care!
Writing assembly_valheim_modded.dll…
Access error: Access to the path 'C:\WINDOWS\system32\assembly_valheim_modded.dll' is denied.Getting this error, would appreciate it a lot if someone helps (:
Hey Kush,
It looks like you’re trying to save the Valheim dll into the C:\Windows\System32 folder. I would just try saving it to the desktop or somewhere like that as the system32 folder is a protected system location and is basically giving you permission errors. If you save it somewhere else it should go!
Just wanted to link to this Github issue about Epic Loot’s excessive bandwidth. The creator of the mod himself looked at it.
Hey Dalayeth,
Thanks a ton for linking to this. I know I had asked earlier if anyone had contacted the dev about this specific issue. You gave a lot of technical details and it looks like it’s going somewhere. Great work, you may have made the ultimate difference here and are actually going to fix this (for everyone) after months and months of this being an issue!
has anyone tried this NetworkFix package?
Looks like it is patching the two methods using harmony just like we have been doing manually, should work in the same manner… just make sure the host and server are configured the same.
Does it work? and if yes, how to do it.. i downloaded the NetworkFix in thunderstore but there is no readme or howto anywhere. Does it increase those limits as sonn as i launch the game with it? Is it permament? Do i have to configure those values somehow?
this doesnt seem to be the case for me, this is on a modded server but without changes to sendqueue or socket parameters:
Without the changes to the send queue size why would it change? How would it change? How would more data be flowing if it was the same size?
How would the clients know to send you more data? They don’t basically, and if you don’t modify the clients they will be sending the wrong size queue and we have seen what happens in this situation (nothing good).
You need to modify all clients as well if you’re going to change the send/receive window size to the extent it will show on a graph instead of just loosening things up on the server. If they don’t match between server / clients it will be worse than not doing it all as there will be packet loss. Patk covers this really well if you want the graphs to show more throughput!
It isn’t really a throughput issue though. It kind of is but it’s artificial constriction and actually has nothing at all to do with your connection or any player’s connection. Every single one of your player’s connections has enough bandwidth to send 1000x the data Valheim is sending. It’s actually really complicated why it happens but if I were to try to sum it up Valheim doesn’t leave a lot of extra room in the packets for “extra” stuff from mods. You hit a point where the networking code is no longer behaving efficiently and basically lags/chokes itself to death (on purpose, it’s trying to limit the flow of data because it doesn’t understand why there’s this much of it, it doesn’t know about your mods). The conditions are created artificially basically. Valheim wasn’t designed to transfer that much data so their code / numbers don’t facilitate it and things start going very wrong even though it’s honestly not a lot of bandwidth. Run a bandwidth monitor on your connection and you will see it’s nowhere near saturated.
It’s essentially a design issue that Valheim coded hard limits that fit just their needs but don’t adapt / can’t be changed if you are doing anything like running additional mods. If they even gave us an option to have a larger window in the options none of this would be necessary at all (like many multiplayer games do).
The mod from this article loosens these restrictions (it raises the standards before packets start to be dropped). Will you see it in the graph? No, because we didn’t change the send/receive limits for the in article mod, we only changed the criteria before the server starts dropping packets if the queue is already a certain size (meaning after my change it will try to send those packets instead of drop them and resend them again later). There’s no way for more data bits to physically flow in a way that will show on the graphs though unless you actually change the window size on the server and all clients.
Does it help just doing the server mod from the article? Definitely. It helps because it isn’t a connection-related throughput issue that is causing the lag, it’s a design issue and this alleviates some of that poor design. The reason you won’t see a change on those throughput graphs though is that they aren’t measuring what we changed. Before the mod those packets would have been dropped and resent. What does this look like? It would just look like your packets being more “full” and closer to the limit next time. The packets will just keep getting dropped until there’s enough free extra space left over on another packet that it can fit.
This is the vicious cycle of lag that starts that you can’t really get yourself out of and is kind of a death spiral. The article mod greatly increases the size of the sendQueueSize before it will try dropping *anything* and will just ram them through. There’s limits to this though and people running 50+ mods will hit the TCP window size 100% of the time and have no choice but to do the client-based mod as well or it will be unplayable. There won’t be any packets with extra free space to piggy back on ever basically and that is when you will reach the breaking point.
It will not measure though on your graph because what you are measuring isn’t the actual problem / cause of the lag but just another side effect/manifestation of the underlying reality that Valheim’s networking code wasn’t designed for the amount of data you’re sending over it. To observe this effect on a connection monitor what you would see is that the size of the Valheim packets would be creeping closer and closer to 15000ish bytes. They will never cross that because there’s nowhere for more bytes to go, that is the limit. If you want to break that limit, you have to change that limit on all clients. Being close to 100% full on each packet would indicate poor health. There should be empty space and there won’t be when these conditions start.
Another more extreme way to alleviate it is to literally change the send/receive window size in all clients / the server. If you want to literally change the limits so they will show up on a graph you will be REQUIRED to modify every single game client every single patch but it’s possible and well covered here! It has to be that way. There’s no way to just do this on the server side because your clients won’t know about the change. Hopefully that helps!
yeah this is what i see for packet size seems to chill below 1k bytes
i do see sometimes players will “desync” from the server. i’m guessing they are lagging and their queue gets so full that it just is perma full and drops new packets.
so in that case increasing the queue size might help them recover. but still it will hit a limit eventually. i don’t want to fill up their memory.
That sounds about right. The article modification will get you to that point and then patk’s modification will let you raise that ceiling further. You are right though, at a certain point a lot of other resource constraints come into play but you could probably get away with raising the send/receive limits multiple times.
It sounds like some progress is being made on getting the code running the mods improved. Dayaleth just posted an update on a thread showing the dev looking at it that looks promising!
No matter how much I increase the value it appears that my data send and receive are capped at 150kbps. This definitely appears to be helping with stability but at least for me it doesn’t seem to actually be increasing the throughput. My group and I are seeing 150kbps as caps for send and receive even when using a value of 512000. We are using Valheim+ and EpicLoot both.
Hey AlistairChiss,
That sounds exactly right. If you want to change the #s that show in the actual then check out patk’s mod here in the comments (pretty popular for servers using a lot of mods). It requires modifying all game clients as well for each player otherwise their send/receive window size wouldn’t match (not good, we’ve seen it before during testing, lots of packet loss and things are overall slightly worse) but with everyone having it done correctly those #s will show as higher in those windows.
Basically this mod used to modify the send/receive limits when Valheim first came out but the code was changed requiring a rewrite and this new version is more like tweaking their pressure release valve to let more through before packets start getting dropped. It’s kind of a misnomer though because they changed how the Valheim code worked so the version in the article doesn’t technically do that anymore (as it requires a mod on all clients in this version) but you can still literally modify the send/receive limits (it’s just a little bit more in depth in the current version). Hopefully that helps!
Thought I’d add a weird foible I’ve come across since the H & H update and have not been able to figure out at all. I’m going to detail what we observed and see if anyone else has ideas, because we are stumped. We had made the changes successfully and they worked fine for a while, but lately we’ve been having an odd issue with the Spawning Stones and the area around it. I’ll try to describe it as best as I can. The first person that logs in to the stones loads them fine, but anyone else that loads into the stones can’t see them or any structures. They can’t see it UNTIL the person the it was initially loaded for leaves the area. The person that can see the terrain can log out, teleport, or even run away, but once they are out of range, the terrain appears for everyone that couldn’t see it. This seems to be an issue only with the Spawn Stones, as we’ve tried to replicate this in other areas and it doesn’t happen. It also seems to happen strictly on logging out and logging in. Once that portion of the world has loaded in for someone, it remains loaded for them whether they teleport away or run away.
It also happens even if you don’t log in near the stones, but someone is currently at the stones when you login. We had a teleporter set up from one island a significant way away back to the Spawn Stones. If someone was already at the stones when you logged in, even though you weren’t anywhere near the stones, if you tried to take the portal back to the stones it would not work despite showing that it was connected. Basically the portal didn’t “exist” on the other side for you. You COULD destroy the portal, rebuild it with the same name, and THEN it would work, but if you went through, ONLY the portal would exist within the area of the stones and the nearby terrain would not load in, again, until the person it was initially loaded for left the area. I believe it’s because the object was interacted with by linking the portals. We noticed that when the world wasn’t loaded in for someone who logged in, if a person interacted with one of the objects that had not spawned in, it would appear. For example: if I loaded into the spawn stones unable to see anything because someone else had been in the area when logging in, objects that THEY interacted with would appear, such as a door opening or a tree being hit with an axe. Enemies did not disappear, strictly the stones and surrounding landscape.
Now, we did try several things to attempt fixing it: pulled out mods one by one, reverified the files then added the modded assembly_valheim.dll files back in. The thing that seemed to work was using vanilla, unmodded assembly_valheim.dll files. I know it’s a bit long, but I just could not figure our exactly what was going on.
And I should add: we tried new characters and new servers as well.
I have observed similar weirdness randomly near the spawn stones also, for us it is more like a delay in the server sending the data to the client (the rec rate is usually sitting around the 0.1 – 0.5kbps mark), after 5 – 10 seconds on approach to the area all the trees / structures suddenly load in with a big burst of incoming data. The spawn area is the only place we have seen this behaviour so we haven’t been too concerned getting to the bottom of it (may even be related to a mod rather than the changes listed here).
Fast reply, ty. Yea, we can play around it pretty easily. The frustration was just trying to figure it out. It’s like an annoying hangnail: a minor inconvenience but one that can be played around. At least now we know for sure it’s the spawn and can mostly avoid building around it or having to much traffic around it.
I have found that this issue doesn’t seem to present if you take a portal to the spawn stones, it only appears for me if you approach the stones via walk/running.
wait what? this is the thread I was searching for. We built our base around said stones and some people noticed that terrain changes around that area wont load in or half the houses disappear. But once the host leaves that area it return to normal, only in rare cases a roof actually breaks. So we tried 3 different network mods with all kinds of settings and observed the same thing every time.
Can you list your mods so we can find a possible culprid? Or can anyine try Analyze what is happening here?