Revisiting Fixing Valheim Lag – Modifying Send/Receive Limits
Published: June 15, 2021 / Last Modified: December 3, 2023 / Gaming / 252 Comments
My Valheim Dedicated Server Castle
I’ve previously covered fixing Valheim’s dedicated server lag by modifying the dedicated server’s send receive limits. Since I wrote that article not long after launch though the game has been updated that the old method no longer applies. I had theorized in the comments of that article how to address this and today Sam R. confirmed that this method did indeed alleviate most congestion on the server in the places that the send/receive limits are known to bottleneck.
This updated guide will show some updated methods you can use to help reduce lag on your Valheim dedicated server!
Symptoms of lag on Valheim dedicated servers
The symptoms you are experiencing on the server can help narrow down what type of lag you are experiencing and what may be causing it. Lag caused by your CPU being too overloaded for example will have different symptoms than lag that is caused by your server hitting the maximum send/receive limit as an example.
By far the most common type of dedicated server lag seen in the early access Valheim release so far has been mainly due to the low send/receive limit if 64KB/s that is hard coded into the server. This means that when you do something like take your character a heavily populated area on the server it has to send all that information about all the buildings/players through an artificially limited very narrow 64KB/s pipe.
Symptoms of the send/receive limit type of lag include things like chests taking a very long time to open, sometimes 10-30 seconds or never opening while at the same time being able to see players walking around normally. This is because the other people’s X / Y / Z positions is a very small amount of data compared to all the items/names/attributes that are stored in the chest. The other players moving easily fits through the narrow pipe while the chest full of loot’s data chokes since it is much larger.
I’ll specifically lay out how to modify your server and increase this limit in the “Modifying your server’s send/receive limit” section below. Before that though let’s cover a more specific type of lag to watch out for (mostly by being avoided) related to terrain manipulation.
Avoid extensive terrain manipulation (for now)
Another commonly observed source of lag has to do with terrain manipulation. Some players have found that if they take a big chunk out of the ground in Valheim and make extensive terrain modifications that the area gets filled with separate “instances” of terrain for each modification people make. Think of the original ground as one big piece in the game files. Valheim has the very cool feature of letting you modify terrain by raising it/lower it/level it/etc. Each time you do this though the game has to store what you did as a new “instance” or a modification.
The way Valheim seems to work now is it renders that original piece and then the modifications afterward. Early reports suggest having a whole bunch of them in one spot seems to sometimes cause everyone’s in-game FPS on the server to drop as low as 40 when the clients normally are much much higher than that.
The best way to avoid this type of lag at the moment is to limit how much terrain modification you are doing in areas near your base. It is likely this will be addressed/eased in future patches. There should be some things the developer team can do to clean up/merge these instances or at least reduce their effect on the rendering process so it doesn’t drop the in-game FPS so dramatically.
At this time there is no options file for Valheim unfortunately. The only way to change it is to modify the server’s code and recompile the module that controls this.
Fortunately this is a lot easier than it sounds because the module we need to change is in .NET instead of something like C++ which would be much more difficult to do this with. I did not come up with this method, a very clever user on reddit (maximgame) shared it here and I want to give credit where credit is due!
Despite it not being too difficult these modifications are at your own risk. I haven’t experienced any problems so far but technically we are modifying the server’s code with the following procedure so understand that this isn’t an officially supported method and could have side effects or if you make a mistake could cause serious problems. Make sure you have backed up everything (your game world files especially) before you try anything!
Another important note is that it’s very likely you will have to make the changes again each update since the file we are going to modify is going to get replaced by updates. Until they add either a launch option or options file to officially change/control this property that is unfortunately unavoidable.
With all that being said if you understand the above and still feel comfortable let’s proceed!
Get dnSpy utility
We are going to use the utility dnSpy to modify the server code and recompile the module. The utility is free and is available here (download the Win64 build or dnSpy-net-win64.zip): GitHub dnSpy Official Page
Extract the archive and run the program dnSpy.exe from inside the folder you extracted. The dnSpy application will open.
Modifying server with dnSpy
We are going to go to the “File” menu in the top left and choose “Open” like this:
dnSpy – File -> Open Menu
Now you need to navigate to the Valheim dedicated server folder which is located in your “steamapps” folder. If you left all the options default when you installed it’s typically “C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed”.
We are looking for the file “assembly_valheim.dll” in that folder:
dnSpy – Selecting Valheim server assembly file
Select the file “assembly_valheim.dll” and click “Open” and the server’s assembly will load into dnSpy.
Once it has finished loading we are going to go to the “Edit” menu and click “Search Assemblies” like this:
dnSpy – Search Assemblies
For the search criteria enter “ZDOMan” like this:
dnSpy – Searching for ‘ZDOMan’
After the search completes you should see a list of results like the one above. Double click on one of these entries on the list and it should place you in the ZDOMan section of the assembly (seen on the left hand side of the screen):
dnSpy – ‘ZDOMan’ section of assembly
Perfect. We are now in the right place to make the changes. We just need to locate the line that has the send/receive limitation. To find that press Ctrl+F to bring up the find box (or choose “Edit” -> Find from the menu) and search for:
sendQueueSize
which should bring up the following lines:
dnSpy – sendQueueSize Variable
If you are having trouble finding the lines make sure you’re using Ctrl+F to search this time and not the “Search Assemblies” from earlier at the bottom of the screenshot, that’s a different type of search. Make sure you’ve disabled the case sensitive options (the Aa right below the box) or any other search filters. Your search query box should be in the top right of the screen like the above screenshot that has sendQueueSize in it.
Now right click on this line:
int num = 10240 - sendQueueSize
and click “Edit Class (C#)…”. Another window will open that will let you make changes to the file. We are going to modify ‘10240’ to a variable of our choosing. I made mine 30720 since after doing some conversion math that is the equivalent increase from my previous launch version guide that provided great performance and I kept them the same. If you followed my last guide this will give you the exact same performance as the old one did.
There are two instances of this variable. The second instance is 3 lines above the highlighted line we searched for and is:
if (!flush && sendQueueSize > 10240)
Modify both instances of 10240 to 30720.
Theoretically setting a limit too high could cause the server to swamp your internet connection if you have a large number of players on and there’s essentially no limit but there shouldn’t be any other effects of a very high number.
Once you’ve changed both numbers to the desired value we need to press the “Compile” button in the bottom right corner of this new window. This window will close and take you back to the previous one.
Saving the new assembly
It’s time to save our changes into a new .dll file. Choose “File” -> “Save Module…”:
dnSpy – Save Module as assembly_valheim_modded.dll
Change the file name to assembly_valheim_modded.dll. Don’t overwrite the original file just yet as we will want to take a backup and make sure that the server is closed before we do that. Press “OK” to write the new assembly_valheim_modded.dll file.
Backing up original file and replacing with our new one
Now it’s time to back up the original file and replace it with our new modded file. First make sure your dedicated server is closed otherwise the dll file will be in use. It is also a good time to make a backup of your world file just in case (see my Valheim Dedicated Server Backup Location / Guide for instructions)
Navigate in a file explorer window to your Valheim server directory where we saved the new assembly file (usually C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed).
First make a copy of the original assembly_valheim.dll file. You can just select the file and “copy” then paste a copy of it in that same folder and Windows will make a “valheim_assembly Copy(1).dll” file for you or you can back it up to a separate folder. Now remove the original file and rename your “valheim_assembly_modded.dll” file to the original file name of “valheim_assembly.dll”.
That’s it, you’ve now replaced the old assembly with the new one! Go ahead and start your server normally and the changes will have taken effect.
If you know of other Valheim lag issues that I didn’t cover here or are still having problems let me know in the comments and I will update this guide with other tips/tricks and try to answer any questions. It’s still a very new game and there isn’t very much out there for it yet. Have fun and enjoy!
I’m having trouble following the guide. When I try to compile the edited section using dnSpy, I encounter compile errors. First error is “Identifier expected” on line 730 and the rest appear to be due to UnityEngine not being found. Switching to editing the method instead of the whole class removes the first error but still has the UnityEngine error.
It means you’ve taken some liberty on the instructions at some point. It probably means you copied the file somewhere away from the main files or it’s isolated so it’s not finding any of the other libraries.
Some common ways people would do this is for example just transferring the .dll file to their desktop and editing it. Yes, that won’t work. You need to open the file with the entire folder present.
Here’s a reference. It just says you need to open the dll with the entire folder present like I just did so this is a dependency error.
Hopefully that helps!
Karmizy
1 year ago
Hi, there is a good dedicated server hosting, I’m on windows ?
Not really, no. The dedicated servers for this game are permanently broken essentially due to the multiplayer design. They have a bad multiplayer model that can’t scale past a few players. This article is for a mod you can do to try to make it slightly less broken. It will maybe let you have 5-6 players on without massive lag instead of 4 essentially.
We have a pretty good record here of Valheim’s history if you want more information via the comments. You can see all of the work that has been done over the years and the different results people had.
Hopefully that helps!
Luna
1 year ago
I did all these changes on my server hosted on GTXGaming. It did not work the times I tried and once I reverted back to my original dll file the world reset to day 1. I’m not sure how that even happened. Luckily I had backups so I restored everything.
Did you transfer your entire Valheim server folder over from GTXGaming? Or did you make the mod using files from your computer? I’d assume this is the problem if you didn’t transfer the entire folder. It’s not clear to me whether their servers are running Linux or Windows even. I don’t see that anywhere. That’s really the only time we’ve seen anyone manage to crash it is when they try to use Windows files on a Linux machine or vice versa.
As far as the world is resetting it’s really hard to say without seeing GTXGaming’s custom stack they are running to make this all work. I can tell you that the mod touches nothing to do with your world files and that it’s never happened to anyone before so I’d assume this is probably related to it crashing. Definitely always have backups before trying anything like this and I’m very glad you did.
From my experience the most likely reason this happened is that you used files from your own computer to make the mod and uploaded it to the server. It’s likely whatever software the server is running is not matching yours and we’ve definitely seen that before. I’d assume GTXGaming is doing something to try to handle the crash or something unusual is happening that wouldn’t happen if you were running the dedicated server yourself.
There’s really no way you could have made a mistake while making the mod that would have resulted in this. We’re not going near any code that is involved in saving the world or anything to do with that. It’s possible for it to result in crashes though if mistakes are made / platforms are mismatched and I’d assume it’s the crashing that triggered something to happen that reset that world (and even then it wouldn’t have been something from Valheim’s side most likely but from GTXGaming’s side).
Maybe you should try this via mods? We had CW_Jesse here not too long ago talking about the Better Networking mod for Valheim and it sounds like his mod does the exact same thing as this article (as well as some additional tweaks I believe). We had a lively discussion last time he was here and we had some slight disagreements/debate but it’s purely out of passion from both of us for Valheim and we probably agree on like 90% of everything realistically.
If it doesn’t like you directly modifying the files within GTXGaming’s stack why not try via their mod system? It sounds like they’re well set up to support the mods. I think this mod is worth a try for you. It looks like GTXGaming should happily accept this and then you don’t need to worry about what version of Valheim specifically they are running (and which OS, and that it’s not some sort of custom version for server hosting companies, etc.).
Hopefully that helps!
4LL
1 year ago
Dropping a comment to say that following both yours and Patk88’s suggestions, I’m hosting a server for several people running epicloot and valheim plus – no major lag!
I’ve consolidated the instructions below, this takes less than 5 minutes:
************
**SERVER**
************
1. Use dnSpy to open: C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed\
[[ James A Chamber sendQueueSize Fix ]]
2. Within dnSpy toolbar { Edit -> Search Assemblies } or Windows Hotkey { Ctrl+Shift+K } to search for: ZDOMan
3. Within ZDOMan file use dnSpy toolbar { Edit -> Find} or Windows Hotkey { Ctrl+F } to search for: sendQueueSize
4. Right click on “int num = 10240 – sendQueueSize” and select Edit Class (C#)
5. Replace both 10240 to 30720
6. Compile
[[ patk88’s Server Fix ]]
7. Search Assemblies { Ctrl+Shift+K } to search for: zSteamSocket
8. Within zSteamSocket search { Ctrl+F } for: RegisterGlobalCallbacks
and edit its class
9. Find gchandle3 and replace its number with 524288
10. Insert the following lines of text just before gchandle.Free()
Is this DLL change the same for Linux docker based hosted servers? I see the DLL in my server files, but anytime I try and change it with a modded compiled DLL, the server hangs on startup and never fully loads ….client is working fine
I don’t think anyone has put up a Linux version of it yet prebuilt but it can be done. You typically need to transfer the whole folder so it can resolve all of the dependencies but FreezeLag covered it well.
Hopefully that helps!
vi0lation
1 year ago
Hi James, starting a new post for this.
No need to apologize, I completely understand your frustrations, I too am (almost) at my wits end trying to make the issues discussed better, because as you said without a total rework of the multiplayer code, there is no solution – just things to mitigate the symptoms. I’ll post below some of the scenarios we have tried in our testing.
We run a dedicated linux server, using Gportal as the host.
We play on a heavily modded version of valheim (90+ mods)
On our Friday game nights, have 4-6 players online simultaneously, all usually fighting and playing together in the same zone/instance, and this is where the lag, rubberbanding, teleporting, chest/door delay, weird boar behaviour etc. etc. pop-up.
The 3 mods the supposedly help with network performance I have used, and am currently experimenting with are:
Network: Have been using this pretty much from the get-go (2-3 months of running a server, and trying to find ways to make it less laggy) This is the mod that unlocks the send/rate and has so far shown to improve results, significantly (but the issues still happen with 4-5+ players, and depending on the circumstances of the game etc.)
Compress: Just installed this one, to see if it makes any difference, and waiting for a few game nights to observe results, currently its installed in tandem with the Network mod, which may or may not conflict with each other, if that is the case, will retest with just Compress mod installed.
BetterNetworking Valheim currently has an issue where it doesn’t work on Linux Dedicated Servers, the dev is aware and is working on a resolution – personally I have not tested this mod yet, but will once the fix is in place.
I have not tried direct assembly editing yet, and will include that into the testing with other variables in the coming weeks.
This past Friday, instead of our regular game night we did a testing night, loaded into the server with devconsole so i could spawn mobs and raids under controlled conditions and we can record results. The following results came from that night:
We tested several combinations of different things, including:
– Running the server using PlayFab/Crossplay Network Protocol (Vanilla)
– Running the server using PlayFab/Crossplay Network Protocol (with Network Mod)
– Running the server using Steamworks Network Protocol (Vanilla)
– Running the server using Steamworks Network Protocol (with Network Mod)
– Removal of JewelCrafting Mod
– Spawning fights and raids around the town, and in an empty area, etc.
The best combo so far is steamworks network protocol (crossplay disabled) with network mod, the removal of JC had no impact. After a bunch of testing the following is pretty much the ‘result’:
– Town will always be laggier, due to all the ‘stuff’
– Fights outside of town are much less laggy
– Using Playfab/Crossplay network protocol was massively laggy compared to steamworks, to the point where it was game breaking / unplayable (huge load times, doors taken 5+ seconds to open, etc) (rolling back to steamworks but keeping everything else the same showed a significant improvement, only lag was fighting mobs in town + some other minor hiccups.
For example with 4 of us we were able to fight 20 skeletons, or 15 draugrs, etc. with no lag witnessed for anyone. (In an empty zone, no player built anything, etc) this is all still highly connected to who ‘owns’ the instance of the zone this is happening in. we tested this as well by having everyone except the player with the lowest end machine stay connected, so they assume control of the zone, and then have people reconnect, even in this state there was no/very little lag fighting a significant quantity of mobs, that said – fighting the same amount in our town, that has a large amount of buildings, chests, crops, cooking stations, etc. the same test yielded significant lag when fighting mobs.
I will continue to test different scenarios and report back
Fascinating results, what a great and detailed writeup! Your findings are definitely 100% consistent with others here as well so far. It sounds right to me that the network mod would have the impact you’re saying.
I talked about it a bit in my last reply but 4-5 is about the right number of players for the mod to have an impact. The type of mod I’m covering here won’t get you much higher than 5 players. It basically makes 4-5 players go from unplayable to playable (but definitely not perfect).
Your observations about who “owns” the zone having an impact are also correct and we haven’t really talked about that much yet. I’m not sure how it decides who owns the zone but first come first serve seems logical. I don’t think it does any type of smart balancing where it decides which player has the best connection unfortunately.
There’s just no way that the developers who went with this multiplayer model would even be thinking about a problem like this because if you were you would have never chosen this multiplayer model in the first place (which they shouldn’t have). I’d imagine that once you own the zone the only way to relinquish control is to probably leave the zone or maybe even the game entirely.
I would say from my experience that 5-6 is about the most you will get with the mod I have covered for the client/server (and 6 is really going to depend on the players having high quality connections, all of them). I could see you extending this though with tricks like the ZDO compression. The problem basically is that the Valheim multiplayer model is so inefficient that we are literally hitting the limits of people’s network stacks. You can literally overwhelm their connection at some point and 5-6 players seems to be about where that starts to happen (even with the mods). It feels like you hit a wall where the tweaks we’ve made just aren’t enough to keep the data flowing anymore.
I could absolutely see you extending things to 7-8 though comfortably with some of the mods you’ve spoken about here. The ZDO compression seems like it could potentially buy another player slot before the problems start. It’s all a game of figuring out how far out we can extend that wall where the poor multiplayer model choice catches up with it and brings the game to an unplayable state.
Your messages about the lag around town make a lot of sense too. It used to be terrain modifications would lag it quite a bit but the terrain modification lag has been fixed for the most part (although I think you can still create it by digging out a giant mountainside or something ridiculous). I’d imagine this is related to the settlement entities such as campfires, workbenches, torches which all have “states” such as how much resin is in the burning torch. That all has to be sent to the client and tracked so that when you walk up to it you see the correct amount of resin in the torch or wood in the fire if that makes sense. This is just that much more data which will take us to the breaking point faster than the empty woods for sure.
Tracking resin in torches and wood in fires and the other building entities/items/properties might not seem like that much but it really adds up fast. How many torches/fires are in a single settlement? They’re constantly burning and losing wood/resin. A lot of my settlements probably have dozens or hundreds of them for the really big ones. Some things won’t populate during lag like chests/containers but a lot of entities/buildings you will notice do and those are the ones that I’m talking about. Those are the ones that have states that are being sent to all of the players in the zone all of the time.
Notice that it doesn’t lag showing you how much wood or resin is left in a torch. It always knows because this is *always* sent to you (and way before you walk up to it) otherwise if you walked up a torch when you were lagging it would show something like ? / 4 resin until it stopped lagging. It is a disaster to send all of this unnecessary entity data to everyone when we’re so close to the brink with Valheim’s multiplayer model. We literally don’t have any data to spare.
It would actually be better if all of this data wasn’t sent and it did show ? / 4 resin during lag. That would save a lot of data considering the chances of a player actually walking right up to one specific torch in that entire settlement and staring at it to see how much resin in it is probably incredibly low. They probably will never bother to touch it or even get that close to it until it burns out. Multiply that by an entire settlement of torches and just the developers changing that (or a mod) would have a significant impact reducing lag near any type of settlement as you’ve been observing.
In other words when you try to open a chest you won’t see what is inside it when the server is lagging until the server catches up. If everything in the Valheim settlements worked that way they would not lag the server the way they do. There are other types of entities such as anything that burns that doesn’t seem to work this way in Valheim and are being sent to the players in the zone all of the time.
Thanks for such a detailed report. I look forward to seeing what you find and if you are able to comfortably use these mods to perhaps extend your server to support 6-7 players potentially without hitting the doom spiral of lag where things just keep getting worse and worse until it grinds to a giant 30 second lag spike and all the crazy stuff that can happen. I’ll stay tuned for sure!
All seems to add up indeed, I’m wondering with the introduction of Crossplay this is when several people have started reporting worse performance for Mistlands, as you now have to manually disable crossplay on a host to force it back to steamworks. And the difference between Crossplay/Playfab vs. Steamworks was night and day.
In terms of my understanding of zones (This all comes from hanging around several discords with fairly prominent valheim mod devs and gurus, so I am just repeating what seems to be the general consensus on zone information)
– Zones are 64×64 sized grids
– The first player to reach/load a zone becomes the owner
– That said, to further complicate things, its not exclusive control, another player that interacts with an object (for the first time) inside a zone even if they werent the first there, becomes the owner of that specific object inside the zone. When your watching the log output of bepinx when playing multiplayer, you will often see the console spam with ‘attempting to interact with xxxx – but i am not the owner’ messages, this is essentially your client trying to interact with say a chest, or door that someone interacted with earlier and first, and therefore control it, but due to the backlog of multiplayer lag, your client cannot receive a response until that backlog queue is cleared and the owner can reply with the return, hence the console spam of that message until network congestion clears.
In the zSteamSocket Assembly inside assembly_valheim.dll as per the method presented by Pat in this forum. I’ve asked the mod author to see if that is indeed what it is doing or not, would be interesting to know.
Oh wow, that’s some fascinating information about the zones. What a mess!
I believe it though. I’ve seen some *crazy* bugs in the past related to zones. I’ve seen a one-shotting dragon elite on my screen where all of the other players on the screen saw a forest troll elite. I was hit by a single cold ball projectile from his entire volley wearing full cold resist gear and instantly popped into a grave. When I logged out and back in I saw a forest troll elite like everyone else when going back to my grave.
I was never able to explain that incident because if one person is in charge of the zone that incident doesn’t make sense. How could my monster on my screen disagree with theirs? This is how. Exactly how you said.
Somehow what ended up happening is an elite spawned but on my screen I was in the mountains and on their screens (or whoever was controlling the zone) I must have still been in the forest. I was flirting along the border when this spawn happened. These mechanics you’ve outlined here explain how there could possibly be a disagreement like this between the clients. One or both of our consoles was probably full of log messages saying “I am not the owner” while they were trying to solve this and I got nuked by a bugged damage projectile from a monster that isn’t even supposed to shoot projectiles (I guess he can throw rocks). This was essentially an error state and I wish I had known to check the console at that time (or had the zone owner check their console).
Those are some interesting options for the mods. They definitely do look like they would correspond to them and they would have to modify them I’d think to change this directly or indirectly.
It’s definitely interesting to look at the Valheim code because you just keep thinking the whole time “why did they add this code in to artificially restrict and make their own game perform very poorly”? The answer is if you let it run wild it would use such unimaginable amounts of bandwidth trying to scale this to more than a few players that it’s actually able to bring an enterprise connection to it’s absolute knees. Like if you tried to play Valheim at work you could potentially lag the entire company internet line to nearly a standstill without these limits in place.
We’ve had people try completely removing all of the limits before in the past and this is what happens. Someone got up to like 80 mods installed on their server and had definitely high player counts and what ended up happening is exactly that. It scaled so poorly (and the 80 mods adding additional data weren’t helping) that it basically completely swamped the player’s connections (just with pure data / packets).
And by swamped I mean *nothing* on their networks worked. Streaming would crawl. All web sites would crawl etc. I bet someone looking at a traffic monitor for that server would conclude they were under some kind of massive DDoS attack (or their server had been hijacked and is trying to perform some kind of DDoS attack on your player’s IPs). It certainly wouldn’t look like typical gaming traffic.
That’s how out of control it gets and I’d imagine why they put these artificial constraints in. Valheim doesn’t scale and using this multiplayer model you can do things like bring an entire enterprise’s network to a crawl with less than a dozen people on there (to get a sense of how much it can’t scale, it can’t scale to the point where it’s dangerous to not put these limits in the code).
Definitely let us know what else you find!
vi0lation
1 year ago
Would it be possible to list what version of Valheim the hosted modified assembly is for? Current version of valheim is 0.212.9, is the hosted assembly for this version? Thanks for all the research and hardwork investigating this!
I honestly haven’t ran Windows for years but if you’re running Windows you can just download the file and right click on it and choose “Properties”. It will tell you the version. This won’t work in Linux or probably Mac OS X either if Valheim even runs on that.
Or if you don’t have Windows we can use some basic logic. The file download has the date 12-21-2022 on it. What patch was released around that time? This one.
There’s two ways to find out at least as if I answer your question the answer will be outdated within a week or two. That is how you check!
You will have to replace this every time it’s updated and if Sketaful hasn’t got around to it then you can always do it yourself on any version. I honestly expect to see 100 more posts like this over time including every time it’s updated which is why I’m responding this way. You’re just the first.
I don’t maintain the file and no I’m not willing to update this article every time it’s updated with the latest version. I’d encourage you to ask the developers to fix the game you are playing instead of put those requirements on me and Sketaful. He’s doing a public service here already beyond what I’d be willing to offer.
I’ll tell you guys how to do it but no I won’t update it every time. If Valheim wants to pay me to develop this mod for each of their updates I’m willing to do that. If you’re personally willing to pay me for every Valheim update I will also do it for you to save you 2 minutes of modifying the DLL yourself. Otherwise the people being paid should probably do that.
I don’t think that people understand this isn’t a solution. This isn’t even a good workaround. It’s the best we can do for a game that is broken. That’s it. Asking to have this mod enshrined in some kind of process with posted versions and this whole ordeal… I mean are you going to pay for it? Why did you take the time to ask me to do this rather than ask Valheim why this is an issue in their game? I guarantee that you didn’t do that and that you won’t.
I honestly wonder if I have cursed Valheim by releasing this and giving people an alternative workaround to the broken game. I doubt this will ever be fixed now and it will just be people expecting someone to make this dll for the rest of the game’s existence. If you guys had no choice but to complain to the developers then they wouldn’t get a moment’s peace about this issue and it would have been fixed a long time ago. I might personally be responsible for it on some level. I really believe that at this point.
thanks for the reply, all good – just modified the dll myself based on your instructions, and read the entirety of this thread. 100% the same boat me and the 4-5 ppl I play with are in. same lag, teleporting, rubberbanding, etc. etc when we are all in one instance. I have tried running client/server with smoothbrain-network mod (which supposedly unlocks the caps without having to edit the assembly) and just recently trying the comfy-compress mod, which supposedly compresses zdodata before transfer to cut down on amount, but no testing done on this yet if it makes it worse or better.
Will try the modded assembly on server, and clients and retest as well to see if i can add to the pool of tested scenarios.
Excellent work, sorry for the frustration in my first post, I promise it’s really frustration with the Valheim developers more than anything.
I’m a little frustrated with the players not complaining but to be honest with you after reading your reply I don’t think you fall into that category. It sounds like you’ve been working on this issue for some time and are probably as frustrated with the Valheim developers and the networking code as I am. You’re actively investigating this issue and deploying code/mods to try and address it and that’s what this post is all about.
The smoothbrain-network mod I haven’t heard much about. It does sound like it does things similar to the mod. It would make perfect sense for someone to make something like that!
Compressing the ZDO data makes perfect sense as well. The mod touches on the code that controls that but it doesn’t apply any type of compression of course. We are basically just widening the opening data can flow through and reducing when Valheim tries to throttle itself (which is mostly by dropping packets, yay, and when it isn’t the entire server lags and waits for it to catch up).
I’d actually be very interested in your results. If you could tell me how some of those mods compare to doing the assembly mod (like are any of them doing the same thing, does the modified assembly have other effects than these do, etc.) that would be very interesting information that nobody else has shared here before.
The good news is you are right at the numbers where the mods have really had an impact. 4-5 players is just right. That’s exactly how many I had when I developed the mod and it’s how many a lot of the groups had that were able to beat the game together had that found success using a combination of the server side mod from the article and the client side mod posted about here in the comments by pat.
Worth noting that crossplay does enable compression in vanilla Valheim. It just uses very slow compression, though it achieves a good compression ratio.
Furthermore, ZPlayFabSocket’s.GetSendQueueSize() multiplies its result by 0.25, presumably to send more through the compressed socket. Dividing the result like that instead of subtracting 30KB (for a total effective queue size of 40KB) means slightly inconsistent amounts of data are sent.
That makes sense. Are there any negative consequences of that? We’ve seen people cause some problems like this before by having mismatched queue sizes between each other. Patk noticed massive packet loss when the queue sizes didn’t match and I believe this was even between clients (meaning if one person has the client-side mod on your server they all should).
I suppose if everyone is multiplying their result by the same 0.25 though that everything should always match up. It’s only when they *didn’t* match we saw problems.
That sounds like a reasonable conclusion for people to have reached given how they reached it, but also a wrong one given both my understanding of how Valheim’s networking works and my in-game testing. Better Networking allows you to change the queue size on the fly and that’s never caused an issue. It’s even beneficial to have people on different queue sizes based on their available upload bandwidth. Since you’re modifying the assembly directly, I’d sooner reach the conclusion it’s related to that, but I don’t see how that would explain your experiences either. Sorry I couldn’t be more informative!
Don’t sweat it at all, I think you were still pretty informative here! That definitely makes perfect sense and that’s how these types of settings are meant to be used in other games.
The Valheim code to handle this though is quite a mess. They actually used to have a variable that controlled this. I covered it in my original Valheim mod that worked at launch. They *had* a variable at launch and they *removed* it and hard-coded it!
Have you considered just throwing out the Valheim networking entirely with your mod? How deeply intermingled is this stuff in there? I can only imagine how much harder it would be than it should be due to the mixing of the client-side and server-side models.
Maybe I’m giving away good ideas for free here but it seems like if you rewrote the Valheim network stack it would essentially be a completely different game. If you went to a true client/server model (like I used in my Spark game here, which can still be played at armorcritical.com) it would feel more like hosting a Minecraft server (and probably have about the same # of player capabilities as people could get if they ran a Minecraft server on that same hardware, maybe even better since Minecraft actually has a lot going on with chunks/blocks that Valheim doesn’t).
Imagine a Valheim server that supports 12-24 players (or even potentially a lot more) because it is no longer using this partially client-side or peer-to-peer hybrid model or whatever you want to call it. I really don’t think there’s a proper term I’ve ever heard for the way they mixed and matched different multiplayer models in this game. It has the weaknesses of all of the models (such as if the server lags it screws everything up, but if the client lags it ALSO lags the entire server for certain operations, ugh) and ends up with none of the strengths of any of them.
That has to be the #1 dream of a lot of the bigger and more popular servers out there for Valheim. The player cap we have now pretty much restricts it to a group of friends. If you could rewrite the stack I’d imagine people would pay for that mod (or maybe you offer it for free then above 10 players is a paid unlock, who knows). Obviously all of the clients would have to be running the networking mod as well in this situation as it would basically be a full rewrite of the networking code.
I’ve got my hands full but to be honest with you this is what I would try to do if I attempted to do more with it. It would be an achievement no other mod could ever top. It would almost be a historical event for the game if someone did that because like I said it opens things up that we could never do before on Valheim (but that people desperately want to be able to do).
I can only imagine how much work it would be. It’s even more work doing it to someone else’s game rather than your own game. It’s one of those things though where if it could be done it’s valuable enough that you could find a way to make money with it (especially for servers looking for 9+ player custom solutions).
You would have to pay for all of the developer’s sins yourself to do this. All of their decisions they made such as projectiles all being client-side and having no server-side collision, yep, you’re going to have to do that. You’d have to implement everything they didn’t want to do that led them up to this point. If you did though you would be responsible essentially for making Valheim into a game that can have servers with entire communities (like Minecraft servers) with dozens or potentially hundreds of players.
No one could tell you the potential of such a game. It’s never been done. The Valheim developers couldn’t do it and apparently don’t seem to be planning on doing it any time soon that I’ve heard of. All I could say is that it definitely would have potential (and I’d imagine most Valheim players would agree)!
If you do a true server/client model, then the server needs to do physics simulations. The Serverside Simulations mod does that, but it greatly increases the required server resources, both CPU and upload bandwidth. Eventually, the server becomes the bottleneck. Can we do better?
With the current Valheim method, you could theoretically have hundreds of players on a server as long as they were evenly distributed in different zones. The biggest problem is when the weakest link becomes the zone owner or the strongest link is overloaded.
The current model is actually better than it might appear at first glance… if done well. Simply having smarter logic for choosing who the zone owner is (based on CPU speed and upload bandwidth) would make a world of difference, and I could probably do that for Better Networking 3.0. Having smarter zones instead of a grid would be a further improvement, but that might not even be necessary.
There is a way to make money from this, but it’s not from making a publicly available mod: It’s either from becoming a server provider that outperforms (or is cheaper than) other services, which I could probably do, or using the fact you built a popular mod on your resume to get a job in tech, but I already have a job in tech. Most people don’t want to pay for mods. They’d sooner reverse engineer them than pay. But people do want to pay for servers. I appreciate your thoughts. If I do end up becoming a server provider, I’ll throw some money your way to pay for the idea. 😛
That makes sense to me. You’re essentially saying it would be better to offer it as a service. I think that is smart for sure.
You’re right about people wanting to pay for mods I think. I think a service is a better refinement of the idea in the modern age. Reverse engineering is for sure a reality. I’m mostly saying it would be so much work and effort to do what we’re talking about (going beyond modding more toward a complete revamp/rewrite) it would have to make money somehow (unless one has all the time and resources in the world I suppose).
I am definitely expecting someone to make the exact thing we talked about here though whether either of us do or not. If someone wasn’t already working on it they are now after our conversation. Maybe you can optimize the system enough to achieve basically the same effect as moving to a model that looks more like a Minecraft server (an example I continue to use because of how appropriate it is).
Minecraft wouldn’t be arguably the most popular multiplayer game of all time if it was limited to 5-8 players. It sure would be nice to see what Valheim could do by achieving the advanced multiplayer capabilities Minecraft achieved more than a decade ago.
Minecraft is definitely more demanding to run than a Valheim server but able to run easily on a Raspberry Pi nonetheless. Doesn’t seem like too outrageous of requirements to me in 2023. Much more advanced/deep block and map scales and mechanics going on there than anything in Valheim as well. A Raspberry Pi can still do it unless you have more players than you can have on Valheim on it. I’ve been doing it for more than 5+ years. At least I’d published work by 1.12 on it it looks like showing others how to do it.
It’s pretty tough for me to swallow that it’s going to melt people’s hardware to have a client-server model for Valheim when Minecraft *definitely* has a lot more going on and runs on a Pi with more players than this game can handle. My games are far older than that and all used client-server models and none of them lagged as bad as Valheim does (and they had all server-side damage/physics/the works). That was 15 years ago in some cases. I haven’t made games for more than a decade. The servers had a fraction of the bandwidth as well. We’re talking about like early Steam levels of data (like CS 1.6) and they could all do this back then as well. 15+ years ago I was also running Steam servers that had server-side physics/the works for games like CSS. We had to due to cheating.
It’s beyond archaic to rely on the client. In fact the other problem you will face with this model is exactly that: cheating. Valheim is trusting the client. This is the biggest reason the model won’t work for servers with a large # of players. Cheating is already bad enough on Minecraft on large multiplayer servers and that IS a client-server model game. They still find the tiniest loopholes and the smallest places where the developers trusted the client and milk some crazy exploits out of it. I shudder to think what kind of exploits would exist in a large Valheim server with client-side trust like this. People would be able to do some outright spooky things with a malicious/hacked client (and this probably already happens today I’m guessing). They’d be able to do things you’ve probably never seen cheaters do in practically any other game and it would be absolute mayhem.
All of my arguments are for a perfect world. That would be a debate I would have with the Valheim developers about client-server model vs. what you’re trying to do and you’re right. I think this model *could* work if it’s implemented correctly but I mean this is cutting-edge genius level stuff if you can pull it off properly.
I really don’t think this system was born out of genius. I don’t know the story behind it but it wasn’t a good decision to limit these servers to 5-8 players via this model or it’s implementation (proper or improper, this implementation can’t scale to the kind of numbers people want on the game). In fact like I argued Minecraft wouldn’t have become the game it is with those limitations and I wonder what Valheim could become and what people could do if it didn’t have them.
The game was originally made by 4 people and I imagine one of them had experience with this type of multiplayer code or something like that and the explanation is probably that simple. As the game got fleshed out more and more going back and changing this isn’t easy even though now they have a lot more money and staff.
Definitely don’t hesitate to stop by if you have any news to share about Better Networking 3.0. I think it’s great that you’re working on the mod and giving it out for free. I’m really not expecting the Valheim developers to change the networking at all so I believe your skills will continue to be required for this issue.
I have no issue with you optimizing the zones or improving the system through mods for sure and in fact I’m legitimately interested in what you’re able to do with this and how many players potentially you may be able to get on a server.
Thanks for a very informed conversation and sharing your unique background/experience with the Valheim code. I’ve learned a few things for sure. Take care!
Yeah, cheating is, I think, the main reason to switch to a true server/client model. While you’re right to say Minecraft is more intense than Valheim all things being equal, not all is equal; switching to that model would also require me to optimize other parts of the game, not just the networking stack.
Clearly they spent their limited resources well; it’s a great game with great reviews. For every issue that bothers us, there are dozens of things that are so well done we don’t even think about them and are the reason we’re having this discussion in the first place, and as you said, we don’t know the story behind it. It probably would’ve been easier to just have a server/client model in the first place than to write code for switching zone ownership and such, which maybe means they realized it would be easier to switch to this hybrid model than optimize what was already written. Or maybe multiplayer was added late in the development cycle. Either way, we’re looking at the result of hundreds of good decisions, even if the end result isn’t exactly what we want. There’s a good chance it’s not exactly what they want, either.
Thank you for the great discussion! And thank you for these blog posts and discussions: they were one of my two original sources of education for making the first version of Better Networking in 2021, which is why you’re linked in the mod description.
That’s fantastic, I hadn’t realized that! You did exactly what I was hoping someone would do which is turn this into a more formal mod. You’ve definitely gone beyond anything we’ve done for sure at this point with directly modifying the assembly.
I completely agree with you about what they did with their limited resources. It’s an incredible story. According to Wikipedia and this news article I lied. It was actually 5 people that made Valheim. The article’s title:
Valheim: How a 5-Person Team Created the Most Popular Game on Steam
That’s an achievement nobody can take away from them. The Mistlands have also been a huge success and were honestly a lot crazier than I was expecting (in a good way, like they were weird and interesting and fun to play). They achieved something that AAA studios burn billions of dollars in cash with teams of 100+ with 5 people. It’s legitimately one of the funnest games for multiplayer I’ve played since Minecraft and one of the closest feeling things to it (but much more adult and gritty) but still rewarding a lot of the same things like preparation/exploring/living off the land/etc.
I also think that’s why our discussion was so valuable too and all the discussions we’ve all had about it as a community as well. Everyone is super passionate about Valheim and that’s such a great sign. You know your game is doing great when that is the case. It’s when nobody cares that you have to be worried. We definitely all very much care!
I’m not sure if the plan is still for 9 bosses but we still have a long way to go. I still am secretly hoping that they do decide to change this. Now would absolutely be the time. I think if that doesn’t happen though that people like you and the rest of the community will keep finding more and more clever ways to milk more out of the system.
Definitely keep us posted about new versions and thanks again for the discussion!
KeenKeister
2 years ago
When I try and Compile I keep getting error codes CS1001 and CS0246, any clues?
The type or namespace name 'type/namespace' could not be found (are you missing a using directive or an assembly reference?)
This is a dependency error. You may need to copy your entire folder instead of just the Valheim .dll file as some others have encountered recently. dnSpy is trying to resolve the dependencies and not finding them in your case here. Hopefully that helps!
Hello,
I’m having trouble following the guide. When I try to compile the edited section using dnSpy, I encounter compile errors. First error is “Identifier expected” on line 730 and the rest appear to be due to UnityEngine not being found. Switching to editing the method instead of the whole class removes the first error but still has the UnityEngine error.
Hey Handylimit,
It means you’ve taken some liberty on the instructions at some point. It probably means you copied the file somewhere away from the main files or it’s isolated so it’s not finding any of the other libraries.
Some common ways people would do this is for example just transferring the .dll file to their desktop and editing it. Yes, that won’t work. You need to open the file with the entire folder present.
Here’s a reference. It just says you need to open the dll with the entire folder present like I just did so this is a dependency error.
Hopefully that helps!
Hi, there is a good dedicated server hosting, I’m on windows ?
thanks
Hey Karmizy,
Not really, no. The dedicated servers for this game are permanently broken essentially due to the multiplayer design. They have a bad multiplayer model that can’t scale past a few players. This article is for a mod you can do to try to make it slightly less broken. It will maybe let you have 5-6 players on without massive lag instead of 4 essentially.
We have a pretty good record here of Valheim’s history if you want more information via the comments. You can see all of the work that has been done over the years and the different results people had.
Hopefully that helps!
I did all these changes on my server hosted on GTXGaming. It did not work the times I tried and once I reverted back to my original dll file the world reset to day 1. I’m not sure how that even happened. Luckily I had backups so I restored everything.
Hey Luna,
Did you transfer your entire Valheim server folder over from GTXGaming? Or did you make the mod using files from your computer? I’d assume this is the problem if you didn’t transfer the entire folder. It’s not clear to me whether their servers are running Linux or Windows even. I don’t see that anywhere. That’s really the only time we’ve seen anyone manage to crash it is when they try to use Windows files on a Linux machine or vice versa.
As far as the world is resetting it’s really hard to say without seeing GTXGaming’s custom stack they are running to make this all work. I can tell you that the mod touches nothing to do with your world files and that it’s never happened to anyone before so I’d assume this is probably related to it crashing. Definitely always have backups before trying anything like this and I’m very glad you did.
From my experience the most likely reason this happened is that you used files from your own computer to make the mod and uploaded it to the server. It’s likely whatever software the server is running is not matching yours and we’ve definitely seen that before. I’d assume GTXGaming is doing something to try to handle the crash or something unusual is happening that wouldn’t happen if you were running the dedicated server yourself.
There’s really no way you could have made a mistake while making the mod that would have resulted in this. We’re not going near any code that is involved in saving the world or anything to do with that. It’s possible for it to result in crashes though if mistakes are made / platforms are mismatched and I’d assume it’s the crashing that triggered something to happen that reset that world (and even then it wouldn’t have been something from Valheim’s side most likely but from GTXGaming’s side).
Maybe you should try this via mods? We had CW_Jesse here not too long ago talking about the Better Networking mod for Valheim and it sounds like his mod does the exact same thing as this article (as well as some additional tweaks I believe). We had a lively discussion last time he was here and we had some slight disagreements/debate but it’s purely out of passion from both of us for Valheim and we probably agree on like 90% of everything realistically.
If it doesn’t like you directly modifying the files within GTXGaming’s stack why not try via their mod system? It sounds like they’re well set up to support the mods. I think this mod is worth a try for you. It looks like GTXGaming should happily accept this and then you don’t need to worry about what version of Valheim specifically they are running (and which OS, and that it’s not some sort of custom version for server hosting companies, etc.).
Hopefully that helps!
Dropping a comment to say that following both yours and Patk88’s suggestions, I’m hosting a server for several people running epicloot and valheim plus – no major lag!
I’ve consolidated the instructions below, this takes less than 5 minutes:
************
**SERVER**
************
1. Use dnSpy to open:
C:\Program Files (x86)\Steam\steamapps\common\Valheim dedicated server\valheim_server_Data\Managed\[[ James A Chamber sendQueueSize Fix ]]
2. Within dnSpy toolbar { Edit -> Search Assemblies } or Windows Hotkey { Ctrl+Shift+K } to search for:
ZDOMan3. Within ZDOMan file use dnSpy toolbar { Edit -> Find} or Windows Hotkey { Ctrl+F } to search for:
sendQueueSize4. Right click on “int num = 10240 – sendQueueSize” and select Edit Class (C#)
5. Replace both 10240 to 30720
6. Compile
[[ patk88’s Server Fix ]]
7. Search Assemblies { Ctrl+Shift+K } to search for:
zSteamSocket8. Within zSteamSocket search { Ctrl+F } for:
RegisterGlobalCallbacksand edit its class
9. Find gchandle3 and replace its number with 524288
10. Insert the following lines of text just before
gchandle.Free()SteamGameServerNetworkingUtils.SetConfigValue(ESteamNetworkingConfigValue.k_ESteamNetworkingConfig_SendBufferSize, ESteamNetworkingConfigScope.k_ESteamNetworkingConfig_Global, IntPtr.Zero, ESteamNetworkingConfigDataType.k_ESteamNetworkingConfig_Int32, gchandle3.AddrOfPinnedObject());11. Compile
[[ SAVE ]]
12. Choose “File” -> “Save Module…”
Change the file name to assembly_valheim_modded.dll
13. Choose “File” -> “Close All”
************
**CLIENT**
************
1. Use dnSpy to open:
C:\Program Files (x86)\Steam\steamapps\common\Valheim\valheim_Data\Managed2. Search Assemblies { Ctrl+Shift+K } to search for:
zSteamSocket3. Within zSteamSocket search { Ctrl+F } for:
RegisterGlobalCallbacksand edit its class
4. Find gchandle3 and replace its number with 524288
5. Insert the following lines of text just before
gchandle.Free()SteamNetworkingUtils.SetConfigValue(ESteamNetworkingConfigValue.k_ESteamNetworkingConfig_SendBufferSize, ESteamNetworkingConfigScope.k_ESteamNetworkingConfig_Global, IntPtr.Zero, ESteamNetworkingConfigDataType.k_ESteamNetworkingConfig_Int32, gchandle3.AddrOfPinnedObject());6. Compile
[[ SAVE ]]
7. Choose “File” -> “Save Module…”
Change the file name to assembly_valheim_modded.dll
—————-
Alright great, now all you have to do is replace your servers assembly_valheim.dll with your new assembly_valheim_modded.dll!
And your client’s (and all connecting clients) .dll’s with your brand spankin new assembly_valheim_modded.dll!
Hey 4LL,
Looking good, thanks for sharing this consolidated writeup!
Is this DLL change the same for Linux docker based hosted servers? I see the DLL in my server files, but anytime I try and change it with a modded compiled DLL, the server hangs on startup and never fully loads ….client is working fine
Hey jic,
Great question. So unfortunately no the Linux version is different. FreezeLag outlined how to do it for Linux here.
I don’t think anyone has put up a Linux version of it yet prebuilt but it can be done. You typically need to transfer the whole folder so it can resolve all of the dependencies but FreezeLag covered it well.
Hopefully that helps!
Hi James, starting a new post for this.
No need to apologize, I completely understand your frustrations, I too am (almost) at my wits end trying to make the issues discussed better, because as you said without a total rework of the multiplayer code, there is no solution – just things to mitigate the symptoms. I’ll post below some of the scenarios we have tried in our testing.
We run a dedicated linux server, using Gportal as the host.
We play on a heavily modded version of valheim (90+ mods)
On our Friday game nights, have 4-6 players online simultaneously, all usually fighting and playing together in the same zone/instance, and this is where the lag, rubberbanding, teleporting, chest/door delay, weird boar behaviour etc. etc. pop-up.
The 3 mods the supposedly help with network performance I have used, and am currently experimenting with are:
Network
Compress
BetterNetworking Valheim
Network: Have been using this pretty much from the get-go (2-3 months of running a server, and trying to find ways to make it less laggy) This is the mod that unlocks the send/rate and has so far shown to improve results, significantly (but the issues still happen with 4-5+ players, and depending on the circumstances of the game etc.)
Compress: Just installed this one, to see if it makes any difference, and waiting for a few game nights to observe results, currently its installed in tandem with the Network mod, which may or may not conflict with each other, if that is the case, will retest with just Compress mod installed.
BetterNetworking Valheim currently has an issue where it doesn’t work on Linux Dedicated Servers, the dev is aware and is working on a resolution – personally I have not tested this mod yet, but will once the fix is in place.
I have not tried direct assembly editing yet, and will include that into the testing with other variables in the coming weeks.
This past Friday, instead of our regular game night we did a testing night, loaded into the server with devconsole so i could spawn mobs and raids under controlled conditions and we can record results. The following results came from that night:
We tested several combinations of different things, including:
– Running the server using PlayFab/Crossplay Network Protocol (Vanilla)
– Running the server using PlayFab/Crossplay Network Protocol (with Network Mod)
– Running the server using Steamworks Network Protocol (Vanilla)
– Running the server using Steamworks Network Protocol (with Network Mod)
– Removal of JewelCrafting Mod
– Spawning fights and raids around the town, and in an empty area, etc.
The best combo so far is steamworks network protocol (crossplay disabled) with network mod, the removal of JC had no impact. After a bunch of testing the following is pretty much the ‘result’:
– Town will always be laggier, due to all the ‘stuff’
– Fights outside of town are much less laggy
– Using Playfab/Crossplay network protocol was massively laggy compared to steamworks, to the point where it was game breaking / unplayable (huge load times, doors taken 5+ seconds to open, etc) (rolling back to steamworks but keeping everything else the same showed a significant improvement, only lag was fighting mobs in town + some other minor hiccups.
For example with 4 of us we were able to fight 20 skeletons, or 15 draugrs, etc. with no lag witnessed for anyone. (In an empty zone, no player built anything, etc) this is all still highly connected to who ‘owns’ the instance of the zone this is happening in. we tested this as well by having everyone except the player with the lowest end machine stay connected, so they assume control of the zone, and then have people reconnect, even in this state there was no/very little lag fighting a significant quantity of mobs, that said – fighting the same amount in our town, that has a large amount of buildings, chests, crops, cooking stations, etc. the same test yielded significant lag when fighting mobs.
I will continue to test different scenarios and report back
Hey vi0lation,
Fascinating results, what a great and detailed writeup! Your findings are definitely 100% consistent with others here as well so far. It sounds right to me that the network mod would have the impact you’re saying.
I talked about it a bit in my last reply but 4-5 is about the right number of players for the mod to have an impact. The type of mod I’m covering here won’t get you much higher than 5 players. It basically makes 4-5 players go from unplayable to playable (but definitely not perfect).
Your observations about who “owns” the zone having an impact are also correct and we haven’t really talked about that much yet. I’m not sure how it decides who owns the zone but first come first serve seems logical. I don’t think it does any type of smart balancing where it decides which player has the best connection unfortunately.
There’s just no way that the developers who went with this multiplayer model would even be thinking about a problem like this because if you were you would have never chosen this multiplayer model in the first place (which they shouldn’t have). I’d imagine that once you own the zone the only way to relinquish control is to probably leave the zone or maybe even the game entirely.
I would say from my experience that 5-6 is about the most you will get with the mod I have covered for the client/server (and 6 is really going to depend on the players having high quality connections, all of them). I could see you extending this though with tricks like the ZDO compression. The problem basically is that the Valheim multiplayer model is so inefficient that we are literally hitting the limits of people’s network stacks. You can literally overwhelm their connection at some point and 5-6 players seems to be about where that starts to happen (even with the mods). It feels like you hit a wall where the tweaks we’ve made just aren’t enough to keep the data flowing anymore.
I could absolutely see you extending things to 7-8 though comfortably with some of the mods you’ve spoken about here. The ZDO compression seems like it could potentially buy another player slot before the problems start. It’s all a game of figuring out how far out we can extend that wall where the poor multiplayer model choice catches up with it and brings the game to an unplayable state.
Your messages about the lag around town make a lot of sense too. It used to be terrain modifications would lag it quite a bit but the terrain modification lag has been fixed for the most part (although I think you can still create it by digging out a giant mountainside or something ridiculous). I’d imagine this is related to the settlement entities such as campfires, workbenches, torches which all have “states” such as how much resin is in the burning torch. That all has to be sent to the client and tracked so that when you walk up to it you see the correct amount of resin in the torch or wood in the fire if that makes sense. This is just that much more data which will take us to the breaking point faster than the empty woods for sure.
Tracking resin in torches and wood in fires and the other building entities/items/properties might not seem like that much but it really adds up fast. How many torches/fires are in a single settlement? They’re constantly burning and losing wood/resin. A lot of my settlements probably have dozens or hundreds of them for the really big ones. Some things won’t populate during lag like chests/containers but a lot of entities/buildings you will notice do and those are the ones that I’m talking about. Those are the ones that have states that are being sent to all of the players in the zone all of the time.
Notice that it doesn’t lag showing you how much wood or resin is left in a torch. It always knows because this is *always* sent to you (and way before you walk up to it) otherwise if you walked up a torch when you were lagging it would show something like ? / 4 resin until it stopped lagging. It is a disaster to send all of this unnecessary entity data to everyone when we’re so close to the brink with Valheim’s multiplayer model. We literally don’t have any data to spare.
It would actually be better if all of this data wasn’t sent and it did show ? / 4 resin during lag. That would save a lot of data considering the chances of a player actually walking right up to one specific torch in that entire settlement and staring at it to see how much resin in it is probably incredibly low. They probably will never bother to touch it or even get that close to it until it burns out. Multiply that by an entire settlement of torches and just the developers changing that (or a mod) would have a significant impact reducing lag near any type of settlement as you’ve been observing.
In other words when you try to open a chest you won’t see what is inside it when the server is lagging until the server catches up. If everything in the Valheim settlements worked that way they would not lag the server the way they do. There are other types of entities such as anything that burns that doesn’t seem to work this way in Valheim and are being sent to the players in the zone all of the time.
Thanks for such a detailed report. I look forward to seeing what you find and if you are able to comfortably use these mods to perhaps extend your server to support 6-7 players potentially without hitting the doom spiral of lag where things just keep getting worse and worse until it grinds to a giant 30 second lag spike and all the crazy stuff that can happen. I’ll stay tuned for sure!
All seems to add up indeed, I’m wondering with the introduction of Crossplay this is when several people have started reporting worse performance for Mistlands, as you now have to manually disable crossplay on a host to force it back to steamworks. And the difference between Crossplay/Playfab vs. Steamworks was night and day.
In terms of my understanding of zones (This all comes from hanging around several discords with fairly prominent valheim mod devs and gurus, so I am just repeating what seems to be the general consensus on zone information)
– Zones are 64×64 sized grids
– The first player to reach/load a zone becomes the owner
– That said, to further complicate things, its not exclusive control, another player that interacts with an object (for the first time) inside a zone even if they werent the first there, becomes the owner of that specific object inside the zone. When your watching the log output of bepinx when playing multiplayer, you will often see the console spam with ‘attempting to interact with xxxx – but i am not the owner’ messages, this is essentially your client trying to interact with say a chest, or door that someone interacted with earlier and first, and therefore control it, but due to the backlog of multiplayer lag, your client cannot receive a response until that backlog queue is cleared and the owner can reply with the return, hence the console spam of that message until network congestion clears.
There is a mod that visualizes zones inside the game engine: (It’s pretty interesting to see live.)
I’m currently looking at the config of the betternetworking_valheim mod, and there are several configurable options:
# Setting type: Options_NetworkQueueSize# Default value: _1024
# Acceptable values: _10240, _1024, _512, _128, _10
Queue Size = _1024
I’m wondering if this is patching:
int num = 10240 - sendQueueSizeand
if (!flush && sendQueueSize > 10240)
In the ZDOMan Assembly inside assembly_valheim.dll as per your article.
And:
# Setting type: Options_NetworkSendRateMin# Default value: _300
# Acceptable values: _400, _300, _200, _150, _100, _50
Minimum Send Rate = _300
# Setting type: Options_NetworkSendRateMax
# Default value: _INF
# Acceptable values: _INF, _400, _300, _200, _150, _100, _50
Maximum Send Rate = _INF
Which I’m wondering if its patching:
GCHandle gchandle3 = GCHandle.Alloc(153600, GCHandleType.Pinned);In the zSteamSocket Assembly inside assembly_valheim.dll as per the method presented by Pat in this forum. I’ve asked the mod author to see if that is indeed what it is doing or not, would be interesting to know.
Hey vi0lation,
Oh wow, that’s some fascinating information about the zones. What a mess!
I believe it though. I’ve seen some *crazy* bugs in the past related to zones. I’ve seen a one-shotting dragon elite on my screen where all of the other players on the screen saw a forest troll elite. I was hit by a single cold ball projectile from his entire volley wearing full cold resist gear and instantly popped into a grave. When I logged out and back in I saw a forest troll elite like everyone else when going back to my grave.
I was never able to explain that incident because if one person is in charge of the zone that incident doesn’t make sense. How could my monster on my screen disagree with theirs? This is how. Exactly how you said.
Somehow what ended up happening is an elite spawned but on my screen I was in the mountains and on their screens (or whoever was controlling the zone) I must have still been in the forest. I was flirting along the border when this spawn happened. These mechanics you’ve outlined here explain how there could possibly be a disagreement like this between the clients. One or both of our consoles was probably full of log messages saying “I am not the owner” while they were trying to solve this and I got nuked by a bugged damage projectile from a monster that isn’t even supposed to shoot projectiles (I guess he can throw rocks). This was essentially an error state and I wish I had known to check the console at that time (or had the zone owner check their console).
Those are some interesting options for the mods. They definitely do look like they would correspond to them and they would have to modify them I’d think to change this directly or indirectly.
It’s definitely interesting to look at the Valheim code because you just keep thinking the whole time “why did they add this code in to artificially restrict and make their own game perform very poorly”? The answer is if you let it run wild it would use such unimaginable amounts of bandwidth trying to scale this to more than a few players that it’s actually able to bring an enterprise connection to it’s absolute knees. Like if you tried to play Valheim at work you could potentially lag the entire company internet line to nearly a standstill without these limits in place.
We’ve had people try completely removing all of the limits before in the past and this is what happens. Someone got up to like 80 mods installed on their server and had definitely high player counts and what ended up happening is exactly that. It scaled so poorly (and the 80 mods adding additional data weren’t helping) that it basically completely swamped the player’s connections (just with pure data / packets).
And by swamped I mean *nothing* on their networks worked. Streaming would crawl. All web sites would crawl etc. I bet someone looking at a traffic monitor for that server would conclude they were under some kind of massive DDoS attack (or their server had been hijacked and is trying to perform some kind of DDoS attack on your player’s IPs). It certainly wouldn’t look like typical gaming traffic.
That’s how out of control it gets and I’d imagine why they put these artificial constraints in. Valheim doesn’t scale and using this multiplayer model you can do things like bring an entire enterprise’s network to a crawl with less than a dozen people on there (to get a sense of how much it can’t scale, it can’t scale to the point where it’s dangerous to not put these limits in the code).
Definitely let us know what else you find!
Would it be possible to list what version of Valheim the hosted modified assembly is for? Current version of valheim is 0.212.9, is the hosted assembly for this version? Thanks for all the research and hardwork investigating this!
Hey vi0lation,
I honestly haven’t ran Windows for years but if you’re running Windows you can just download the file and right click on it and choose “Properties”. It will tell you the version. This won’t work in Linux or probably Mac OS X either if Valheim even runs on that.
Or if you don’t have Windows we can use some basic logic. The file download has the date 12-21-2022 on it. What patch was released around that time? This one.
There’s two ways to find out at least as if I answer your question the answer will be outdated within a week or two. That is how you check!
You will have to replace this every time it’s updated and if Sketaful hasn’t got around to it then you can always do it yourself on any version. I honestly expect to see 100 more posts like this over time including every time it’s updated which is why I’m responding this way. You’re just the first.
I don’t maintain the file and no I’m not willing to update this article every time it’s updated with the latest version. I’d encourage you to ask the developers to fix the game you are playing instead of put those requirements on me and Sketaful. He’s doing a public service here already beyond what I’d be willing to offer.
I’ll tell you guys how to do it but no I won’t update it every time. If Valheim wants to pay me to develop this mod for each of their updates I’m willing to do that. If you’re personally willing to pay me for every Valheim update I will also do it for you to save you 2 minutes of modifying the DLL yourself. Otherwise the people being paid should probably do that.
I don’t think that people understand this isn’t a solution. This isn’t even a good workaround. It’s the best we can do for a game that is broken. That’s it. Asking to have this mod enshrined in some kind of process with posted versions and this whole ordeal… I mean are you going to pay for it? Why did you take the time to ask me to do this rather than ask Valheim why this is an issue in their game? I guarantee that you didn’t do that and that you won’t.
I honestly wonder if I have cursed Valheim by releasing this and giving people an alternative workaround to the broken game. I doubt this will ever be fixed now and it will just be people expecting someone to make this dll for the rest of the game’s existence. If you guys had no choice but to complain to the developers then they wouldn’t get a moment’s peace about this issue and it would have been fixed a long time ago. I might personally be responsible for it on some level. I really believe that at this point.
Hopefully that helps!
thanks for the reply, all good – just modified the dll myself based on your instructions, and read the entirety of this thread. 100% the same boat me and the 4-5 ppl I play with are in. same lag, teleporting, rubberbanding, etc. etc when we are all in one instance. I have tried running client/server with smoothbrain-network mod (which supposedly unlocks the caps without having to edit the assembly) and just recently trying the comfy-compress mod, which supposedly compresses zdodata before transfer to cut down on amount, but no testing done on this yet if it makes it worse or better.
Will try the modded assembly on server, and clients and retest as well to see if i can add to the pool of tested scenarios.
Hey vi0lation,
Excellent work, sorry for the frustration in my first post, I promise it’s really frustration with the Valheim developers more than anything.
I’m a little frustrated with the players not complaining but to be honest with you after reading your reply I don’t think you fall into that category. It sounds like you’ve been working on this issue for some time and are probably as frustrated with the Valheim developers and the networking code as I am. You’re actively investigating this issue and deploying code/mods to try and address it and that’s what this post is all about.
The smoothbrain-network mod I haven’t heard much about. It does sound like it does things similar to the mod. It would make perfect sense for someone to make something like that!
Compressing the ZDO data makes perfect sense as well. The mod touches on the code that controls that but it doesn’t apply any type of compression of course. We are basically just widening the opening data can flow through and reducing when Valheim tries to throttle itself (which is mostly by dropping packets, yay, and when it isn’t the entire server lags and waits for it to catch up).
I’d actually be very interested in your results. If you could tell me how some of those mods compare to doing the assembly mod (like are any of them doing the same thing, does the modified assembly have other effects than these do, etc.) that would be very interesting information that nobody else has shared here before.
The good news is you are right at the numbers where the mods have really had an impact. 4-5 players is just right. That’s exactly how many I had when I developed the mod and it’s how many a lot of the groups had that were able to beat the game together had that found success using a combination of the server side mod from the article and the client side mod posted about here in the comments by pat.
I look forward to hearing how it goes!
Worth noting that crossplay does enable compression in vanilla Valheim. It just uses very slow compression, though it achieves a good compression ratio.
Hey CW_Jesse,
That absolutely is worth noting. Thank you for sharing!
Furthermore, ZPlayFabSocket’s.GetSendQueueSize() multiplies its result by 0.25, presumably to send more through the compressed socket. Dividing the result like that instead of subtracting 30KB (for a total effective queue size of 40KB) means slightly inconsistent amounts of data are sent.
Hey CW_Jesse,
That makes sense. Are there any negative consequences of that? We’ve seen people cause some problems like this before by having mismatched queue sizes between each other. Patk noticed massive packet loss when the queue sizes didn’t match and I believe this was even between clients (meaning if one person has the client-side mod on your server they all should).
I suppose if everyone is multiplying their result by the same 0.25 though that everything should always match up. It’s only when they *didn’t* match we saw problems.
Very interesting info, thanks!
That sounds like a reasonable conclusion for people to have reached given how they reached it, but also a wrong one given both my understanding of how Valheim’s networking works and my in-game testing. Better Networking allows you to change the queue size on the fly and that’s never caused an issue. It’s even beneficial to have people on different queue sizes based on their available upload bandwidth. Since you’re modifying the assembly directly, I’d sooner reach the conclusion it’s related to that, but I don’t see how that would explain your experiences either. Sorry I couldn’t be more informative!
Hey CW_Jesse,
Don’t sweat it at all, I think you were still pretty informative here! That definitely makes perfect sense and that’s how these types of settings are meant to be used in other games.
The Valheim code to handle this though is quite a mess. They actually used to have a variable that controlled this. I covered it in my original Valheim mod that worked at launch. They *had* a variable at launch and they *removed* it and hard-coded it!
Have you considered just throwing out the Valheim networking entirely with your mod? How deeply intermingled is this stuff in there? I can only imagine how much harder it would be than it should be due to the mixing of the client-side and server-side models.
Maybe I’m giving away good ideas for free here but it seems like if you rewrote the Valheim network stack it would essentially be a completely different game. If you went to a true client/server model (like I used in my Spark game here, which can still be played at armorcritical.com) it would feel more like hosting a Minecraft server (and probably have about the same # of player capabilities as people could get if they ran a Minecraft server on that same hardware, maybe even better since Minecraft actually has a lot going on with chunks/blocks that Valheim doesn’t).
Imagine a Valheim server that supports 12-24 players (or even potentially a lot more) because it is no longer using this partially client-side or peer-to-peer hybrid model or whatever you want to call it. I really don’t think there’s a proper term I’ve ever heard for the way they mixed and matched different multiplayer models in this game. It has the weaknesses of all of the models (such as if the server lags it screws everything up, but if the client lags it ALSO lags the entire server for certain operations, ugh) and ends up with none of the strengths of any of them.
That has to be the #1 dream of a lot of the bigger and more popular servers out there for Valheim. The player cap we have now pretty much restricts it to a group of friends. If you could rewrite the stack I’d imagine people would pay for that mod (or maybe you offer it for free then above 10 players is a paid unlock, who knows). Obviously all of the clients would have to be running the networking mod as well in this situation as it would basically be a full rewrite of the networking code.
I’ve got my hands full but to be honest with you this is what I would try to do if I attempted to do more with it. It would be an achievement no other mod could ever top. It would almost be a historical event for the game if someone did that because like I said it opens things up that we could never do before on Valheim (but that people desperately want to be able to do).
I can only imagine how much work it would be. It’s even more work doing it to someone else’s game rather than your own game. It’s one of those things though where if it could be done it’s valuable enough that you could find a way to make money with it (especially for servers looking for 9+ player custom solutions).
You would have to pay for all of the developer’s sins yourself to do this. All of their decisions they made such as projectiles all being client-side and having no server-side collision, yep, you’re going to have to do that. You’d have to implement everything they didn’t want to do that led them up to this point. If you did though you would be responsible essentially for making Valheim into a game that can have servers with entire communities (like Minecraft servers) with dozens or potentially hundreds of players.
No one could tell you the potential of such a game. It’s never been done. The Valheim developers couldn’t do it and apparently don’t seem to be planning on doing it any time soon that I’ve heard of. All I could say is that it definitely would have potential (and I’d imagine most Valheim players would agree)!
If you do a true server/client model, then the server needs to do physics simulations. The Serverside Simulations mod does that, but it greatly increases the required server resources, both CPU and upload bandwidth. Eventually, the server becomes the bottleneck. Can we do better?
With the current Valheim method, you could theoretically have hundreds of players on a server as long as they were evenly distributed in different zones. The biggest problem is when the weakest link becomes the zone owner or the strongest link is overloaded.
The current model is actually better than it might appear at first glance… if done well. Simply having smarter logic for choosing who the zone owner is (based on CPU speed and upload bandwidth) would make a world of difference, and I could probably do that for Better Networking 3.0. Having smarter zones instead of a grid would be a further improvement, but that might not even be necessary.
There is a way to make money from this, but it’s not from making a publicly available mod: It’s either from becoming a server provider that outperforms (or is cheaper than) other services, which I could probably do, or using the fact you built a popular mod on your resume to get a job in tech, but I already have a job in tech. Most people don’t want to pay for mods. They’d sooner reverse engineer them than pay. But people do want to pay for servers. I appreciate your thoughts. If I do end up becoming a server provider, I’ll throw some money your way to pay for the idea. 😛
Hey CW_Jesse,
That makes sense to me. You’re essentially saying it would be better to offer it as a service. I think that is smart for sure.
You’re right about people wanting to pay for mods I think. I think a service is a better refinement of the idea in the modern age. Reverse engineering is for sure a reality. I’m mostly saying it would be so much work and effort to do what we’re talking about (going beyond modding more toward a complete revamp/rewrite) it would have to make money somehow (unless one has all the time and resources in the world I suppose).
I am definitely expecting someone to make the exact thing we talked about here though whether either of us do or not. If someone wasn’t already working on it they are now after our conversation. Maybe you can optimize the system enough to achieve basically the same effect as moving to a model that looks more like a Minecraft server (an example I continue to use because of how appropriate it is).
Minecraft wouldn’t be arguably the most popular multiplayer game of all time if it was limited to 5-8 players. It sure would be nice to see what Valheim could do by achieving the advanced multiplayer capabilities Minecraft achieved more than a decade ago.
Minecraft is definitely more demanding to run than a Valheim server but able to run easily on a Raspberry Pi nonetheless. Doesn’t seem like too outrageous of requirements to me in 2023. Much more advanced/deep block and map scales and mechanics going on there than anything in Valheim as well. A Raspberry Pi can still do it unless you have more players than you can have on Valheim on it. I’ve been doing it for more than 5+ years. At least I’d published work by 1.12 on it it looks like showing others how to do it.
It’s pretty tough for me to swallow that it’s going to melt people’s hardware to have a client-server model for Valheim when Minecraft *definitely* has a lot more going on and runs on a Pi with more players than this game can handle. My games are far older than that and all used client-server models and none of them lagged as bad as Valheim does (and they had all server-side damage/physics/the works). That was 15 years ago in some cases. I haven’t made games for more than a decade. The servers had a fraction of the bandwidth as well. We’re talking about like early Steam levels of data (like CS 1.6) and they could all do this back then as well. 15+ years ago I was also running Steam servers that had server-side physics/the works for games like CSS. We had to due to cheating.
It’s beyond archaic to rely on the client. In fact the other problem you will face with this model is exactly that: cheating. Valheim is trusting the client. This is the biggest reason the model won’t work for servers with a large # of players. Cheating is already bad enough on Minecraft on large multiplayer servers and that IS a client-server model game. They still find the tiniest loopholes and the smallest places where the developers trusted the client and milk some crazy exploits out of it. I shudder to think what kind of exploits would exist in a large Valheim server with client-side trust like this. People would be able to do some outright spooky things with a malicious/hacked client (and this probably already happens today I’m guessing). They’d be able to do things you’ve probably never seen cheaters do in practically any other game and it would be absolute mayhem.
All of my arguments are for a perfect world. That would be a debate I would have with the Valheim developers about client-server model vs. what you’re trying to do and you’re right. I think this model *could* work if it’s implemented correctly but I mean this is cutting-edge genius level stuff if you can pull it off properly.
I really don’t think this system was born out of genius. I don’t know the story behind it but it wasn’t a good decision to limit these servers to 5-8 players via this model or it’s implementation (proper or improper, this implementation can’t scale to the kind of numbers people want on the game). In fact like I argued Minecraft wouldn’t have become the game it is with those limitations and I wonder what Valheim could become and what people could do if it didn’t have them.
The game was originally made by 4 people and I imagine one of them had experience with this type of multiplayer code or something like that and the explanation is probably that simple. As the game got fleshed out more and more going back and changing this isn’t easy even though now they have a lot more money and staff.
Definitely don’t hesitate to stop by if you have any news to share about Better Networking 3.0. I think it’s great that you’re working on the mod and giving it out for free. I’m really not expecting the Valheim developers to change the networking at all so I believe your skills will continue to be required for this issue.
I have no issue with you optimizing the zones or improving the system through mods for sure and in fact I’m legitimately interested in what you’re able to do with this and how many players potentially you may be able to get on a server.
Thanks for a very informed conversation and sharing your unique background/experience with the Valheim code. I’ve learned a few things for sure. Take care!
Yeah, cheating is, I think, the main reason to switch to a true server/client model. While you’re right to say Minecraft is more intense than Valheim all things being equal, not all is equal; switching to that model would also require me to optimize other parts of the game, not just the networking stack.
Clearly they spent their limited resources well; it’s a great game with great reviews. For every issue that bothers us, there are dozens of things that are so well done we don’t even think about them and are the reason we’re having this discussion in the first place, and as you said, we don’t know the story behind it. It probably would’ve been easier to just have a server/client model in the first place than to write code for switching zone ownership and such, which maybe means they realized it would be easier to switch to this hybrid model than optimize what was already written. Or maybe multiplayer was added late in the development cycle. Either way, we’re looking at the result of hundreds of good decisions, even if the end result isn’t exactly what we want. There’s a good chance it’s not exactly what they want, either.
Thank you for the great discussion! And thank you for these blog posts and discussions: they were one of my two original sources of education for making the first version of Better Networking in 2021, which is why you’re linked in the mod description.
Hey CW_Jesse,
That’s fantastic, I hadn’t realized that! You did exactly what I was hoping someone would do which is turn this into a more formal mod. You’ve definitely gone beyond anything we’ve done for sure at this point with directly modifying the assembly.
I completely agree with you about what they did with their limited resources. It’s an incredible story. According to Wikipedia and this news article I lied. It was actually 5 people that made Valheim. The article’s title:
Valheim: How a 5-Person Team Created the Most Popular Game on SteamThat’s an achievement nobody can take away from them. The Mistlands have also been a huge success and were honestly a lot crazier than I was expecting (in a good way, like they were weird and interesting and fun to play). They achieved something that AAA studios burn billions of dollars in cash with teams of 100+ with 5 people. It’s legitimately one of the funnest games for multiplayer I’ve played since Minecraft and one of the closest feeling things to it (but much more adult and gritty) but still rewarding a lot of the same things like preparation/exploring/living off the land/etc.
I also think that’s why our discussion was so valuable too and all the discussions we’ve all had about it as a community as well. Everyone is super passionate about Valheim and that’s such a great sign. You know your game is doing great when that is the case. It’s when nobody cares that you have to be worried. We definitely all very much care!
I’m not sure if the plan is still for 9 bosses but we still have a long way to go. I still am secretly hoping that they do decide to change this. Now would absolutely be the time. I think if that doesn’t happen though that people like you and the rest of the community will keep finding more and more clever ways to milk more out of the system.
Definitely keep us posted about new versions and thanks again for the discussion!
When I try and Compile I keep getting error codes CS1001 and CS0246, any clues?
Hey KeenKeister,
CS0246 is:
The type or namespace name 'type/namespace' could not be found (are you missing a using directive or an assembly reference?)This is a dependency error. You may need to copy your entire folder instead of just the Valheim .dll file as some others have encountered recently. dnSpy is trying to resolve the dependencies and not finding them in your case here. Hopefully that helps!
TY!
No problem at all, enjoy and take care!