I recently received my Bobcat 300 miner* after months of waiting. Much of the advice I’d read/seen on the subject suggested to leave the hotspot alone and that it takes a very long time to sync.
This is true but it assumes you configured your network correctly first and have the correct port opened to the outside world. If it’s not it will take even longer to sync the blockchain and your hotspot will have a yellow “Relayed” status. This is slowing down your sync and will harm your rewards even when it finishes since your responses to challenges will be “relayed” and will often time out before they are relayed through other hotspots.
In this guide I’ll show you how to get your network configured correctly for your hotspot to sync as fast as possible and remove the “Relayed” status!
Overview
The Helium hotspot uses peer to peer networking similar to other cryptocurrency wallets (with “nodes” etc.) as well as BitTorrent and other peer to peer applications. Xbox Live and PSN also both use peer to peer networking where one console with a high quality internet connection will be chosen to be the “host” and the other players in the game lobby create a peer to peer connection to the host console.
Just like every other peer to peer application not having the correct ports open on your firewall impacts your speed. On our Xbox Live/PSN example you would get a “poor” connection rating and never be chosen as a host. With Helium the “Relayed” status is the equivalent of the Xbox Live/PSN “Poor” connection quality rating. The ports are not publicly accessible over the internet and you get penalized / a downgraded experience for this.
Here’s an example of a relayed hotspot:
Relayed Helium Hotspot
The solution is to open the correct port on your firewall by creating a “port forwarding” entry in your router.
Port Forwarding
The specific instructions for your router will vary but can be found with a quick Google search. It’s likely you’ve encountered this before for something else but if you haven’t it isn’t too difficult.
Once you feel confident you have the knowledge/instructions you need ready to set up port forwarding the main critical port you need to forward on your router is TCP Port:
44158
This is the port that will eliminate the “relayed” status and allow inbound connections to your device. This will also speed up your sync as outbound only points are at a disadvantage here since sometimes the people who connect to you end up sending you blocks!
Once you’ve set up the port forwarding it should update the status on it’s own eventually or if you want to speed things up you can unplug the power cable from the hotspot for about 10 seconds and then plug it back in. With all the hotspots joining it may take 15-30+ minutes (depending on many factors) but it should show up as solid green with no “relayed” status:
Happy Status Helium Hotspot
Observe that even though I am not 100% sync’d I still am green status with 91.31% sync’d. This can get very sluggish to update sometimes but unless you’re seeing no activity whatsoever for at least a few hours I would urge leaving it alone for the most part.
Note on Offline Status
It’s not unusual for the hotspots to go offline when trying to sync with the blockchain. They should however still be connecting periodically and you should see your sync slowly making progress.
My hotspot has got stuck several times where it went off for the entire night and when I turned it back on it had made no progress.
My best rule of thumb is to leave it alone as long as the light is “green” (for the Bobcat miner*) unless you aren’t seeing any change or progress for a long period of time. This may vary based on your miner but the problems actually seem to be with the Helium network as a whole. Which miner you have can have an impact but most of the congestion is on the Helium network itself from so many new hotspots joining and syncing at once!
Conclusion
I’m still not fully synchronized yet with the blockchain but it’s staying green and doesn’t have the relay status. So many points are joining the network that the hardware is having a difficult time keeping up!
Remember, the relay status does penalize your mining rewards. It also prevents you from participating in several other features in the Helium blockchain so make sure you have a green non-relayed status.
It’s definitely frustrating how slow it can be but as long as you periodically monitor that the percentage is going up (wait a few hours minimum before power cycling) and have your ports opened you will sync as fast as anyone else is (slowly) but you won’t be at a disadvantage!
I woke up this morning to find the hotspot 99.79% sync’d. It was offline and after a quick unplug and plug back in I’m syncing again and at 99.81%. This is the way folks, open that port and get out of relayed!
Update 6/24/2021
Success! My hotspot has completely sync’d. I’ve issued and completed a challenge now and am just waiting to witness my first hotspot and even earned 0.05 HNT for some reason (maybe my issued or successfully completed challenge?):
Helium Hotspot Fully Sync’dHotspot First Earnings
All in all it took about 5.5-6 days. This is because of so many new hotspots coming online so if you’re in a similar time range don’t panic unless you’re still at a really low % sync’d. Don’t give up, mine took forever but it’s finally working.
Update 6/29/2021
I’ve investigated the effects of relayed vs. non relayed status. If you observe the connections with an enterprise firewall you can see how many connections it has at a time. When you are in relayed status you will only have about 4 connections. When your ports are opened you’ll have closer to 20-100 connections! All of these connections are transferring data so it definitely helps speed things up.
The point has been running for a few days now and has already earned about 15 HNT! Hang in there, it’s worth it!
Update 8/7/2021
If your miner and radio aren’t the same unit (not the case with newer miners) you should also port forward UDP port:
The document says that if your miner and radio are the same unit then port forwarding this from the internet is not important so the Bobcat 300 (and all other Helium miners that have a name) do not need this but I wanted to add it here for completeness.
In other words, your Bobcat 300 (and the other newer “approved” models) and your radio are already the same device so they don’t need any ports opened or forwarded to talk to each other even though you can see it listening on the diagnostics. On older miners (back when you were able to make your own homebrew frankenminers and join the network without a special enrollment key) this wasn’t necessarily the case and then you would need to forward this port.
A port forward should not be necessary for most people unless you know your radio is separate from your miner (it probably isn’t, and you can’t make these types of miners anymore).
If you didn’t build your hotspot yourself, from parts, by hand, you do not have this. An external antenna does not apply, That is NOT a radio unit. It’s just an antenna and it can’t do anything by itself without a radio unit controlling it.
A “radio unit” is the computer/device that the end of your antenna plugs into. It has to have a CPU to process signals from / send signals to the antenna. If your antenna plugs directly into your miner then your miner *IS* the radio unit for sure and this is almost certainly the case for you.
It’s no longer possible to do this (only approved manufacturer hotspots can join the Helium network now, but you used to be able to homebrew) and hasn’t been for a long time because people used to make tons of hacked/spoofed hotspots (still a problem to this day but it used to be much, much worse).
Was your hotspot manufactured by a company like Bobcat, RAK, Synchrobit, any of them? Then you didn’t build a homebrew hotspot from parts by hand so you don’t have one and this does not apply to you. There has never been one sold like this that has a name or brand. If you can tell me what your hotspot “is” and that actually means something you don’t have it!
Hopefully this clears up some confusion for anyone else who is investigating this. The new Bobcat 300 diagnostic tool will undoubtedly make many other people wonder about this as well. The answer is that yes the Bobcat is listening on that port, but it’s talking to another chip that is also literally a part of the Bobcat. It’s like a computer talking to localhost or 127.0.0.1.
It doesn’t matter what your internet connection’s firewall is doing in this situation. It’s only a “localhost” connection in a modern Helium miner’s case because it doesn’t need to make any external connections like old school homebrew setups may have had to at some point.
You for sure do not have to worry about this now with any hotspot that has a “brand” or “name” and you didn’t build yourself!
Hi James, as always, I appreciate your professional / thoughtful replies. I agree with your assessment about the “gossip part being very slow around letting miners know who’s online for witnesses / PoC especially when so many miners are being added daily”, but I thought that was why Helium started the VALIDATOR program. Validator nodes are a new entity on the Helium Blockchain that perform consensus group work, including verifying transactions and adding new blocks to the blockchain. It seems to me, it’s gotten worse even after adding VALIDATORS.
I also can’t help but wonder how flawed Helium’s algorithms are when I see three of the exact same Bobcat Miners at the exact same height, each are the only ones in their respective polygon but all connected (3 polygons in a row), one miner makes 27 HNT a month, while the other two miners make 7 HNT & 13 HNT. It makes no sense, especially when the are between 1/2 mile to 1.5 miles a part.
I watch the Reddit forum and get alerts when new comments are posted and no one has a good answer as to why such a big discrepancy in earnings exist. I’m venting I know, but I believe HELIUM needs to start owning the real issues that exist in the design of their blockchain and I hope the more people that read your blog and posts like mine start holding HELIUM’s proverbial feet-to-the-fire. I’m done ranting 🙂
That’s a fair question to ask why the validators didn’t impact this. Before they added the validators we were actually having chain halts all the time (some of which broke every hotspot on the network until there was a firmware update!). It actually used to be worse believe it or not! The consensus group would get hung and delayed all the time breaking all transactions on the network and you’d just have to wait and if it wouldn’t fix itself they would have to do emergency patches.
That kind of stuff really hasn’t been happening since validators came online so they did fix that piece of it but it’s still not perfect and some parts of the system benefitted more than others while others probably need additional tuning / an overhaul to allow it to continue to scale. It seems like something is still always broken but the validators did at least fix some things that used to be a constant headache that some people with newer hotspots may not even know were possible / used to happen.
Gossip actually seems slower to me than before validators but to be honest with you it’s hard to tell if that is because the explorer is really really slow to update (it was purely explorer 1.0 back then which seemed to work and update differently internally) or if the gossip is actually taking that long to traverse the network. It may actually just be the explorer issue since the “gossip” traffic is usually just about your status and the explorer is generally your only way to see that. I’m guessing there are other ways / alternative APIs out there potentially and it would be interesting to see how much faster they update than the explorer to see how much of the delay is the explorer and how much is gossip travelling the network.
The Bobcat mining sync issues are actually their own thing as well and reasonable criticism. Bobcat has actually admitted to this and said the recent OTA should help (but I would expect this to continue to be an issue, like an extra issue on Bobcats that is happening a lot more than other hotspots).
We expect this OTA will fix some of the most commonly seen issues that have appeared in the last week, including hotspots going back to syncing, hotspots unable to load snapshots, and hotspots stuck in syncing state for long periods of time.
I believe this is why you’re seeing such a difference in earnings as well. My Bobcat has *really* been struggling the past few days and hasn’t done anything for 24 hours. It’s about 3000-4000 blocks behind and keeps going yellow and red off and on so even with this update I’m still personally having Bobcat related firmware issues.
None of this is meant to be a defense of the Helium team as that is not my job. I just wanted to explain that you are having an extra bad run of luck here and are getting hit by like most of the major issues going on on the entire network at once at once (as well as myself for some of them) so your frustration is very understandable. I really am still looking forward for my Nebra outdoor point to arrive so I can compare it to the Bobcat’s operation (and I’m nearly positive it will work more reliably).
The good news is the Bobcat diagnoser actually is getting really nice and I can barely keep up with the new features they’re adding (it looks like the Net Speed test is now back again since I blinked). This is a good thing because we used to just look at it and wonder why the light was red (literally, that is about all you could do).
Excellent rant, I applaud it and just wanted to add in some nuggets of knowledge here and there!
Wow, awesome response once again. I must say, and please don’t hate me, I’m glad to hear other people are experiencing what I am. I’m a little technical and I was getting frustrated why I couldn’t pinpoint the root cause of my problems and why I was the poor unlucky soul this was happening too. Once again, I agree this is both a Hardware and a Software problem, I just wish both Helium & Bobcat had the support teams to help calm and navigate the storms. They haven’t perfected their LoRA network and their ready to launch 🚀 5G 😖.
BTW, I have my Bobcat Minor 24 feet high outside on my roof with an 8 dBi Omni Outdoor 915 MHz Antenna for LoRa HaLow / Helium and a 16 ft. Low Loss Antenna extension coaxial cable from Rockland. I can only witness 1 miner that is 1.25 miles from me and have been unable to reach the other 3 miners that are 3 miles from me. There are lots of trees 🌳 🌲 in between us so I’m hoping once the foliage is gone, maybe I will be able to reach this other miners but I’m not holding my breath.
On the bright side, I’m excited to get my Linkdot miners in December if all supply chain components work out.
Thanks Again for all your insight and let’s hope I can pay-off my Bobcat miner by December before I get my 3 new Linkdot’s.
No worries at all! I actually completely agree with the 5G take. I’m not a fan at all of this launch. All they had was that FreedomFi point and you had to pay for a waiting list reservation (which has been long gone for months).
To that I have a pretty hostile and tacit response: the Helium team and the FreedomFi teams can both suck my balls. I’m not interested in your crappy partner access point and will wait for a real company to launch a 5G point that is offered to everyone. Until then the 5G network can and will be trash and none of us will care about it and the coverage will be HORRIBLE. How many of those did they even make for those paid waitlist spots? Honestly disgusting and anticompetitive behavior for crypto, if you’re going to launch it then launch it fairly and across the board or it’s going to be trash.
If someone else makes a 5G network to compete with Helium I would not be surprised if someone gets the leg up on them. Helium might think they’re in a prime position to dominate the wireless/LoRa space forever but I think they’re underestimating just how easy it is to switch and how little loyalty anyone really has to Helium itself. We are all mining and getting in on the new wireless mining technology and I do not care what the currency calls itself underlying that or if I end up mining a different one for 5G than HNT. That’s what exchanges are for so I can turn it into what I do want.
I would not lose any sleep if I set up a different 5G network point at my house than a Helium one if there are good competitors available! I honestly would probably prefer it because I’m tired of technology always having one/two companies dominate entire fields. The second they do they start abusing it, and Helium is. I got my Bobcat a few months ago. It would literally be nothing to switch away from Helium entirely or choose a different one for 5G. The Helium team has never wrote me a check or gave me anything special to deserve me desperately clinging to their single-partner paid-waitlist preorders. There already are other LoRa crypto networks and there will be other 5G networks if there aren’t already and they will grow every time Helium does something anticompetitive.
I’m mining no less than 10 cryptos at home at this very moment and it would take me a minute to think of what they even all are! I hope the Helium team is actually planning a proper and fair launch for the rest of the manufacturers but if they don’t someone else will, they always do in crypto just like the Helium team themselves were once trailblazers and then once getting on top slowly became the villain. This is an extra harsh take (especially from me) but I feel like those teams really need some cold water poured on them right now. Their futures will be decided by their next steps here in a way I’m not sure they are quite fully conscious of.
Matt Blum
2 years ago
James, you’re a gentleman and a scholar with the critical updates you provide. I also have a Bobcat Miner 300 going on 3 months now and while I’m trying to remain positive, I can’t help but get enraged and negative on both Helium’s horribly designed / executed LoRA network and hotspot miner equipment manufacturers like Bobcat. I unplugged my Bobcat miner to move it to a different location and it was still showing ONLINE in HELIUM’s app for over 28 hours. Then this week my Bobcat miner all of sudden fell into a vicious sync cycle for 96 hours.
It was because of your blog that I learned of their utility tool and noticed they recently pushed a new update for FAST SYNC (alpha). I launched it and in 12 hours the utility now tells me I’m fully sun chef. But guess what, the Helium App is showing I’ve been synching for 20 hours when in fact I’m fully synced. Helium takes no responsibility for their crappy LoRA network that’s put together with bandaids & duct tape and can’t even provide near real-time updates on how your miner is performing. I have ZERO FAITH & TRUST in HELIUM especially when they rely on folks like yourself and the Reddit community to make sense out of the senseless.
I’m so frustrated because I never see an engineer from Helium post a YouTube video, provide commentary on their Reddit forum etc. What are your thoughts on this subject?
I completely share your frustration with how slow the explorer is at updating. It’s the “gossip” part that is really slow which is how your miner tells other miners it is online as well as whether it is relayed or not. It took someone’s 16 hours to update recently!
My main thoughts about this subject are that you are already on the right track starting to rely on the Diagnoser and other tools instead of the explorer. The new diagnoser gives you up to the second information about what the miner is *really* doing. It will tell you whether it’s online and exactly what block it is on. This is *WORLDS* more useful information than the explorer gives.
I’ll go even a little bit further though. The explorer is basically so slow it is useless (the only thing that shows up relatively quickly are basically transactions) and I would largely ignore it. Remember, the good news is that the explorer doesn’t actually *DO* anything. It doesn’t decide when you send a beacon. It doesn’t decide if you witnessed anyone. Nothing “depends” on the explorer to decide whether you earn HNT or not. Anyone could make their own HNT explorer just like lots of blockchains have multiple explorers (there probably are alternative HNT ones as well I just haven’t seen yet). It’s purely cosmetic / informational. You’re right that it’s near-useless for status updates because it’s so far behind it literally tells you nothing even if the miner has been off for 10+ hours!
The “fast sync” option is pretty interesting. At first I thought this was the same as just syncing from the “snapshot” that I’ve discussed here earlier but this does look like something different. The full feature breakdown is here. I’ll get the guide updated in the near future here but this does look worth trying if you are behind on sync (it says it won’t work unless you’re behind a minimum of 400 blocks).
I don’t have any silver bullets other than to tell you I share your frustration and can only say I recommend using just about anything else like you’ve been learning to do recently and continue on that path! Maybe they’ll improve the explorer later and it will update faster but for sure in the mean time I think you’re on the right track!
Martin
2 years ago
As a part of the broader Helium community, I would like to thank you for the knowledge you share and the effort you put in helping others. People like you make all the difference.
I have a question myself. I have 2 sites that I can not do port forwarding for various reasons. I’m looking for a way around this. I was thinking of a small box with a VPN service that allows port forwarding. I reside in Europe. Do you have any idea of a hardware/software combination that could help me resolve this? Do you know any VPN provider in Europe that offers VPN with port forwarding capabilities?
As a side question, the way I’ve understood it, the lag that the VPN will add will hurt me in terms of Helium returns. If I have understood it correctly having lower lag is better that having higher lag. Is this correct?
Thanks for the kind and gracious words, it is very much appreciated! I actually have heard of such a thing being used for my Minecraft server management project. Here’s a link to the thread. He’s also from Europe (Germany) as a bonus!
Theoretically you could run the ngrok daemon on one of the computers on your local network and have the daemon forward to your Bobcat’s address on 44158. The daemon runs on both Windows and Linux and you can see some more about it here: ngrok.com
They upgraded to a “premium” option and I believe you would too in order for it to be able to pass enough traffic to not impact your mining negatively. I’ve never heard of anyone doing this for Helium mining but is it possible? Absolutely it’s possible for the same reason this person was able to host a Minecraft server people could reach through a public IP from the outside. It’s doing exactly what you want it to do in this case, just for a different use!
I think this would work theoretically but I don’t know of a service that doesn’t need to run some sort of daemon on your local network so you would need to put the daemon on a computer that is up all the time for sure to be able to do this (and probably upgrade to a paid option). Since it does have the free option though it wouldn’t hurt to give it a try on the free option (even though it would slow down traffic you could confirm it 100% works before upgrading to a paid tier). That is an idea to investigate at least!
So the lag part of your question is really interesting and I’m glad you asked it in this way so I could get a chance to explain. There’s a couple of things to consider here. ngrok seems to be popular in Europe and have fast connections there but if you chose an alternative connection you would probably want to connect to it on your computer and see what your “ping” is.
If your ping is only going up by let’s say ideally 10-50ms but not a lot more than <100-150ms (bump up the numbers if you're in a remote location) then the benefits of being out of relayed will far outweigh the ping increase. It's not inconsequential though, you are right that it can negatively impact things (think of borderline cases where you barely miss the timeout but if your ping was 200ms less you would have made it).
The benefit of getting out of relayed status is that you will have so many more connections to the network (both inbound and outbound when you have a "clean" status) that your hotspot actually has a much easier time getting messages through to the rest of the network when some parts are busy/choked off. Basically you have more links/pipes to a larger and more diverse set of points and can navigate busy/congested times much easier than relayed points which have a limited number of outbound links only.
Why is this? Because a lot of the lag on the Helium network is because things are backed up (sometimes backed up for seconds or even minutes when it's really bad) so all these additional "pipes" give it alternative paths to still get through to the rest of the network when things are really busy. Basically there's different types of lag, and the ping time (latency) is only one type of them and a lot of the timeouts/problems on the Helium network are severe congestion measured in minutes at some points and not in milliseconds!
So that's the main tradeoff. The sweet spot will be to find one that has a minimal ping impact in Europe (the lower the better) but getting out of relayed status also dramatically improves your network link quality (and will often far outweigh even if you had the slightly or even moderately lower latency in some cases). Hopefully that helps, definitely let us know what happens if you try it because this is hot off the presses and not been done before that I know of!
Oops! Err, actually I did just think of one problem. I'm not sure how you would get the Helium miner to advertise your ngrok IP. It would have to be a service that is somehow controlled through some kind of web interface I would think or possibly a connection-wide or router-wide integrated VPN services. There might be a way to overcome this I'm not aware of / not thinking of in the moment but the main issue is the miner would be advertising your normal internet connection's IP and would not know to advertise a different one.
Which does give me one other idea. You could set up a VPN connection on a router with one of the VPN services that supports port forwarding. There are such VPN services that support both being set up connection-wide in a router AND support port forwarding like here. That’s not a specific endorsement of this service as I’ve never used it but is merely an example of a provider offering router-level integration and port forwarding services at the same time.
Here’s some other candidates that support port forwarding (the site claims most don’t, and you’d just want to make sure you can use router-level integration with them and it will be fast in your location).
Even if you just bought a router just to connect to Helium mining on a router-wide VPN that only the miner is connected to (using a VPN service that supports port forwarding on all ports, a lot of them are restricted to specific ports so double check this) that is my next best idea to get it working. This solves the problem ngrok would have because the router would actually give your miner your VPN public IP address so your miner would actually be advertising the VPN IP that your VPN-integrated/VPN-enabled router assigned it (as the WAN IP).
In the context of a Helium miner buying some cheap $40-50ish router that supports VPN connections (a lot of the providers that support this have specific models they even recommend) this might actually be worth getting out of relayed status (from a financial/investment perspective even, you would gain enough from being out of relayed status in additional HNT to offset the cost in some x period of time).
I can’t recommend any specific model/provider unfortunately as this is all highly theoretical based on my experience from others getting around NAT/firewall limitations in creative ways but if anyone reading has used good VPN services that support port forwarding / router integration and have low latency let us know. I’ve never heard of anyone doing this before and anyone who has isn’t talking about it and probably isn’t happy that I just said it was possible.
My personal belief though is that having the network not be filled with dead-weight restricted-networking points is better for everyone including the network/myself/everyone else and hoarding secrets/knowledge is not the way and harms the network and my personal stake because the network sucks to use so nobody wants to hurting everyone’s investment value (happens all the time in crypto). That’s my best fallback idea to get around this at least!
Leroy
2 years ago
Not sure why you think relay affects sync. Unrelated.
Update 6/29/2021
I’ve investigated the effects of relayed vs. non relayed status. If you observe the connections with an enterprise firewall you can see how many connections it has at a time. When you are in relayed status you will only have about 4 connections. When your ports are opened you’ll have closer to 20-100 connections! All of these connections are transferring data so it definitely helps speed things up.
You only have 4 connections to the network (all outbound, they have to be) when you are in relayed status and you have 60-80+ outside of relayed status. Those 4 outbound connections are all you have to work with to sync in relayed status and nothing more. You can also monitor it with enterprise gear and watch the data rates / sync speed to verify this. It’s extremely obvious if you even open an enterprise connection monitor status page when there’s 4 connections vs 100. Even a lot of consumer networking equipment has a basic connection monitor built in and you can literally see it with your own eyes in real time.
This isn’t some unknowable thing or my opinion. We don’t have to guess whether it does or not and I wasn’t. It’s really easy to test and I suggest you do so if you are running/managing points with this assumption so you can see for yourself as soon as possible.
Look at Helium’s own recommendations and explanation. Your hotspot can barely even connect to the internet in relayed status. Look at the diagram. How could you think this doesn’t impact your sync speed? Do you have some mistaken impression that only radio traffic is affected and sync traffic can magically make it through your relayed status when that literally means you can’t accept inbound connections of *any* kind? Which circles are you running in that are gossiping this nonsense that literally make no sense due to how networks work? You’re dead wrong.
I’m definitely not above reproach but this was tested/documented and it didn’t take a genius to figure it out (4 connections to the Helium network vs 80, hmmm…) but it’s difficult to accept this criticism when the conclusion was tested/measured by me personally but I do not feel your criticism is coming from a similar place. It seems like something you heard or mistakenly misunderstood to me and that you have no actual reason to believe it.
Are 4 outbound connections to the Helium network faster or the same at syncing vs. 80+ connections of mixed inbound/outbound? No, of course not. That’s why relayed status is a thing. It’s bad (for everything) especially the network and syncing, and not just for you. You’re making everyone else’s experience worse as well staying in relayed because now we have to try to carry your point’s dead weight for you and most of it will time out before we get to it anyway (especially beacons and witnessing which have very strict timeout windows, meaning you lose most your income being in relayed status except for during the least busy and most ideal periods on the network).
Again, this is why you are penalized for it and people with a clean status are richly rewarded for it. This is in both the network and the (properly configured) miner’s best interest and if anything I would expect relayed points to get punished even more in the future as the network continues to grow and the relayed points continue to hold it back / keep it from scaling the way it should with additional points being added (it doesn’t if they’re relayed points, they’re just dead weight).
If you can provide me with a single shred of evidence of what you just said (which flies against the documented data as well as the recommendations from Helium / Bobcat / probably all manufacturers) I will absolutely consider it and test/validate it. You’re also free to validate it yourself and not reply to this and go back to wherever you heard this from and start setting the record straight!
Tim
2 years ago
Hi James,
This is by far the place with the best information about helium miners. Thanks!
I read all questions and replies. Very usefull, but i have not found the answer to solve my issue.
I have a Bobcat 300 and I use a D-Link DWR921 4G LTE router. I use a utp cable directly from this router to the Bobcat miner.
The Bobcat is synced, but relayed.
And yes, port 44158 is responsible for this problem.
I did everything to open it, but the port test websites shows that it is still closed.
I tried to open the port conform below steps:
Navigate through the router’s admin panel and look for “DHCP Reservations”, Click on DHCP Reservations and assign a static local ip address for your gaming console or end-device and make a note a note of the ip address.
Then i went to next step and add the bobcat miner at Virtual Server. I could select The Bobcat device in the dropdown menu and I added the static ip that i made in previous step. I added the port to forward (44158) and select the enable checkbox and saved it.
This is all what is mentioned online that you need to do for this router to open a port, but the port is still closed.
Thanks for the kind words! Your configuration actually looks 100% correct. I think you may be having a different problem.
A lot of 4G / 5G internet providers don’t give you a real public IP address (it’s NAT). Verizon is one of them. AT&T also doesn’t give you real public IPs by default and hasn’t for years according to this.
It sounds like these networks are behind NAT and I think that is the problem. Are you able to open any ports no matter what port it is and have them show as opened from the outside? My guess would be no unless it’s one that your provider is “allowing” through. You apparently aren’t even able to ping them from the outside supposedly if your line is under these restrictions.
It seems like all of these companies offer real IPs but they charge extra for them. That AT&T link even made it sound like you have to get a business line to get a real public IP that would allow port forwarding. Verizon sounds like it’s a one time $500 fee and also considered business class service ($$$) from this document.
Having a static IP is not the same as having a real public facing IP (you can have a dynamic public IP address which is what CenturyLink gives me against my will) but it is treated mostly the same by a lot of these wireless providers from what I can tell and they seem to have gone with straight NAT other than dynamic public IPs.
It doesn’t really seem like there’s a dynamic public IP option, it’s either NAT (can’t port forward) or a business class static IP but it may depend on your provider for sure. From what I can see the major providers here in the US have you by the balls as every single one I checked doesn’t give you a public IPV4 address that would allow port forwarding. Hooray for fake late-stage crony capitalism where the objective is essentially the same as the Monopoly game you played as a child!
Accessing ancient 1960s/1970s public IP address technology has become quite expensive/impossible (at least in the US, but international commenters have told me it’s not much easier (or far worse) to get public IPs or static IPs there in most other places too seemingly) thanks to these market conditions (anti-trust hasn’t been enforced for decades in nearly any industry, most of them only have 2-3 companies left, it’s a runaway train and the train went bye bye long ago).
In this brave new world if the 2-3 companies left decide you don’t need a public IP address anymore as a consumer/residential customer then that is immediately the way it is across the land (and often times the world in the really bad industries like cell phone operating systems i.e. Android/iOS). Who are you going to switch to?
Which provider do you have? Can you try testing if you can open any ports from the outside (other than 44158)? It seems like you’re behind your provider’s NAT (willingly or unwillingly) potentially here!
Tried to open 2 other port, without any positive result.
I found this information from my provider Orange.
It is translated to English:
Do you like to play online games and use mobile internet? Meet the Orange VPN service
Orange VPN – is a service that allows you to access the Internet with a dynamic public IP address. It allows, among other things: to use mobile internet and to connect to selected services, for example XBOX Live, Playstation Network or other game services.
Orange VPN Static provides internet access with a public IP address. The service allows the customer who uses mobile Internet as a home to use multimedia functions such as smart home, connection to web cameras, online game servers, etc.
Orange VPN is a special service:
• Players using mobile internet
• IoT Internet of Things
• Smart home solutions
• Mobile video surveillance
Orange VPN is a service that allows you to access the internet with a public IP address (variable or permanent) from a device equipped with a SIM card, giving you remote access to this device.
However, remember to protect access to devices in your home with a strong password
.
The service is available in two variants:
1. With dynamic IP address in combination with APN “vpn” – VPN
2. 2. With a fixed IP address as part of connections to APN “vpn.static.pl”
So i need option 2?
BTW, option 1 cost 1.85 usd per month and option 2 cost 5.27 usd per month aprox.
That’s actually fantastic news in this case that you aren’t in the US. That is exactly what you need to get! The Xbox Live / PSN part is the dead giveaway. Basically those services will give you a “poor” connection if you don’t have a public IP address that can allow port forwarding (usually through UPnP or Universal Plug and Play but the ports can be manually opened/forwarded). This is because your console can be the “host” or “server” since many/most console games are peer to peer networking. If you can’t forward ports you can’t be the host so you have a “degraded” connection in that case and both services will complain about it. Even though we don’t care about our Xbox / PSN connection in this case it’s actually a really good sign on a technical level that the service is going to do what you want it to do.
This likely caused them to receive enough complaints to start offering this service as an extra since there are situations like that where you will have a “degraded” experience without a public IP. This is for the exact same reason you can’t forward to your miner. It can’t be done in this type of a network and they need to actually assign you a public IP for other members of the public to be able to connect to you specifically from the outside. You can’t “share” an IP with 100 other people which is what they are doing on most of these 4G/LTE networks. IPv4 addresses are limited and have been for a while which is why they do this. Every ISP has a limited block of these addresses so they’re charging you a fee to take up one of these limited reservations. The fee is lower if you’ll take whatever is available (dynamic) and much higher if you always want the same IP (fixed/static).
To understand why you can’t share one think about what would happen if a single other one of those people were running a Helium miner. How would it know which one to send it to when someone connects to that shared public IP? That’s the fundamental underlying problem they are solving by assigning you a public IP (dynamic or fixed, doesn’t matter). You just need one IP that goes to you and that is how people will connect and talk to your miner. Right now you’re sharing and there is no single IP that actually goes to *just* you in the current setup.
I personally have a dynamic public IP and so do probably most people mining and it will not put you in relayed status. My Bobcat has paid for itself several times over at this point for sure so if it makes a difference it would be marginal (unlike getting out of relayed status which is well known to make a large difference as they pretty much “punish” you for it as everything important will time out before getting relayed >50% of the time). You shouldn’t need to spend 3x-4x as much to get a fixed or static IP. The $1.85 option should work just fine (the dynamic public IP option) and give you a dynamic public IP that can be forwarded!
I would definitely try it as-is first. It looks like their instructions gave you a new APN to possibly put in the hotspot settings but maybe that is done for you automatically, but I would go ahead and try it!
It takes several hours at least for the “relayed” status to drop off the explorer because the explorer updates very slowly but I would leave everything else the same and give it a try using that service!
Yesterday i changed settings in the router and installed static public ip.
Port tested via website and it shows that 44158 is open. So it works, so you were right so far, but…….now, 12 hours later, the miner is still in relayed modus.
Pfff…..i know exactly why i am not work in IT business, it drives me crazy, haha.
That’s excellent you got the static lease set up! The explorer taking this long to update doesn’t sound very unusual honestly. My hotspot is fully sync’d and it hasn’t updated the explorer status in 6 hours. The explorer is apparently only updated by your hotspot “gossiping” on the network.
If you aren’t fully sync’d I don’t think it will update the explorer yet. If you’re getting an open port status then you are just waiting for the explorer to update basically which can take quite a while for sure. If it’s showing “open” it’s receiving connections and it’s only a matter of time before your hotspot’s “gossip” spreads around the network.
You can unplug the hotspot and plug it back in to try to get it to “gossip” with some different points on the network (the idea being it will not connect to the same outbound hotspots as before you rebooted it, it picks a new set of random hotspots) which can sometimes speed things up (you’re spreading the “gossip” far and wide by reconnecting to a whole new set of hosts and gossiping to them too) but there’s really no way to make the Helium explorer work faster / do much other than wait unfortunately.
You can see just about everyone that has ever had to wait for relayed status go away go through this here. The people saying they’ve been waiting 3 days definitely still have a misconfiguration but I don’t expect it to take much longer for yours and if it takes longer than I would say even 24 hours (it seems like it’s more like 12-16 when it’s cranky) then yeah something else is still definitely wrong.
The relayed status in explorer means nothing as far as to whether you are penalized. If you can accept inbound connections the effect is basically immediate (once you get sync’d at least) no matter how long the explorer takes to update. I’m guessing the bigger problem is it got behind on sync which seems to happen all the time on the Bobcat if you get a bad few connections for a while that aren’t sending blocks quickly.
It will happen less once you’re fully sync’d and out of relayed status once and for all because you’ll have way more connections to sync with (both inbound and outbound) which makes you far less susceptible to falling behind on sync for sure.
You can literally type your Bobcat’s static public IP in your web browser (on *any* device) like httx://X.X.X.X:44158 (mine at time of writing and for the next little bit was httx://174.23.155.49:44158/, any of you can go to that within the next day or so before my dynamic IP updates and see the same thing) and you should see this:
This page isn’t working
174.23.155.49 sent an invalid response.
ERR_INVALID_HTTP_RESPONSE
That is exactly what you should see because it’s not actually a web server on the other end so your browser can’t actually talk to it but if you get the page isn’t working invalid response then you know the firewall is letting you through and the port forward is working. If you see this then it’s only a matter of time before the explorer catches up with it.
It’s incredibly slow to update the explorer web frontend but if you’re able to reach 44158 from the outside with your web test it will update probably within the next 6-12 hours would be my guess but it couldn’t be more normal for it to have taken this long to update (it would be more surprising if it dropped off already)!
The syncing issue was just an app error, because my other miner showed also syncing status at that moment, while this one is at another location and i havent dont any changes to that miner. Anyway, the syncing status changed into online again.
I did the test with ip and i get the same message that the page isnt working etc. So that sounds good.
It takes about 13 hours now, still relayed. So lets wait for next 12 hours.
But as i understand it correctly, now it does not matter that it shows relayed, since i have inbound connection? How do i know for sure that that is the case and that i get full rewards and no penalty’s
The way you be sure is to do the test of going to your public IP in your browser at httx://X.X.X.X:44158 (your public IP as in the new static IP you got, not a local IP like 192.168.X.X, the one people will reach you from the internet with) and make sure you see:
This page isn’t working
174.23.155.49 sent an invalid response.
ERR_INVALID_HTTP_RESPONSE
Everyone in the world should be able to access that port (literally) from any device. If you get a “can’t connect to server” error then nobody else will be able to either. It should connect and give an invalid response (to a web browser) basically.
“Can’t connect to server” is very bad, “invalid HTTP response” is very good because it means it connected successfully but couldn’t understand what the server said back to us but the connection is working otherwise. If you can connect like this (by public IP) then they can too.
You should be able to access it from your phone (with WiFi turned off to force cellular and test from an external connection), or anything with an internet connection basically. That’s how you can be sure it’s all working!
Yes, now it all works fine.
It took about 16 hours to get rid of relayed status. Now it is fully online.
Pity that someone else just joined my area and that the scale decreased to 0.83 🙁
Fantastic news! I think I honestly dare say that you did it and have joined a very small elite club of wireless (5G etc.) users with open NAT and working port forwarding. This is truly rare and way less than 1% of devices!
Thanks for leaving the great and detailed follow-ups. I think this conversation will help others as well who may be facing this. Take care!
Harshit
2 years ago
How to know what percentage has it sysnced ? For me it is just showing pie chart with no numbers in it. It is about 40 hours now. Can you help me out…
You can use the “Blockchain Height” to find out what block the network is on and the “Miner” button will tell you the block your miner is actually on! The Diagnoser is found at your miner’s IP address (if it’s a Bobcat) but my guide will explain it if the explorer is taking forever to update!
The miner height and blockchain height are the ones you want to look at. The smaller number in front of the combined “height” variable is the epoch and the second # is the specific block so out of those two numbers you posted you want the second but if you scroll further down it breaks them up individually anyway.
You can see my “miner_height” is about 5 blocks within the blockchain height (“blockchain_height”) which is considered close enough to be “synced”. A miner that is still syncing may be thousands or tens of thousands or more behind but the number should be increasing over time if it’s syncing properly.
You can divide the smaller number from the bigger one and then multiply it by 100 to get the percentage. The example for mine:
or about 96.1% which looks like it’s making progress at the very least. Now you just need to monitor it and make sure that is increasing over time and your sync % is increasing. Hopefully that helps!
chris
2 years ago
Apart from the issue of synch , there seem to be wider issues around „relayed status“….. My Nebra miner has been running unstable for a number of days now and the firmware update on 25.08.21 seems to have made this worse. The miner just spends 5 or 6 hours doing nothing and then wakes up for a few hours of activity. It is showing fast flashing green external LED too.
The hot spot is using UPnP to connect over the internet as evidenced by the router so manual port forwarding shouldn´t be an issue…
NOW comes the interesting (annoying!) thing – checking on other hotspots around me in Berlin shows many nearby hotspots that are „Relayed“, however all of them seem to be showing far more activity and 100% to 200% more earnings…..
I’m wondering if these hotspots were unable to update the firmware and are thus working as if surrounding hotspots( that HAVE updated) are not there?…… It certainly looks strange.
It’s definitely firmware related a lot of the time. My Bobcat was stuck for almost 2 days and all the sudden started working and beaconing again normally today and just sent it’s second beacon. It has happened a bunch of times already for sure on Bobcat where they all get stuck/broken until a OTA firmware update is pushed out. I just checked and the Bobcat firmware was for sure updated/fixed yesterday:
Issue Resolved
The OTA will revive miners with a red/yellow light whose diagnoser showed 'docker container not found'. They will start syncing properly again once the update is received.
Mine was definitely completely stuck for almost 2 days with no activity until this fix.
There’s even been a few times where it has been *all* Helium miners network wide until they all received a firmware update. It looks like Nebra did deploy firmware this past week.
When you say “a number of days now” that sure puts it in the range of when they would have pushed out this firmware update. Maybe it’s a flaky update this time but it seems like they’re pushed out every couple of weeks at the most so hopefully they’ll clean it up quickly!
I do have a Nebra outdoor I ordered forever ago (#243XX), but it was April 12th at 1:58am and I think the cutoff for orders was literally hours before that to even optimistically get it by September (AND it sounds like they have had some additional production delays from their recent email). I ordered my Bobcat on or around the same day and have had it for months!
With the Bobcat there is almost *nothing* you can do if it’s acting up except wait for them to fix it (at least until recently, it sounds like they added some more options to the Diagnoser within the past couple of days that may let you do some basics, I’ll have to investigate and update that article with the new options). I’m not positive if there’s much you can do with a Nebra since I’m still waiting on mine!
Nebra sounds like they are working on their own version of the Bobcat Diagnoser tool which is basically just a simple web interface to run some basic tests/fixes on the miner (the replies to this tweet are BRUTAL, people are not happy about the shipping situation). Unless you are able to get into the Raspberry Pi that powers the Nebra and have a lot more diagnostic tools at your disposal than the Bobcat permits it sounds like something you may have to wait for Nebra to take care of as well but I’m not positive.
From my experience it has been a constant struggle for the Helium team as well as the individual miner teams to keep the network running since I received my Bobcat and it always feels like it’s lagging and on the verge of crashing/breaking down but usually these types of things are fixed pretty quickly so far!
Hi James, as always, I appreciate your professional / thoughtful replies. I agree with your assessment about the “gossip part being very slow around letting miners know who’s online for witnesses / PoC especially when so many miners are being added daily”, but I thought that was why Helium started the VALIDATOR program. Validator nodes are a new entity on the Helium Blockchain that perform consensus group work, including verifying transactions and adding new blocks to the blockchain. It seems to me, it’s gotten worse even after adding VALIDATORS.
I also can’t help but wonder how flawed Helium’s algorithms are when I see three of the exact same Bobcat Miners at the exact same height, each are the only ones in their respective polygon but all connected (3 polygons in a row), one miner makes 27 HNT a month, while the other two miners make 7 HNT & 13 HNT. It makes no sense, especially when the are between 1/2 mile to 1.5 miles a part.
I watch the Reddit forum and get alerts when new comments are posted and no one has a good answer as to why such a big discrepancy in earnings exist. I’m venting I know, but I believe HELIUM needs to start owning the real issues that exist in the design of their blockchain and I hope the more people that read your blog and posts like mine start holding HELIUM’s proverbial feet-to-the-fire. I’m done ranting 🙂
Hey Matt,
That’s a fair question to ask why the validators didn’t impact this. Before they added the validators we were actually having chain halts all the time (some of which broke every hotspot on the network until there was a firmware update!). It actually used to be worse believe it or not! The consensus group would get hung and delayed all the time breaking all transactions on the network and you’d just have to wait and if it wouldn’t fix itself they would have to do emergency patches.
That kind of stuff really hasn’t been happening since validators came online so they did fix that piece of it but it’s still not perfect and some parts of the system benefitted more than others while others probably need additional tuning / an overhaul to allow it to continue to scale. It seems like something is still always broken but the validators did at least fix some things that used to be a constant headache that some people with newer hotspots may not even know were possible / used to happen.
Gossip actually seems slower to me than before validators but to be honest with you it’s hard to tell if that is because the explorer is really really slow to update (it was purely explorer 1.0 back then which seemed to work and update differently internally) or if the gossip is actually taking that long to traverse the network. It may actually just be the explorer issue since the “gossip” traffic is usually just about your status and the explorer is generally your only way to see that. I’m guessing there are other ways / alternative APIs out there potentially and it would be interesting to see how much faster they update than the explorer to see how much of the delay is the explorer and how much is gossip travelling the network.
The Bobcat mining sync issues are actually their own thing as well and reasonable criticism. Bobcat has actually admitted to this and said the recent OTA should help (but I would expect this to continue to be an issue, like an extra issue on Bobcats that is happening a lot more than other hotspots).
I believe this is why you’re seeing such a difference in earnings as well. My Bobcat has *really* been struggling the past few days and hasn’t done anything for 24 hours. It’s about 3000-4000 blocks behind and keeps going yellow and red off and on so even with this update I’m still personally having Bobcat related firmware issues.
None of this is meant to be a defense of the Helium team as that is not my job. I just wanted to explain that you are having an extra bad run of luck here and are getting hit by like most of the major issues going on on the entire network at once at once (as well as myself for some of them) so your frustration is very understandable. I really am still looking forward for my Nebra outdoor point to arrive so I can compare it to the Bobcat’s operation (and I’m nearly positive it will work more reliably).
The good news is the Bobcat diagnoser actually is getting really nice and I can barely keep up with the new features they’re adding (it looks like the Net Speed test is now back again since I blinked). This is a good thing because we used to just look at it and wonder why the light was red (literally, that is about all you could do).
Excellent rant, I applaud it and just wanted to add in some nuggets of knowledge here and there!
Wow, awesome response once again. I must say, and please don’t hate me, I’m glad to hear other people are experiencing what I am. I’m a little technical and I was getting frustrated why I couldn’t pinpoint the root cause of my problems and why I was the poor unlucky soul this was happening too. Once again, I agree this is both a Hardware and a Software problem, I just wish both Helium & Bobcat had the support teams to help calm and navigate the storms. They haven’t perfected their LoRA network and their ready to launch 🚀 5G 😖.
BTW, I have my Bobcat Minor 24 feet high outside on my roof with an 8 dBi Omni Outdoor 915 MHz Antenna for LoRa HaLow / Helium and a 16 ft. Low Loss Antenna extension coaxial cable from Rockland. I can only witness 1 miner that is 1.25 miles from me and have been unable to reach the other 3 miners that are 3 miles from me. There are lots of trees 🌳 🌲 in between us so I’m hoping once the foliage is gone, maybe I will be able to reach this other miners but I’m not holding my breath.
On the bright side, I’m excited to get my Linkdot miners in December if all supply chain components work out.
Thanks Again for all your insight and let’s hope I can pay-off my Bobcat miner by December before I get my 3 new Linkdot’s.
Hey Matt,
No worries at all! I actually completely agree with the 5G take. I’m not a fan at all of this launch. All they had was that FreedomFi point and you had to pay for a waiting list reservation (which has been long gone for months).
To that I have a pretty hostile and tacit response: the Helium team and the FreedomFi teams can both suck my balls. I’m not interested in your crappy partner access point and will wait for a real company to launch a 5G point that is offered to everyone. Until then the 5G network can and will be trash and none of us will care about it and the coverage will be HORRIBLE. How many of those did they even make for those paid waitlist spots? Honestly disgusting and anticompetitive behavior for crypto, if you’re going to launch it then launch it fairly and across the board or it’s going to be trash.
If someone else makes a 5G network to compete with Helium I would not be surprised if someone gets the leg up on them. Helium might think they’re in a prime position to dominate the wireless/LoRa space forever but I think they’re underestimating just how easy it is to switch and how little loyalty anyone really has to Helium itself. We are all mining and getting in on the new wireless mining technology and I do not care what the currency calls itself underlying that or if I end up mining a different one for 5G than HNT. That’s what exchanges are for so I can turn it into what I do want.
I would not lose any sleep if I set up a different 5G network point at my house than a Helium one if there are good competitors available! I honestly would probably prefer it because I’m tired of technology always having one/two companies dominate entire fields. The second they do they start abusing it, and Helium is. I got my Bobcat a few months ago. It would literally be nothing to switch away from Helium entirely or choose a different one for 5G. The Helium team has never wrote me a check or gave me anything special to deserve me desperately clinging to their single-partner paid-waitlist preorders. There already are other LoRa crypto networks and there will be other 5G networks if there aren’t already and they will grow every time Helium does something anticompetitive.
I’m mining no less than 10 cryptos at home at this very moment and it would take me a minute to think of what they even all are! I hope the Helium team is actually planning a proper and fair launch for the rest of the manufacturers but if they don’t someone else will, they always do in crypto just like the Helium team themselves were once trailblazers and then once getting on top slowly became the villain. This is an extra harsh take (especially from me) but I feel like those teams really need some cold water poured on them right now. Their futures will be decided by their next steps here in a way I’m not sure they are quite fully conscious of.
James, you’re a gentleman and a scholar with the critical updates you provide. I also have a Bobcat Miner 300 going on 3 months now and while I’m trying to remain positive, I can’t help but get enraged and negative on both Helium’s horribly designed / executed LoRA network and hotspot miner equipment manufacturers like Bobcat. I unplugged my Bobcat miner to move it to a different location and it was still showing ONLINE in HELIUM’s app for over 28 hours. Then this week my Bobcat miner all of sudden fell into a vicious sync cycle for 96 hours.
It was because of your blog that I learned of their utility tool and noticed they recently pushed a new update for FAST SYNC (alpha). I launched it and in 12 hours the utility now tells me I’m fully sun chef. But guess what, the Helium App is showing I’ve been synching for 20 hours when in fact I’m fully synced. Helium takes no responsibility for their crappy LoRA network that’s put together with bandaids & duct tape and can’t even provide near real-time updates on how your miner is performing. I have ZERO FAITH & TRUST in HELIUM especially when they rely on folks like yourself and the Reddit community to make sense out of the senseless.
I’m so frustrated because I never see an engineer from Helium post a YouTube video, provide commentary on their Reddit forum etc. What are your thoughts on this subject?
Hey Matt,
I completely share your frustration with how slow the explorer is at updating. It’s the “gossip” part that is really slow which is how your miner tells other miners it is online as well as whether it is relayed or not. It took someone’s 16 hours to update recently!
My main thoughts about this subject are that you are already on the right track starting to rely on the Diagnoser and other tools instead of the explorer. The new diagnoser gives you up to the second information about what the miner is *really* doing. It will tell you whether it’s online and exactly what block it is on. This is *WORLDS* more useful information than the explorer gives.
I’ll go even a little bit further though. The explorer is basically so slow it is useless (the only thing that shows up relatively quickly are basically transactions) and I would largely ignore it. Remember, the good news is that the explorer doesn’t actually *DO* anything. It doesn’t decide when you send a beacon. It doesn’t decide if you witnessed anyone. Nothing “depends” on the explorer to decide whether you earn HNT or not. Anyone could make their own HNT explorer just like lots of blockchains have multiple explorers (there probably are alternative HNT ones as well I just haven’t seen yet). It’s purely cosmetic / informational. You’re right that it’s near-useless for status updates because it’s so far behind it literally tells you nothing even if the miner has been off for 10+ hours!
The “fast sync” option is pretty interesting. At first I thought this was the same as just syncing from the “snapshot” that I’ve discussed here earlier but this does look like something different. The full feature breakdown is here. I’ll get the guide updated in the near future here but this does look worth trying if you are behind on sync (it says it won’t work unless you’re behind a minimum of 400 blocks).
I don’t have any silver bullets other than to tell you I share your frustration and can only say I recommend using just about anything else like you’ve been learning to do recently and continue on that path! Maybe they’ll improve the explorer later and it will update faster but for sure in the mean time I think you’re on the right track!
As a part of the broader Helium community, I would like to thank you for the knowledge you share and the effort you put in helping others. People like you make all the difference.
I have a question myself. I have 2 sites that I can not do port forwarding for various reasons. I’m looking for a way around this. I was thinking of a small box with a VPN service that allows port forwarding. I reside in Europe. Do you have any idea of a hardware/software combination that could help me resolve this? Do you know any VPN provider in Europe that offers VPN with port forwarding capabilities?
As a side question, the way I’ve understood it, the lag that the VPN will add will hurt me in terms of Helium returns. If I have understood it correctly having lower lag is better that having higher lag. Is this correct?
Hey Martin,
Thanks for the kind and gracious words, it is very much appreciated! I actually have heard of such a thing being used for my Minecraft server management project. Here’s a link to the thread. He’s also from Europe (Germany) as a bonus!
Theoretically you could run the ngrok daemon on one of the computers on your local network and have the daemon forward to your Bobcat’s address on 44158. The daemon runs on both Windows and Linux and you can see some more about it here: ngrok.com
They upgraded to a “premium” option and I believe you would too in order for it to be able to pass enough traffic to not impact your mining negatively. I’ve never heard of anyone doing this for Helium mining but is it possible? Absolutely it’s possible for the same reason this person was able to host a Minecraft server people could reach through a public IP from the outside. It’s doing exactly what you want it to do in this case, just for a different use!
I think this would work theoretically but I don’t know of a service that doesn’t need to run some sort of daemon on your local network so you would need to put the daemon on a computer that is up all the time for sure to be able to do this (and probably upgrade to a paid option). Since it does have the free option though it wouldn’t hurt to give it a try on the free option (even though it would slow down traffic you could confirm it 100% works before upgrading to a paid tier). That is an idea to investigate at least!
So the lag part of your question is really interesting and I’m glad you asked it in this way so I could get a chance to explain. There’s a couple of things to consider here. ngrok seems to be popular in Europe and have fast connections there but if you chose an alternative connection you would probably want to connect to it on your computer and see what your “ping” is.
If your ping is only going up by let’s say ideally 10-50ms but not a lot more than <100-150ms (bump up the numbers if you're in a remote location) then the benefits of being out of relayed will far outweigh the ping increase. It's not inconsequential though, you are right that it can negatively impact things (think of borderline cases where you barely miss the timeout but if your ping was 200ms less you would have made it).
The benefit of getting out of relayed status is that you will have so many more connections to the network (both inbound and outbound when you have a "clean" status) that your hotspot actually has a much easier time getting messages through to the rest of the network when some parts are busy/choked off. Basically you have more links/pipes to a larger and more diverse set of points and can navigate busy/congested times much easier than relayed points which have a limited number of outbound links only.
Why is this? Because a lot of the lag on the Helium network is because things are backed up (sometimes backed up for seconds or even minutes when it's really bad) so all these additional "pipes" give it alternative paths to still get through to the rest of the network when things are really busy. Basically there's different types of lag, and the ping time (latency) is only one type of them and a lot of the timeouts/problems on the Helium network are severe congestion measured in minutes at some points and not in milliseconds!
So that's the main tradeoff. The sweet spot will be to find one that has a minimal ping impact in Europe (the lower the better) but getting out of relayed status also dramatically improves your network link quality (and will often far outweigh even if you had the slightly or even moderately lower latency in some cases). Hopefully that helps, definitely let us know what happens if you try it because this is hot off the presses and not been done before that I know of!
Oops! Err, actually I did just think of one problem. I'm not sure how you would get the Helium miner to advertise your ngrok IP. It would have to be a service that is somehow controlled through some kind of web interface I would think or possibly a connection-wide or router-wide integrated VPN services. There might be a way to overcome this I'm not aware of / not thinking of in the moment but the main issue is the miner would be advertising your normal internet connection's IP and would not know to advertise a different one.
Which does give me one other idea. You could set up a VPN connection on a router with one of the VPN services that supports port forwarding. There are such VPN services that support both being set up connection-wide in a router AND support port forwarding like here. That’s not a specific endorsement of this service as I’ve never used it but is merely an example of a provider offering router-level integration and port forwarding services at the same time.
Here’s some other candidates that support port forwarding (the site claims most don’t, and you’d just want to make sure you can use router-level integration with them and it will be fast in your location).
Even if you just bought a router just to connect to Helium mining on a router-wide VPN that only the miner is connected to (using a VPN service that supports port forwarding on all ports, a lot of them are restricted to specific ports so double check this) that is my next best idea to get it working. This solves the problem ngrok would have because the router would actually give your miner your VPN public IP address so your miner would actually be advertising the VPN IP that your VPN-integrated/VPN-enabled router assigned it (as the WAN IP).
In the context of a Helium miner buying some cheap $40-50ish router that supports VPN connections (a lot of the providers that support this have specific models they even recommend) this might actually be worth getting out of relayed status (from a financial/investment perspective even, you would gain enough from being out of relayed status in additional HNT to offset the cost in some x period of time).
I can’t recommend any specific model/provider unfortunately as this is all highly theoretical based on my experience from others getting around NAT/firewall limitations in creative ways but if anyone reading has used good VPN services that support port forwarding / router integration and have low latency let us know. I’ve never heard of anyone doing this before and anyone who has isn’t talking about it and probably isn’t happy that I just said it was possible.
My personal belief though is that having the network not be filled with dead-weight restricted-networking points is better for everyone including the network/myself/everyone else and hoarding secrets/knowledge is not the way and harms the network and my personal stake because the network sucks to use so nobody wants to hurting everyone’s investment value (happens all the time in crypto). That’s my best fallback idea to get around this at least!
Not sure why you think relay affects sync. Unrelated.
I documented it in the article:
You only have 4 connections to the network (all outbound, they have to be) when you are in relayed status and you have 60-80+ outside of relayed status. Those 4 outbound connections are all you have to work with to sync in relayed status and nothing more. You can also monitor it with enterprise gear and watch the data rates / sync speed to verify this. It’s extremely obvious if you even open an enterprise connection monitor status page when there’s 4 connections vs 100. Even a lot of consumer networking equipment has a basic connection monitor built in and you can literally see it with your own eyes in real time.
This isn’t some unknowable thing or my opinion. We don’t have to guess whether it does or not and I wasn’t. It’s really easy to test and I suggest you do so if you are running/managing points with this assumption so you can see for yourself as soon as possible.
Look at Helium’s own recommendations and explanation. Your hotspot can barely even connect to the internet in relayed status. Look at the diagram. How could you think this doesn’t impact your sync speed? Do you have some mistaken impression that only radio traffic is affected and sync traffic can magically make it through your relayed status when that literally means you can’t accept inbound connections of *any* kind? Which circles are you running in that are gossiping this nonsense that literally make no sense due to how networks work? You’re dead wrong.
I’m definitely not above reproach but this was tested/documented and it didn’t take a genius to figure it out (4 connections to the Helium network vs 80, hmmm…) but it’s difficult to accept this criticism when the conclusion was tested/measured by me personally but I do not feel your criticism is coming from a similar place. It seems like something you heard or mistakenly misunderstood to me and that you have no actual reason to believe it.
Are 4 outbound connections to the Helium network faster or the same at syncing vs. 80+ connections of mixed inbound/outbound? No, of course not. That’s why relayed status is a thing. It’s bad (for everything) especially the network and syncing, and not just for you. You’re making everyone else’s experience worse as well staying in relayed because now we have to try to carry your point’s dead weight for you and most of it will time out before we get to it anyway (especially beacons and witnessing which have very strict timeout windows, meaning you lose most your income being in relayed status except for during the least busy and most ideal periods on the network).
Again, this is why you are penalized for it and people with a clean status are richly rewarded for it. This is in both the network and the (properly configured) miner’s best interest and if anything I would expect relayed points to get punished even more in the future as the network continues to grow and the relayed points continue to hold it back / keep it from scaling the way it should with additional points being added (it doesn’t if they’re relayed points, they’re just dead weight).
If you can provide me with a single shred of evidence of what you just said (which flies against the documented data as well as the recommendations from Helium / Bobcat / probably all manufacturers) I will absolutely consider it and test/validate it. You’re also free to validate it yourself and not reply to this and go back to wherever you heard this from and start setting the record straight!
Hi James,
This is by far the place with the best information about helium miners. Thanks!
I read all questions and replies. Very usefull, but i have not found the answer to solve my issue.
I have a Bobcat 300 and I use a D-Link DWR921 4G LTE router. I use a utp cable directly from this router to the Bobcat miner.
The Bobcat is synced, but relayed.
And yes, port 44158 is responsible for this problem.
I did everything to open it, but the port test websites shows that it is still closed.
I tried to open the port conform below steps:
Navigate through the router’s admin panel and look for “DHCP Reservations”, Click on DHCP Reservations and assign a static local ip address for your gaming console or end-device and make a note a note of the ip address.
Then i went to next step and add the bobcat miner at Virtual Server. I could select The Bobcat device in the dropdown menu and I added the static ip that i made in previous step. I added the port to forward (44158) and select the enable checkbox and saved it.
This is all what is mentioned online that you need to do for this router to open a port, but the port is still closed.
Do you have an idea how to solve it?
Thanks and best regards,
Tim
Hey Tim,
Thanks for the kind words! Your configuration actually looks 100% correct. I think you may be having a different problem.
A lot of 4G / 5G internet providers don’t give you a real public IP address (it’s NAT). Verizon is one of them. AT&T also doesn’t give you real public IPs by default and hasn’t for years according to this.
It sounds like these networks are behind NAT and I think that is the problem. Are you able to open any ports no matter what port it is and have them show as opened from the outside? My guess would be no unless it’s one that your provider is “allowing” through. You apparently aren’t even able to ping them from the outside supposedly if your line is under these restrictions.
It seems like all of these companies offer real IPs but they charge extra for them. That AT&T link even made it sound like you have to get a business line to get a real public IP that would allow port forwarding. Verizon sounds like it’s a one time $500 fee and also considered business class service ($$$) from this document.
Having a static IP is not the same as having a real public facing IP (you can have a dynamic public IP address which is what CenturyLink gives me against my will) but it is treated mostly the same by a lot of these wireless providers from what I can tell and they seem to have gone with straight NAT other than dynamic public IPs.
It doesn’t really seem like there’s a dynamic public IP option, it’s either NAT (can’t port forward) or a business class static IP but it may depend on your provider for sure. From what I can see the major providers here in the US have you by the balls as every single one I checked doesn’t give you a public IPV4 address that would allow port forwarding. Hooray for fake late-stage crony capitalism where the objective is essentially the same as the Monopoly game you played as a child!
Accessing ancient 1960s/1970s public IP address technology has become quite expensive/impossible (at least in the US, but international commenters have told me it’s not much easier (or far worse) to get public IPs or static IPs there in most other places too seemingly) thanks to these market conditions (anti-trust hasn’t been enforced for decades in nearly any industry, most of them only have 2-3 companies left, it’s a runaway train and the train went bye bye long ago).
In this brave new world if the 2-3 companies left decide you don’t need a public IP address anymore as a consumer/residential customer then that is immediately the way it is across the land (and often times the world in the really bad industries like cell phone operating systems i.e. Android/iOS). Who are you going to switch to?
Which provider do you have? Can you try testing if you can open any ports from the outside (other than 44158)? It seems like you’re behind your provider’s NAT (willingly or unwillingly) potentially here!
Hi,
I think we are on the right way.
Tried to open 2 other port, without any positive result.
I found this information from my provider Orange.
It is translated to English:
Do you like to play online games and use mobile internet? Meet the Orange VPN service
Orange VPN – is a service that allows you to access the Internet with a dynamic public IP address. It allows, among other things: to use mobile internet and to connect to selected services, for example XBOX Live, Playstation Network or other game services.
Orange VPN Static provides internet access with a public IP address. The service allows the customer who uses mobile Internet as a home to use multimedia functions such as smart home, connection to web cameras, online game servers, etc.
Orange VPN is a special service:
• Players using mobile internet
• IoT Internet of Things
• Smart home solutions
• Mobile video surveillance
Orange VPN is a service that allows you to access the internet with a public IP address (variable or permanent) from a device equipped with a SIM card, giving you remote access to this device.
However, remember to protect access to devices in your home with a strong password
.
The service is available in two variants:
1. With dynamic IP address in combination with APN “vpn” – VPN
2. 2. With a fixed IP address as part of connections to APN “vpn.static.pl”
So i need option 2?
BTW, option 1 cost 1.85 usd per month and option 2 cost 5.27 usd per month aprox.
Greetings from Europe!
Hey Tim,
That’s actually fantastic news in this case that you aren’t in the US. That is exactly what you need to get! The Xbox Live / PSN part is the dead giveaway. Basically those services will give you a “poor” connection if you don’t have a public IP address that can allow port forwarding (usually through UPnP or Universal Plug and Play but the ports can be manually opened/forwarded). This is because your console can be the “host” or “server” since many/most console games are peer to peer networking. If you can’t forward ports you can’t be the host so you have a “degraded” connection in that case and both services will complain about it. Even though we don’t care about our Xbox / PSN connection in this case it’s actually a really good sign on a technical level that the service is going to do what you want it to do.
This likely caused them to receive enough complaints to start offering this service as an extra since there are situations like that where you will have a “degraded” experience without a public IP. This is for the exact same reason you can’t forward to your miner. It can’t be done in this type of a network and they need to actually assign you a public IP for other members of the public to be able to connect to you specifically from the outside. You can’t “share” an IP with 100 other people which is what they are doing on most of these 4G/LTE networks. IPv4 addresses are limited and have been for a while which is why they do this. Every ISP has a limited block of these addresses so they’re charging you a fee to take up one of these limited reservations. The fee is lower if you’ll take whatever is available (dynamic) and much higher if you always want the same IP (fixed/static).
To understand why you can’t share one think about what would happen if a single other one of those people were running a Helium miner. How would it know which one to send it to when someone connects to that shared public IP? That’s the fundamental underlying problem they are solving by assigning you a public IP (dynamic or fixed, doesn’t matter). You just need one IP that goes to you and that is how people will connect and talk to your miner. Right now you’re sharing and there is no single IP that actually goes to *just* you in the current setup.
I personally have a dynamic public IP and so do probably most people mining and it will not put you in relayed status. My Bobcat has paid for itself several times over at this point for sure so if it makes a difference it would be marginal (unlike getting out of relayed status which is well known to make a large difference as they pretty much “punish” you for it as everything important will time out before getting relayed >50% of the time). You shouldn’t need to spend 3x-4x as much to get a fixed or static IP. The $1.85 option should work just fine (the dynamic public IP option) and give you a dynamic public IP that can be forwarded!
Ok, i will try the option 1 for 1.85 with dynamic public ip.
Other setting like i already did before should i keep it the same?
Hey Tim,
I would definitely try it as-is first. It looks like their instructions gave you a new APN to possibly put in the hotspot settings but maybe that is done for you automatically, but I would go ahead and try it!
It takes several hours at least for the “relayed” status to drop off the explorer because the explorer updates very slowly but I would leave everything else the same and give it a try using that service!
Hi James,
Yesterday i changed settings in the router and installed static public ip.
Port tested via website and it shows that 44158 is open. So it works, so you were right so far, but…….now, 12 hours later, the miner is still in relayed modus.
Pfff…..i know exactly why i am not work in IT business, it drives me crazy, haha.
Do you have any idea what can went wrong?
Best regards,
Tim
And it started syncing again now….6 of 7 steps are finished, only syncing started again.
That’s excellent you got the static lease set up! The explorer taking this long to update doesn’t sound very unusual honestly. My hotspot is fully sync’d and it hasn’t updated the explorer status in 6 hours. The explorer is apparently only updated by your hotspot “gossiping” on the network.
If you aren’t fully sync’d I don’t think it will update the explorer yet. If you’re getting an open port status then you are just waiting for the explorer to update basically which can take quite a while for sure. If it’s showing “open” it’s receiving connections and it’s only a matter of time before your hotspot’s “gossip” spreads around the network.
You can unplug the hotspot and plug it back in to try to get it to “gossip” with some different points on the network (the idea being it will not connect to the same outbound hotspots as before you rebooted it, it picks a new set of random hotspots) which can sometimes speed things up (you’re spreading the “gossip” far and wide by reconnecting to a whole new set of hosts and gossiping to them too) but there’s really no way to make the Helium explorer work faster / do much other than wait unfortunately.
You can see just about everyone that has ever had to wait for relayed status go away go through this here. The people saying they’ve been waiting 3 days definitely still have a misconfiguration but I don’t expect it to take much longer for yours and if it takes longer than I would say even 24 hours (it seems like it’s more like 12-16 when it’s cranky) then yeah something else is still definitely wrong.
The relayed status in explorer means nothing as far as to whether you are penalized. If you can accept inbound connections the effect is basically immediate (once you get sync’d at least) no matter how long the explorer takes to update. I’m guessing the bigger problem is it got behind on sync which seems to happen all the time on the Bobcat if you get a bad few connections for a while that aren’t sending blocks quickly.
It will happen less once you’re fully sync’d and out of relayed status once and for all because you’ll have way more connections to sync with (both inbound and outbound) which makes you far less susceptible to falling behind on sync for sure.
You can literally type your Bobcat’s static public IP in your web browser (on *any* device) like
httx://X.X.X.X:44158(mine at time of writing and for the next little bit washttx://174.23.155.49:44158/, any of you can go to that within the next day or so before my dynamic IP updates and see the same thing) and you should see this:This page isn’t working174.23.155.49 sent an invalid response.
ERR_INVALID_HTTP_RESPONSE
That is exactly what you should see because it’s not actually a web server on the other end so your browser can’t actually talk to it but if you get the page isn’t working invalid response then you know the firewall is letting you through and the port forward is working. If you see this then it’s only a matter of time before the explorer catches up with it.
It’s incredibly slow to update the explorer web frontend but if you’re able to reach 44158 from the outside with your web test it will update probably within the next 6-12 hours would be my guess but it couldn’t be more normal for it to have taken this long to update (it would be more surprising if it dropped off already)!
Thanks again.
The syncing issue was just an app error, because my other miner showed also syncing status at that moment, while this one is at another location and i havent dont any changes to that miner. Anyway, the syncing status changed into online again.
I did the test with ip and i get the same message that the page isnt working etc. So that sounds good.
It takes about 13 hours now, still relayed. So lets wait for next 12 hours.
But as i understand it correctly, now it does not matter that it shows relayed, since i have inbound connection? How do i know for sure that that is the case and that i get full rewards and no penalty’s
Best regards,
Tim
Hey Tim,
The way you be sure is to do the test of going to your public IP in your browser at
httx://X.X.X.X:44158(your public IP as in the new static IP you got, not a local IP like 192.168.X.X, the one people will reach you from the internet with) and make sure you see:This page isn’t working174.23.155.49 sent an invalid response.
ERR_INVALID_HTTP_RESPONSE
Everyone in the world should be able to access that port (literally) from any device. If you get a “can’t connect to server” error then nobody else will be able to either. It should connect and give an invalid response (to a web browser) basically.
“Can’t connect to server” is very bad, “invalid HTTP response” is very good because it means it connected successfully but couldn’t understand what the server said back to us but the connection is working otherwise. If you can connect like this (by public IP) then they can too.
You should be able to access it from your phone (with WiFi turned off to force cellular and test from an external connection), or anything with an internet connection basically. That’s how you can be sure it’s all working!
Yes, now it all works fine.
It took about 16 hours to get rid of relayed status. Now it is fully online.
Pity that someone else just joined my area and that the scale decreased to 0.83 🙁
Thanks again for your advise. You solved it 🙂
B.r.
Tim
Hey Tim,
Fantastic news! I think I honestly dare say that you did it and have joined a very small elite club of wireless (5G etc.) users with open NAT and working port forwarding. This is truly rare and way less than 1% of devices!
Thanks for leaving the great and detailed follow-ups. I think this conversation will help others as well who may be facing this. Take care!
How to know what percentage has it sysnced ? For me it is just showing pie chart with no numbers in it. It is about 40 hours now. Can you help me out…
Hey Harshit,
If it’s a Bobcat you can use the Bobcat Diagnoser tool.
You can use the “Blockchain Height” to find out what block the network is on and the “Miner” button will tell you the block your miner is actually on! The Diagnoser is found at your miner’s IP address (if it’s a Bobcat) but my guide will explain it if the explorer is taking forever to update!
Thank you for your feedback. You mean after finding out the Blockchain height, I should see the following under Miner Tab right?
“height”: [
“24468 955285”,
“”
But here, there are 2 numbers. Which one to consider?
Hey Harshit,
That’s right, you’re in the right place here. Here’s mine:
"timestamp": "2021-09-02 16:00:09 +0000 UTC","temp0": "52 °C",
"temp1": "52 °C",
"temp_alert": "normal",
"ota_version": "1.0.2.19",
"miner": {
"State": "running",
"Status": "Up 18 hours",
"Names": [
"/miner"
],
"Image": "quay.io/team-helium/miner:miner-arm64_2021.08.24.4_GA",
"Created": 1630532362
},
"height": [
"25617 993802",
""
],
"epoch": "25617",
"miner_height": "993802",
"blockchain_height": "993807",
The miner height and blockchain height are the ones you want to look at. The smaller number in front of the combined “height” variable is the epoch and the second # is the specific block so out of those two numbers you posted you want the second but if you scroll further down it breaks them up individually anyway.
You can see my “miner_height” is about 5 blocks within the blockchain height (“blockchain_height”) which is considered close enough to be “synced”. A miner that is still syncing may be thousands or tens of thousands or more behind but the number should be increasing over time if it’s syncing properly.
You can divide the smaller number from the bigger one and then multiply it by 100 to get the percentage. The example for mine:
(minerheight / blockchainheight * 100)or
(993802 / 993807 * 100) = 99.99949688420387% syncedand for the numbers you posted (6 hours ago but we can still calculate from it):
(955285 / 993807 * 100) = 96.12379466033143% syncedor about 96.1% which looks like it’s making progress at the very least. Now you just need to monitor it and make sure that is increasing over time and your sync % is increasing. Hopefully that helps!
Apart from the issue of synch , there seem to be wider issues around „relayed status“….. My Nebra miner has been running unstable for a number of days now and the firmware update on 25.08.21 seems to have made this worse. The miner just spends 5 or 6 hours doing nothing and then wakes up for a few hours of activity. It is showing fast flashing green external LED too.
The hot spot is using UPnP to connect over the internet as evidenced by the router so manual port forwarding shouldn´t be an issue…
NOW comes the interesting (annoying!) thing – checking on other hotspots around me in Berlin shows many nearby hotspots that are „Relayed“, however all of them seem to be showing far more activity and 100% to 200% more earnings…..
I’m wondering if these hotspots were unable to update the firmware and are thus working as if surrounding hotspots( that HAVE updated) are not there?…… It certainly looks strange.
Hey chris,
It’s definitely firmware related a lot of the time. My Bobcat was stuck for almost 2 days and all the sudden started working and beaconing again normally today and just sent it’s second beacon. It has happened a bunch of times already for sure on Bobcat where they all get stuck/broken until a OTA firmware update is pushed out. I just checked and the Bobcat firmware was for sure updated/fixed yesterday:
Issue ResolvedThe OTA will revive miners with a red/yellow light whose diagnoser showed 'docker container not found'. They will start syncing properly again once the update is received.
Mine was definitely completely stuck for almost 2 days with no activity until this fix.
There’s even been a few times where it has been *all* Helium miners network wide until they all received a firmware update. It looks like Nebra did deploy firmware this past week.
When you say “a number of days now” that sure puts it in the range of when they would have pushed out this firmware update. Maybe it’s a flaky update this time but it seems like they’re pushed out every couple of weeks at the most so hopefully they’ll clean it up quickly!
I do have a Nebra outdoor I ordered forever ago (#243XX), but it was April 12th at 1:58am and I think the cutoff for orders was literally hours before that to even optimistically get it by September (AND it sounds like they have had some additional production delays from their recent email). I ordered my Bobcat on or around the same day and have had it for months!
With the Bobcat there is almost *nothing* you can do if it’s acting up except wait for them to fix it (at least until recently, it sounds like they added some more options to the Diagnoser within the past couple of days that may let you do some basics, I’ll have to investigate and update that article with the new options). I’m not positive if there’s much you can do with a Nebra since I’m still waiting on mine!
Nebra sounds like they are working on their own version of the Bobcat Diagnoser tool which is basically just a simple web interface to run some basic tests/fixes on the miner (the replies to this tweet are BRUTAL, people are not happy about the shipping situation). Unless you are able to get into the Raspberry Pi that powers the Nebra and have a lot more diagnostic tools at your disposal than the Bobcat permits it sounds like something you may have to wait for Nebra to take care of as well but I’m not positive.
From my experience it has been a constant struggle for the Helium team as well as the individual miner teams to keep the network running since I received my Bobcat and it always feels like it’s lagging and on the verge of crashing/breaking down but usually these types of things are fixed pretty quickly so far!