The Bobcat 300* Helium Miner finally has an interface you can use to be able to tell what it’s doing! Better yet, it lets you do a few things like conduct an internet speed test. You can even get a detailed breakdown of which block your miner is on!
In this guide I’ll break down how to access the web interface (easy) and run through some of the features that are available at launch (1.0). The utility started installing on Bobcat 300’s via OTA updates on August 3rd 2021. Let’s begin!
Hardware Used
Bobcat 300 Helium Miner
The Nebra outdoor miner is a Helium miner designed to be installed permanently outdoors. It is powered by a Raspberry Pi Compute Module 3 and uses a eMMC chip for storage via a
Hopefully you already know what the IP address of the Bobcat 300 is on the network as you will need to know that to get your miner out of relayed status and set up the “port forward”.
You can find your IP by logging into your router’s web interface / app or you can use the Helium app’s “Diagnostics” feature to get it as well!
Accessing Web Interface
Once you’ve determined your miner’s IP type it in your web browser like this:
http://192.168.X.X
It does not like https (port 443) and will only work with http (port 80).
Diagnoser Utility Options
Let’s cover what you can do using this utility at launch.
Temperature
Temperature is a serious obstacle for Bobcat 300 miners. If they are outside or in a very hot / poorly ventilated place they are well known to overheat. This feature helps you manage this and make sure your miner isn’t getting cooked.
If you choose the temperature option you will get the temperature of your unit (as temp.json in JSON format, useful if you are a developer) like this:
This lets you keep an eye on your miner’s temperature. This one is inside an air conditioned building and it’s still nearly 49C or roughly 120F degrees but would be considered a “good” temperature reading!
Sync Status (Update 9/3/2021)
New option added in 1.0.3 that is described as:
Introducing the sync status feature on the Bobcat Diagnoser, which answers the very simple yet important question of whether or not your Bobcat miner is properly synced to the network. Stop incessantly rebooting or waiting for the helium app to update the miner’s status. Now you’ll have access to real time information at the push of a button!
The “gap” is how many blocks behind you are, so I am 2 blocks behind with a gap of -2. The math adds up with the “blockchain_height” representing the height the entire network is at and “miner_height” representing the block height your miner is at / sync’d to.
Miner
The “Miner” choice gives you an extremely detailed breakdown of your miner. You want to be careful with this option because the menus warn you that this option slows down the Bobcat’s mining (it’s a resource intensive / high CPU operation).
We can actually see a lot of stuff that was previously not possible to see on your Bobcat like the enrollment key, various internal components and their serial numbers, etc.
Our P2P connection status (what the light represents on your Bobcat, green yellow etc.) is revealed in great detail as well. Look at this detailed breakdown here:
This is just a link to the Helium blockchain explorer. Not very useful!
Helium API
Connects to the Helium API and returns some information in JSON format about your specific miner. This is Helium-specific information that is tracked across all miners and not Bobcat specific.
Onboarding
This one is a little more interesting. It redirects you to your hotspot’s onboarding entry at onboarding.dewi.org. The results are given to you in JSON like the other commands.
I had not seen this site before but sure enough my onboarding entry appeared as expected!
Net Speed (Internet Speed Test) (Update 9/3/2021: Removed, Secret Link Available)
Finally a fun one! This one lets you test your Bobcat 300’s internet connection speed.
I ran the connection test (using WiFi) and this was my output:
Those results are actually terrible for my connection speed (gigabit fiber) but not unexpected for WiFi using G/N speeds.
At any rate this is very useful information to have whether you’re using wired / wireless. Having a tool like this lets you test the Bobcat’s connection specifically instead of running a general speed test on your computer. That only diagnoses your internet connection in general and not the Bobcat’s connection specifically so having this tool is a giant leap forward and worth using.
It does make your miner slow down / stop mining for about 30 seconds so don’t overdo it!
UPDATE 9/3/2021: In 1.0.3 this option is no longer in the list of options. It’s still available on the miner as speed.json. To access the option replace the IP in the URL below with your own miner’s IP address and go to:
http://192.168.X.X/speed.json
After a 20 second pause or so you will see the speed test (for now, they removed the menu option but the JSON request is still available/permitted):
This is the first really serious troubleshooting option we’ve been given for the Bobcat. It was added in 1.0.2 and the description given by hovering your mouse over the option is “It will delete the miner data, let the miner resync to the blockchain from scratch”.
Basically they’re saying this will wipe the local blockchain data from your miner and let it resync from scratch. You would only want to do this if your miner is completely stuck / corrupted.
Do you remember how long it took to sync your Bobcat when you got it? You definitely don’t want to have to go through that again unless your miner is completely broken and won’t sync and nothing else has worked / troubleshooting has led you to this conclusion.
This finally gives users the ability to clear and resync their Bobcat though which is a fantastic improvement from the launch 1.0.0 version!
Also requires a username / password:
Username: bobcat
Password: miner
Reboot (Update 9/2/2021)
This was a new option added in version 1.0.2. It’s as simple as it sounds and allows you to reboot the miner through the web GUI. Very handy!
This option will prompt you for a username and password. Here they are:
Username: bobcat
Password: miner
Blockchain Height
Blockchain height just gets the current height of the Blockchain from the official Helium API. This would be useful if you wanted to write a very simple script that wanted to compare if your hotspot’s block matched the current block on the blockchain.
A lot of the other available commands give the blockchain_height already (such as “Sync Status” and “Miner”) so it’s not extremely useful.
Bobcat OTA Update / Troubleshooting (Community Res)
Don’t get too excited about these ones. These are not at all what they sound like. They’re both just web links to community resources (at least at 1.0.0 release).
Fast Sync (Alpha) (Update 9/11/2021)
This new feature was added in 1.0.4. It’s meant to help catch you up quickly to the blockchain. I had the chance to use it today as Bobcat has had a lot of issues with hotspots falling out of sync and mine has been impacted.
The function takes about 30 minutes to sync and you should be at least 400 blocks behind to use it. From Bobcat:
Fast Sync
This is an alpha feature that gives healthy miners a sync boost, so they can quickly catch up to the top of the blockchain from the block they are syncing at.
IMPORTANT: this feature will NOT work for healthy miners whose gap to the blockchain is smaller than 400 blocks.
The whole time I’ve had my Bobcat it has been a “black box”. Who knows what it’s doing. You can’t log into it with SSH or using the serial console (set your speed to 1,500,000 baud (yes, 1.5 million) and the serial console “data port” does work over USB in something like Putty, this is some kind of security through obscurity baud rate) because all the accounts are either disabled or have a nologin shell.
There’s only a couple of useful “tools” in 1.0.0 release that actually make the Bobcat do anything but it’s so nice to be able to finally see and be able to do *SOMETHING*! I’d imagine that more menu options will continue to be added to the Bobcat 300’s new “Diagnoser” tool over time.
I definitely at least recommend running the “Speed Test” and checking your unit’s temperature to make sure everything is running well!
That’s a really good question. I really hope it doesn’t listen on the WAN IP since some of the options in there are to literally wipe the miner clean and sync from scratch (going to take a while). If it does then forwarding port 80 to the miner would let you access it as well as literally everyone else in the world. Make sure you understand that, if it lets you do this then make sure you turn it off after you’re done or remember that anyone has full access to your Bobcat miner!
My house has a VPN set up inside the router (lots of routers support this but it’s less common on consumer ones you’d find at Best Buy and more common on enterprise/business ones). What I mean exactly by this is that my router hosts it’s own VPN service basically as my public IP address and you connect on a VPN client to my public IP. Since it has that I can actually access anything on my local network from any device in the world that has an internet connection (including my phone) and that would be a good/secure way to set it up.
You could even use something like LogMeIn / TeamViewer or one of the gazillions of cloud screen sharing apps to do this. Chrome Remote Desktop as basically a part of your browser is a pretty accessible solution as well. If you install a cloud enabled one on your computer for example you can use your cloud remote desktop session and access any service you want including the Bobcat diagnoser from another computer, your phone through an app, etc.
Basically my recommendation would be to set up some kind of remote desktop for yourself (lots of free ones available for home users). You really don’t want to forward the Bobcat traffic to your local network through something like a port forward if they actually even do listen on the WAN IP for connections on port 80. Definitely the remote desktop setup (which can be free) is designed with security in mind for remote access and will let you do this better and so much more securely!
David bobcat
2 years ago
James, fantastic website/ guides. Super easy to use and understand, thank you for doing this. I am having an issue with my miner ever since the update 4 days ago (1.0.2.23S).
The miner had no issues and was synced/mining, then it went to a syncing status and got stuck at 99 percent. I went ahead and use the new feature, fastsync last night and it did clear up errors that were previously there.
This morning it’s still syncing and now says it needs attention. Below are the lastest logs, the only thing i noticed is that im behind on blocks. Any advice on a way forward? timestamp "2021-09-14 15:55:55 +0000 UTC"
temp0 "72 °C"
temp1 "71 °C"
temp_alert "alert"
ota_version "1.0.2.23S"
sync_status ""
miner
State "running"
Status "Up 10 hours"
Names
0 "/miner"
Image "quay.io/team-helium/miner:miner-arm64_2021.09.03.0_GA"
Created 1631590913
height
0 "25850 1001680"
1 ""
epoch "25850"
miner_height "1001680"
blockchain_height "1010832"
Thanks for the kind words! I think I can help. Yours actually looks a little bit different than some of the previous ones posted here. Your miner is blazing hot at the moment at 72 degrees C (161.6 F for SI challenged Americans like myself). You actually have a temperature alert/warning which means you’re at risk of lockups/crashes (or worse if it goes on for too long):
temp0 "72 °C"
temp1 "71 °C"
temp_alert "alert"
Is this miner outside by chance? You aren’t far behind enough on blocks to do anything with it. It’s only about ~150 blocks behind and it needs to be a minimum of 400 for the Bobcat to even let you do anything to try to make it sync faster.
Your issue is it’s most likely locking up/crashing due to high temperatures or might even be throttling to keep itself from completely overheating/crashing out. It might just have the airholes blocked/obstructed but the temps are too high and are literally in the redzone / warning zone. Lowering your temps should fix all of it including the syncing!
James, great observation and feedback. The miner is inside on the third floor/attic space. It does get pretty hot and I’ve been turning off the AC to save on power. I will crank the AC and see what happens.
Insightful about the blocks as well, I calculated that I was 9,152 blocks behind. I subtracted the blockchain height with the miner height to get that answer, am I doing that wrong? If so if you have any resource on how to determine how far back I am it would be truly appreciated. Also, hope you don’t mind, but I’ve shared your site with my friends on discord :).
That sounds right! The attic gets pretty hot. You shouldn’t need to get the temperature down too much. I believe the warning starts at 70C which is pretty standard for electronics (some will take 80-90 but 70C is when a lot of things are really going to start cooking).
You’re right on the math. I missed that one number started with 1010 and the other was 1001 which is how I came up with 152. You did the calculation correctly by subtracting the miner height from the blockchain height to get the difference so I wanted to clarify that this was my mistake on this one and you are correct on the blocks!
In that case you actually can use the “fast sync” feature to catch up once you get the miner a few degrees cooler (definitely a few ways you can do this, even just putting a fan up there blowing directly on the miner would probably do it even with the warmer air because it will pull enough heat off the miner to get it out of the redzone). Once it’s back in the 60s the warning should drop off and you’ll stop getting throttled. I’ve used the fast sync a couple of times since it came out and it worked for me both times after about 30 minutes.
Mine was really struggling a few days ago staying synced and fell very far behind. Finally Bobcat released some firmware updates that seem to have stabilized it and then I used fast sync one more time and after about 30 minutes it will have you caught up!
Thanks for the clarification on the math and owning the error! Also, you are brilliant. I fortunately can just put my miner in front of my AC and got the above results. The “needs attention” alert on the app went away so it’s a step in the right direction. I will now try the fast sync feature once more and report back on the result. I love when you troubleshoot something and actually get to a resolution, best feeling ever.
Oh wow, that turned out fantastic! I think you’re on the right track for sure here now. It makes sense that it was running at like 25-50% speed under throttling etc.
I look forward to hearing how it all turns out but I have a good feeling about it so far!
Here is the latest, it looks like my miner height timed out now, the bobcat miner is stuck on yellow. I gave it some time (a couple hours) and am wondering if I should unplug the miner and try the fast sync again. Any thoughts James before I try that?
Yeah your plan looks good here. When I ran the fast sync the Bobcat Diagnoser page will actually kind of change and some parts will turn yellow. It will also tell you the miner light will turn yellow.
Before you do that though I would log into the Bobcat Diagnoser and see what the exact status you have is. Take a look at what block it is on now. Did the fast sync do anything? It might be getting close or it might have made zero progress in which case yeah it’s time for a power cycle and starting it again. It’s an alpha feature so it’s definitely not at all inconceivable it could have got stuck and needs another try before it’s going to work.
If you check it though and it actually has made a lot of progress and is getting close I would let it finish. The fast sync will only take you to within about 400 blocks of current then you still have to sync the “last mile” yourself (this is actually why they don’t let you use it if you aren’t at least 400 blocks behind, it literally wouldn’t do any good because the snapshot only goes up to that point).
(Oops, I realized you posted your status after I posted this and it has an error status, yes definitely reboot this one!)
2 hours seems too long though. I would check the status and if it hasn’t made any progress give the process another start here!
Just stopping in to give an update, I went ahead and rebooted and used fast sync again and the same thing happened, I was stuck on the same block height as before, which was “1001680”.
I went ahead and used the full re-sync feature and things seem to be moving towards the right direction, kinda. I checked my block height and it did change to “994345”, then I checked about an hour later and it’s “994475” currently. It’s at the very least moving so I will be patient and check in a few hours.
This is an alpha feature that gives healthy miners a sync boost, so they can quickly catch up to the top of the blockchain from the block they are syncing at.
IMPORTANT: this feature will NOT work for healthy miners whose gap to the blockchain is smaller than 400 blocks.
Resync Miner
This action deletes the blockchain data. It does not, however, delete the helium software. Your miner will start syncing from 0.
Apparently if your miner is not “healthy” to begin with the fast sync may not work and you did exactly the right thing. I’m guessing by healthy they mean that your miner is really far behind but that the copy of the blockchain you have is intact / uncorrupted.
It’s very possible invalid data got recorded when it was that hot as weird things (“undefined behavior”) can start to happen (and much stranger than this even) under those temperatures. Now that the temperatures are under control a resync should result in 100% valid data.
Let us know as it continues to move in the right direction / catches up!
Interesting about the importance about the temperatures, but also you brought up a pretty interesting point. Since it’s early I may go ahead and give this a shot and just restart if things get messed up again but, I think I should use the fast sync option now since my miner is “healthy”. The gap is currently 16,000 so it would be a good benchmark. Thank you for the consultation, I love the bouncing of ideas. I am going to give this a shot and will report back here the results.
Glad I could help! I think you can go ahead and use it now safely too. It is not typical for it to get corrupted like that and it actually caught me a little off guard when I first read your message but then I thought a little bit more about it and yeah anything can happen when the temps get that high (especially corruption/normally impossible things). It may have been like this for some time before it started having symptoms enough to notice.
Definitely let us know how it works out if you do for sure!
Don’t appear to be lucky with the fast sync, climbing slowly though.. at this rate ill be done in 8 or 9 days lol "994345"
"994475"
"994577"
"994696"
"994720"
Yeah I agree that doesn’t look like the fast sync. My guess is it has to do with the notes saying it will only work for “otherwise healthy miners”. Since yours has never been sync’d fully yet since resetting maybe you don’t meet that criteria. I’m not sure because the features are only a week or two old and there’s not anything more to go on than what Bobcat tells us basically in their release notes that I constantly link to (this is where I get most of my info, straight from Bobcat’s release notes, I read every line). They’re also in “alpha” status (which is considered even earlier than “beta”) so it could have flat out been a bug that’s never supposed to happen for all we know.
It may just need to sync normally with this type of reset. At least it starts from a snapshot (and not from 0 even though the Bobcat diagnoser tells you it is going to start from 0 I have never seen it go 0, 1, 2, etc. syncing the chain and it always starts at some snapshot, the “blessed” snapshot for full syncs I believe). The sync speed varies too depending on who you are connected to etc. so definitely keep an eye on it but I would definitely let it finish since it doesn’t seem like we have much choice / much else we can do in this case! I don’t expect it to take 8-9 days fortunately as I think it should get picking up a little faster here over time!
This is definitely why I lean so heavily against telling people to do the full reset (it’s like getting it brand new and mine took 6 days to sync brand new but it doesn’t take as long these days because of improvements since I got my point) but I really don’t think you had much of a choice. Your status was completely crashed out into “error” and didn’t even have a blockchain status 2.5 hours after starting the fast sync. This can happen for sure and usually means the miner “container” was crashed out (this is what Bobcat refers to it as, it’s probably a docker container) but a power cycle usually gets it going. It didn’t in your case, which means something was legitimately wrong/broken on the disk/storage.
From Bobcat’s documentation we can assume this means your miner wasn’t “healthy”. Their documentation didn’t really mention though that if it’s not healthy it could completely crash out the miner where you have to fully reset it completely for it to get “healthy” again. Still, this looks like one of the rare appropriate cases to use the *full* reset because the miner was dead, dead, dead. Healthy in this context would refer to the files on your Bobcat’s local storage (namely the blockchain but also the Helium software).
How are you supposed to know if your miner is “healthy” or not? There’s currently no way of knowing other than your intuition if you think your miner is acting up and seems unhealthy to you. Mine acts up all the time (as in it will fall behind on sync or start turning yellow/red randomly) with the typical Bobcat firmware/falling behind on sync occasionally issues but I have never had to do a full reset or been prevented from using the fast sync feature (although maybe I should do a full reset, maybe that’s why mine acts up every week or two because it does need the full reset I’m trying to avoid).
We know these issues are widespread though because Bobcat admits to them in the patch notes and makes fixes to alleviate them *constantly*. They release a firmware update once or twice a week and have since I got my point and almost all of them have some sort of stability fix. You can see this for yourself as well here and everything that says “Bobcat Miner OTA” is a firmware update and if you click it you can see the notes. I’m guessing finding Bobcats that don’t occasionally stop syncing / go to yellow/red status for no apparent reason is quite rare.
Nothing in the status report tells us whether your copy of the blockchain / local files on the Bobcat are “healthy” or anything like that so hopefully Bobcat makes some improvements here to give us an idea of what kind of condition our miner’s disk and filesystem are in and whether those are “healthy”.
If it’s on WiFi right now it may sync faster on wired (my Bobcat’s wireless link even to APs 20-30 feet away is at best 300Mbit and usually closer to 100-150Mbit, fine for normal activity but syncing the whole chain it’s better to have more) but otherwise let’s let it finish and you should be good to go!
Hey James – You were right that it wouldn’t take 8-9 days, currently my gap is around 2272, “1010118” gap “2272”. Looks like things are going to be working here by this evening.
I wanted to ask, what should my height block be to be syncing? I think I read somewhere that it had to be at or above the blockchain height for it to be synced, is that accurate? A friend of mine just used the fast sync feature with success and he’s around 30 blocks behind, is that considered to be synced?
I’m so relieved that things did speed up, I actually felt bad that everything led to you having to do the full reset but I really 100% think it was for the best after doing some more research/learning today and going back over Bobcat’s notes. I really think it’s going to be better than ever once it finishes up here and I’m it’s looking as soon as this evening!
That is a great question about the sync status. I’m seeing conflicting answers about it too. Bobcat says this:
A fully synced miner will have a miner height that is larger than the blockchain height.
I believe this is what you must have read as well. The only problem is that can’t be correct and doesn’t make any sense. The blockchain height is exactly what it sounds like, it is what block the network is on or the “real time” active block. Your miner can’t be ahead of it (unless it’s something like a mining point is always 1 block ahead because that’s the active block everyone on the network is always working on or a technicality like that).
If this is actually true, it can only be true because Bobcat’s own status display’s “blockchain_height” (or this variable might even come from the Helium client) has some kind of delay and it’s behind (meaning it’s a display issue). It can’t actually be true in reality, because the blockchain height is the current block that *all* miners are on.
Being several blocks ahead of a blockchain doesn’t even make sense to say. How would it still be “linked” together? What is filling those links in-between the block you are on and the block the blockchain is on? If it was linked to the chain it would be a part of the blockchain and a part of the height so they must be floating predetermined magic links and it just makes no sense when you break it down.
5. I see a negative gap in Sync Status. Is it normal?
Yes. It means your miner is fully synced and ahead of the blockchain. This is normal.
Again Bobcat, no, this would not be normal if it was a reflection of what was happening in reality. How could your miner be blocks ahead of the entire blockchain? This has to be some kind of display/update issue but Bobcat seems pretty sure this is normal so we can be reasonably sure it’s normal for the Bobcat, but I am not sure the word normal means what they think it means. They seem to be right about this being normal behavior (basically Helium network lag) but with a complete misunderstanding of what is actually happening (or what is actually possible).
That’s okay, they are mining hardware manufacturers. No one said they were blockchain programming experts (remember, it runs the Helium core software/firmware internally as all Helium points do which takes care of most of those parts) so I’m not hating on them, but it’s a little embarrassing that this explanation breaks down with the most basic thinking through of what a blockchain actually is and how the blocks link together etc.
Maybe in these cases even what your miner thinks the blockchain height is ends up being several blocks behind what your active mining thread/process is actively working on (which would basically be a display issue due to lag/delays in what it thinks the blockchain_height is). It wouldn’t surprise me if the other miner models might see similar behavior (it may be a Helium thing) but I’m still waiting for my Nebra outdoor point to have something to compare the Bobcat with! It may be “normal” but is actually lag/latency and not at all what Bobcat is saying with your miner being multiple blocks ahead of the blockchain, that is not possible.
In my experience (and I believe I read this somewhere but I can’t find it after searching for some time) it’s something like within ~50 blocks is considered fully synced (at least as far as the Explorer is concerned, that might be different than what Bobcat shows in the Bobcat Diagnoser mining status for example). It might even be one of the hover over or descriptions on the Helium explorer or the app (which would explain why I can’t find a link) but it’s something close to that and there is some x amount of breathing room given before you fall back into “Syncing” status officially.
It depends who is measuring though as Bobcat has their criteria / Helium has it’s criteria but it’s probably your official Helium “status” we’re concerned about and that is the one that has some breathing room before it will start gossiping that you’re behind and in “Syncing” status. The official Helium syncing status is “gossiped” over the network so it can be super slow to update (just like getting out of relayed status) just as a heads up. Hopefully that helps!
Thank for your response, I agree with your logic that it doesn’t make sense for the miner block to pass the blockchain height.
status "syncing"
gap "1964"
miner_height "1010983"
blockchain_height "1012947"
epoch "26130"
the gap is getting smaller but the progression slowed down. Do you think I should try the fast sync again or be patient? I would hate for the progress made thus far to be for nothing.
Yeah I definitely wouldn’t do the fast sync at this point on it, it’s way too close to risk having to start over. It’s definitely normal for it to slow down a little bit (or even moderately) as it gets closer. The reason is that a lot of the fellow miners you are connected to are also syncing and the closer you get to the newest block the fewer of your peers there will be that actually have the newest blocks because they themselves don’t have them yet either. Some of them may be 50 blocks behind, some maybe 500 behind, others still 10,000 or more etc. This cuts both ways as you also weren’t able to help them/others sync their newer blocks this whole time either.
If it feels like it’s *really* slowed down a lot the most I would do is power cycle it so that it gets a new set of random connections. It is a thing that sometimes you get “unlucky” and will be connected to a bunch of slow peers that aren’t sending blocks very quickly. If you’re in relayed status this is much more likely but it can still happen in regular normal mode syncing as well (it’s just all probabilities, you can theoretically get a set of 4 bad connections or 20 bad connections etc.).
I would definitely take no action though since I think you’re closer now than you were before the reset! Once it syncs and you get out of relayed status if it falls behind in the future you should be able to safely use the fast sync from now on now that the temperatures are stable. Doing a power cycle usually won’t set you back too far though and can definitely get things unstuck sometimes.
Have you done the “Net Speed” test on the miner? It’s not wireless and getting like a flaky 10Mbps connection to the attic is it? (The Bobcat’s WiFi card is terrible from what I can tell, I have 6 Unifi APs in my house with a central system (Unifi’s “Network” app) that measures all devices signal strength and it’s by far the lowest weakest signal device 100% of the time by a solid 10-20% even line of sight 20-30ft from an AP).
It’s definitely worth checking to make sure that the net speed test doesn’t say you’re getting a horrendous connection as that would 100% explain it. If it’s hard wired then this won’t help us of course but I thought I would mention it. The attic isn’t line of sight and WiFi gets hurt *bad* going through walls and on my house the attic floor (my ceiling depending on your perspective) is SUPER thick.
It would be smarter/wiser to use the “reboot” from the Bobcat Diagnoser menu instead of literally pull the power since there’s always a chance when you pull the power during a write it could corrupt the disk again. Since you don’t want to start over the safest thing to do is give it a gentle reboot with the Bobcat Diagnoser menu and it looks like it should finish within the next few hours or so here!
status "syncing"
gap "1881"
miner_height "1011614"
blockchain_height "1013495"
epoch "26149"
The gap has closed a little but at this rate I don’t think it will ever catch up so I will consider the power cycle today. Another variable that I just thought of is the antenna, would this have anything to do with syncing? I don’t think it would but I purchased x2, 8dbi and x2, 5.8dbi. The antennas that have the 5.8 seem are both operating while the ones that have the 8dbi antennas don’t. Should I try swapping to the 4 dbi antenna the miner came with?
Unfortunately it’s the Bobcat breaking again. My Bobcat’s earnings have dropped to 36 cents per 24 hours (soon to be zero, the last little pittance of HNT hasn’t dropped off but will soon). It can’t stay caught up. My gap grew from 51 last night to 80 today:
These things break *all* *the* *time*. It has broke about once a week since I got it for months and months and months and it breaks for everyone else too best I can tell constantly. The Bobcat has to be the least reliable one by far (and the easiest one to get by far).
Look how many people in that last one. Welcome to the Bobcat owner club! You will not a find a single person happy with it. At this point you can go back months and months in my comments here and see how many times it has broken for sure. This isn’t their first rodeo either and if you think my outlook was negative here wait until you see the wrath of reddit. Lots of them are straight up proclaiming the Helium network dead which is the melodrama from reddit you would expect. It is getting pretty tiring though for sure that they can’t just get/keep it working at all. They drop like flies and apparently people have been tracking the Bobcat’s all around their areas and they all go down all the time.
It’s their software/firmware. They always manage to get it barely working again somehow but it never lasts more than a week or so before the same problem start again. I honestly think they just put way too little RAM / CPU power on the Bobcat and really cheaped out. All the hotspot manufacturers I’m sure wish at this point they would have got an extra 1GB of RAM and not been so cheap. They are paying for it now by trying to get a relatively demanding application at this point (Helium) to run on SBCs/embedded computers with 1GB of RAM (or less) on board which is slow/underequipped even by SBC standards (I was buying bigger Pis than the manufacturers gave us with these hotspots literally 5-8 years ago).
All we can really do is wait for the next Bobcat firmware update that will probably get it working for a few days again before it will break yet again. Sorry for the negative outlook, but yeah this is like my 20th time and I honestly cannot wait for the Nebra to get here! It’s a Raspberry Pi that uses a SD card so I’ll definitely be able to break into that one and see what is going wrong (which Bobcat doesn’t really let us do). My guess is that we will see a firmware update pushed today or tomorrow for this.
Snapshot Updates
This release fixes some issues with snapshot loading, specifically with a Net Emissions related variable that was causing snapshot loading to fail and download cleanup issue that was leaving large partial downloads on disk.
Networking Improvements
This release also includes two networking improvements that should help Hotspots report the correct IP address to other blockchain participants.
We recommend Hotspot owners read the Helium Engineering Blog post as there were a lot of changes in this Helium miner update.
This sounds exactly like what you are getting hit with. That large downloads on disk one probably explains your crash a couple of days ago. There was a major required firmware update pushed to address exactly that. It says it would cause snapshot loading to fail which would have gave you your “error” crashed out mining container that wouldn’t start exactly like you got. That may be mystery solved on that one.
If you have the firmware update it may be safe for you to try the fast sync again. The snapshot loading issue is supposedly resolved (if yours has updated already, if it hasn’t it’s just as likely to crash as last time because it would mean you have the broken version still, it’s randomly scheduled for twice a day so that not all miners check at the same time).
Even if you end up having the can’t stay caught up issue like a lot of the rest of us are having (such as the reddit posts I linked to with other examples) you may as well take advantage of this time to catch it up as far as the fast sync will take you. The firmware update might even fix that too since it sounds like network syncing was also specifically addressed.
It looks like this might not even have been Bobcat’s fault (this time) and was actually a Helium issue/update. With this change, it may be worth power cycling your miner and see if you can trigger the firmware updates (works sometimes but definitely well under 25% of the time, it seems to depend on how close you are to your regularly scheduled check or if you missed one previously it will immediately check to make up for it).
If it doesn’t work after trying it once then you aren’t close enough to one and trying it 10 more times won’t work either, the only thing that might work is trying it after some period of time to see if it is close enough to give it to you. The best is if you missed a check for some reason (network connectivity, etc.) and have it queued up in there (Bobcat *has* confirmed that it will make up for missed checks, this change was made to address issues with people missing it and being stuck on old versions for a really long time and they updated/fixed it).
It’s not random chance based which is why rebooting it over and over won’t do anything. It’s really based on time and seems to maybe have a little wiggle room based on if you are within a certain period but even that is speculation (not confirmed) and it may just be if you haven’t “missed” one it’s not going to happen until the randomly scheduled time passes.
We don’t know what the timer is scheduled for so even if we have a gut feeling there’s some wiggle room there’s no way of seeing/knowing so anything we’ve observed so far almost certainly has some confirmation bias or other factors may be at play that we don’t understand / can’t see.
You’d want to wait at least an hour or two before trying to trigger it again because overdoing the power cycles definitely does a lot of damage (it takes a long time to reestablish all your network connection links, you lose even more syncing time, and other events running on timers may be missed as well) and has it’s own risks associated with it. UPDATE: Bobcat has also confirmed that they are pushing the firmware updates out in “batches”. This means the best thing to do is leave your miner plugged in and online!
Software power cycles (using the Bobcat Diagnoser reboot option) are dramatically safer than yanking the power from the wall as well since you don’t have the risk of power being lost in the middle of a write to storage. This happens all the time on computers and makes Windows have to run a chkdsk (Disk Checker), and potentially a fsck for Linux/Mac to clean up the mess. You won’t be able to do this on the miner (there’s no fsck option) and your only option will be to let it wipe it clean and reformat the disk blank and resync the blockchain.
Awesome Bonus Link: Mind the Gap! How to Fix a Huge Sync Gap from the Bobcat Blog. This confirms that it should be safe to use the fast sync to catch up. It even has a way to trigger it from the CLI using curl! Here’s a sample:
Many of you have probably suffered from a large sync gap recently, leading to a lack of activity and rewards. The Bobcat Discord has been flooded with user complaints and the support channels are swamped and hence, a lot of hotspots are stuck with a steep mountain to climb.
UPDATE #2: It’s starting to look like the update that fixed the snapshot issue might be partially causing the sync issues. It definitely seems like there’s still ongoing issues but we’ll see how it develops here throughout the day/tomorrow.
Thanks for info as always James, I didn’t get a chance to mess with the miner today but may have some time this evening/ tomorrow. I am going to go ahead and power cycle as it seems based on my snapshots the miner is starting to lag behind the gap.
(Yesterday) status "syncing"
gap "1881"
miner_height "1011614"
blockchain_height "1013495"
epoch "26149"
(Today/ right now) status "syncing"
gap "1886"
miner_height "1012234"
blockchain_height "1014120"
epoch "26168"
For the latest helium firmware update, is there any indication where I can see if its been applied or what version I am on? I believe that update you mentioned is 2021.09.14.0.
For now I am going to do a fast sync (if that doesn’t work)–> then a power cycle into a fast sync and see what happens. But of course first I need to figure out if I got the latest firmware update.
Also, this is crazyyyyy lol I also wish they didn’t cheap out —> “It’s their software/firmware. They always manage to get it barely working again somehow but it never lasts more than a week or so before the same problem start again. I honestly think they just put way too little RAM / CPU power on the Bobcat and really cheaped out. All the hotspot manufacturers I’m sure wish at this point they would have got an extra 1GB of RAM and not been so cheap. They are paying for it now by trying to get a relatively demanding application at this point (Helium) to run on SBCs/embedded computers with 1GB of RAM (or less) on board which is slow/underequipped even by SBC standards (I was buying bigger Pis than the manufacturers gave us with these hotspots literally 5-8 years ago).”
Welp great news, it seems like your deduction at putting two and two together worked. Identifying that the issue was related to the recent update, then advising on the fast sync again.
status "syncing"
gap "44"
miner_height "1014128"
blockchain_height "1014172"
epoch "26226"
While the status is still showing syncing, I am sure it will be remedied here in a much shorter time. Do you happen to have a discord or anything like that? I’d love to stay in touch james… otherwise I will keep bugging you on here :)! .. and will 100 percent confirm if the miner comes out of its flatlined state.
That’s great news! It has been quite a ride. I don’t have a Discord set up yet but I’ve considered it but you can always find me anywhere here for sure!
It’s been quite a ride as you got caught up right in the middle of this firmware mess but I look forward to final confirmation that it’s 100% working!
Thanks for your detailed notes you left throughout the process (will no doubt help others) and feel free to stop by any time. I’ll add some more content as well (I usually go on a content blitz and will publish a whole bunch all at once) so stay tuned and thanks again!
Dennis Boboc
2 years ago
Can someone help?? My miner is showing Relayed and I went thru all of the steps (at least I think I did) and I can’t get it to completely sync.
I would for sure recommend to let it sync and stop pulling the power from it now because all of your diagnostic output looks correct (open NAT, port status open). It can take anywhere from 8-16 hours for your “relayed” status to stop showing in the explorer. It will have a really hard time catching up now unless you just let it sit and sync and then before too long here the relayed status should drop off the explorer here.
It’s only ~300ish blocks behind and honestly it’s probably from messing with the settings / removing the power / rebooting (3 minutes uptime on this output). That isn’t even far enough to use the “Fast Sync” feature (which requires a minimum of 400 blocks behind) so yours is considered too close to need to do anything dramatic (nor will it let you).
It’s also about how far we would expect it to be behind after a small amount of downtime. It can take a little bit of time to ramp up which is why I was suggesting leaving the power plugged in so it can ramp up and catch up. I honestly think you’re good to go here and just need to let it sit.
Your ports / NAT are open. You’re just waiting for the explorer and it’s a very long wait!
Kevin
2 years ago
Hello James, first off i wanna say your information on your blogs are wonderful and your insight is greatly appreciated!!! I cannot connect to my bobcat diagnoser at all. I can confirm i am using the rite IP address of my miner, i am on the same local network, and the format i am entering is httx://192.168.X.X/. Google chrome gives me this error ERR_CONNECTION_TIMED_OUT everytime i try.
I do not want to open port 80 because of the security risk you have mentioned in the previous comments referring to a miner takeover (Please correct me if i am misunderstood). So how do i connect to my miner via google chrome (windows) with my private IP address instead of my public, as i assume this is what is causing the issue after reading what has previously been commented. Sorry if this is a silly question but any help would be appreciated!!!
That’s very strange but I do have an idea based on a previous comment someone left that let them get around this for some reason. Can you try going to httx://192.168.X.X:80/ with the :80 appended on the end there? That got it for someone else that was having the same issue!
tried this but no luck, no my understanding isnt this the way to do it if you have opened port 80. i have not done this, should i do this to be bale to use the diagnoser? i assumed opening the port would put my miner at more of a risk?
Not unless you’re accessing it by your public IP address which if you open port 80 for your public IP address then anyone could access the diagnoser theoretically (hopefully the Bobcat is smart enough to not even listen on your public IP whether the port is opened or not, most programs are as this is a big security no-no, but I’m not totally sure if they did this or not but I certainly hope so given everyone has the same password you can’t change).
You don’t need to open ports for a private IP address because nobody can access a private IP address unless they are on your local network anyway. Are you trying to access it by your internet IP by chance?
The other possibility is you have two different networks. Is the miner on WiFi and your computer is wired in or vice versa? Does your wireless router hand out different IPs than your wired router? This type of a setup won’t work and is called double NAT. If your miner is wired then connect to it from a wired computer. If it’s wireless then connect from your phone or a WiFi device. Definitely double check those two things and it should hopefully get you going!
my bobcat is on wifi, and i am trying to access it via my windows 10 laptop on wifi, and they are both on the same wireless network. and i am using a unfi setup with no double NAT. i have gotten on to my other miners at my other locations where the wifi is alot more simple. I have somewhat of an idea what im doing when it comes to networking kinda haha. this is why i decided the use a whole unfi setup but at my home. sadly out of the 5 bobcats i have this is the only one i cannot connect to via IP.
That context helps a lot! It also helps to know you only have the one unit doing it.
What happens if you take the miner somewhere else or put it on a different connection? Can you get it to it then?
We probably need to narrow down if it’s the miner or the network. If it’s the miner then this may be one of the rare occasions where it may be worth trying one of the “Reset Miner” options to reset back to factory defaults and let it set it up again from scratch. That’s because if you’ve eliminated the network then something must have gone wrong during the deployment of the miner and likely just setting it up again will fix it.
If it’s the network we need to take a closer look at your network. What is your laptops IP compared to the miner’s IP? Can you ping it?
thanks so much for your help so far no luck yet :/……how do you recommend pinging the minor from my laptop (windows 10)???… The minor and the laptop both have similar Ip addresses only differing by the last 4 numbers. and both devices are connected to the same unifi wireless access point on the same floor at my house and they are all on the same LAN…
after trying tp ping within windows cmd, it says destination host unreachable 4x then it loads to 4 packets sent / 4 packets received / 0 Lost…. it does this everytime i try running the ping command, im not sure if this is means its successfully communicating with my network or not.
the IP adress that is coming back through the cmd ping is not the one im entering in, it looks as if its coming from the public IP adress of the miner. my reasoning for this is it specifically says, reply from (ip address, which is not the private one i entered to ping) destination host unreachable 4x then it loads to 4 packets sent / 4 packets received / 0 Lost
That means they probably have “Device Isolation” turned on in the Unifi network settings. That means devices on WiFi (or sometimes the entire LAN even) won’t be able to communicate with each other unless there are specific rules in place to allow it. This is common in enterprise/work networks for sure. It’s a security feature because 90% of the time wireless devices really do have no business talking to each other and nothing good usually comes from it (just bots scanning other devices on networks they infect, etc.).
This would need to be changed or the miner moved to a different network that isn’t in “isolation” mode and then your laptop would also need to be on that network to talk to it. Device isolation mode blocks pings, web traffic, everything from device to device even on the same LAN. You’re basically being locked out by security features if that makes sense!
thanks for your help, i believe you have identified the problem now i must find the remedy as i am still new to unifi. the unifi setup is actually my home i did this setup for a hobbyish approach but i can see this is going to be interesting. Do you have any idea on where to look to figure the rules needed within unifi to establish connection to the miner? i most certainly have full admin access to the network but nor full admin knowledge lol!
Kyle Smith
2 years ago
Hello James,
I’m getting relayed on 6 of my seven deployed miners. I’m using mifi and I have them all port forwarded with the bobcat diagnostic show the port open. However, I notice my “nat” settings in the diagnostic showed symmetric. Yours showed none. Any advice. My home unit works great!
Basically what is going on is that virtually all wireless (as in 4G, 5G, LTE, etc.) providers are often not actually giving you a real public IP. Typically you’re sharing a public IP with hundreds/thousands of other customers at once. They’re “cheating” and you likely actually are behind NAT right now already before it even gets to your router/hotspot unfortunately. They seem to all do this (apparently even internationally according to the thread I linked above).
In his case there was a super cheap (from an American perspective, $1.85/mo US I believe he said, good luck getting a US wireless provider to do anything for $1.85, it probably costs more than that to even expand your voice mail capacity) dynamic public IP addon service (meant primarily to address Xbox Live and PSN issues, which also need an open NAT for the best experience) he could get but he is in Europe.
(Sidenote: since very few of the presumably billions of wireless devices on any of these networks actually have real public IPV4 addresses (at least not one for each device unless they paid for a reservation, it’s NAT) if they actually did we would for sure not have enough IPv4 addresses for all of them. It’s probably also why cellphones outright prefer IPV6 and only use IPV4 as a last resort while computers often barely touch IPV6 and don’t require having it still to this day to do basically anything)
Who is your provider? Here’s some MiFi specific information about public IPs from AT&T here. AT&T you seem to need to get a static IP to get an open NAT which you don’t even want to know how much it costs (but if you do it’s in the link, and you need to switch to business class). Verizon also seems to be NAT. If you have one of these providers you may want to actually contact them and see if they have some kind of updated or cheaper option available than what they’re saying in these threads because those are truly outrageous.
Are you by chance outside of the US? You may have better options if you are (or if you live in part of the US that has some of the few surviving regional carriers that may still offer reasonable services/prices to non-business customers). I lucked out last time and they were!
Oliwer
2 years ago
Do you know credentials for Reboot option? It would be great to have that option, but without credentials it’s dummy option
Great question. I did not know that this had a username and password when I updated the guide because I didn’t follow through rebooting all the way and stopped after the first or second warning message during testing assuming that the reboot would actually complete all the way. Sorry about that!
The username is: bobcat
The password is: miner
I’m not sure what the point of putting this on there was if we can’t change it and they’re all the same. I guess maybe to prevent people who just stumbled onto it and don’t know what it is? At any rate that should take care of it (it’s also required for the resync option)!
Hi James, some great detailed info here! Thank you for that. I do have an issue though and perhaps you’d be able to give me some advise. Since 2 days ago everything flatlined. I logged in, saw that my miner “desynced” so to speak. Block height 1, epoch 1… that is all the way from being fully synced.
However, the main issue now is that I can not login to it from the Web, being on the same subnet and all, I also can not turn on the Bluetooth on it to run any diagnostics from my mobile app… so basically nothing at all… I’m lost to be honest. The lights switch from short timed yellow to long timed red. Rarely cyan.
My other bobcat is accessible and working…
The updated Helium firmware contains a snapshot update. This release should help Hotspots sync up to the most recent block faster. The blessed snapshot in this release is from block 994321. Read the Helium engineering blog for all the details here.
That “snapshot” should be the starting point theoretically. It definitely sounds like the miner is crashed out (especially from the web interface and everything else being inaccessible).
The first thing I would do (if you haven’t already) is unplug it for about 60 seconds and plug it back in and then wait for a little bit to see if it triggers a firmware update to fix it. That firmware I linked to above was released right around the time you started having trouble and it may correct itself after a reboot.
It’s likely it will fix itself (Bobcat has always fixed bricked miners like this in the past with OTA updates eventually) but I do know a sneaky trick. Helium miners (all of them I believe, this doc is for the original Helium miner but RAK does this and Bobcat as well) come with a very old hidden feature that if you hold down the Bluetooth pairing button while booting you can revert to the previous firmware:
To boot the previous firmware, use the following procedure:
Unplug the power
Press down the black pairing button on the side of the Hotspot
Reconnect the power while holding down the black pairing button
While holding down the button, wait for the status LED to light up, then wait an additional 10 seconds
Release the button
This will cause the Hotspot to boot the previous firmware where the race condition was far less likely and the Hotspot will be able to use its configured Wifi credentials.
Now once it boots it will probably just update to the latest firmware again but we really just want to get it unstuck. Something must have gone wrong initially and retrying it may get it to succeed the next time (which is why I said just try rebooting it first and giving it 15-30 minutes or so and then check the status lights).
This happened to me as well not too long ago and my miner was bricked and completely inaccessible over the web interface / app for 2 days (this one was due to a containerd error that was fixed a couple of updates ago in my case) then magically fixed itself so I wouldn’t panic but it’s definitely annoying since I lost money from the downtime and you are too. If power cycling / the firmware revert trick doesn’t get it going then I would leave it plugged in still and connected to the internet (even if it’s showing red).
Thank you very much James for your reply and information! I shall try it as soon as I’m home from work today! 🙂
I’ll update once I have more news…
Have a great day!
Daniel
I am having issues getting a browser to access the diagnostic tool. When I enter the ip address of the miner into any browser or device on my network I get a connection refused error “ERR_CONNECTION_REFUSED”.
Is this a known issue, and do you have any experience resolving something like this?
Great question! The first thing to check is to make sure you are connecting with a private IP address (usually 192.x.x.x or 10.x.x.x) and not your public IP as that will definitely get refused.
You also want to make absolutely sure you put a http:// in front of it. HTTPS will NOT work (it doesn’t have a certificate, won’t respond on the ports). An example of mine at home is:
httx://192.168.42.201/
and that will bring up my Bobcat 300 Diagnoser Interface (says your point’s name right at the top like the screenshot).
If I try to connect with my public IP (the IP address you will get if you type what’s my IP in Google, 174.23.158.254 right now and usually looks nothing like a private IP) I will get connection refused. If you try that IP before my IP changes you’ll actually get taken to one of my remote access login pages. This is because at home my port 80 is forwarded to a different service on a different server. It’s also why it won’t work by public IP. You would have to forward port 80 to the Bobcat and then anyone would be able to access your miner’s interface because there is no password of any kind on this stuff (they could just click the most time consuming tasks over and over and basically do a denial of service (DOS) attack on your miner)!
Does your miner happen to be in relayed status? If so it’s possible that you have a WiFi network and a wired network that have separate private IPs. This will for sure break your miner and put it in “Relayed” yellow status and port forwarding will not work. Your connection in such a setup (this is called a double NAT setup, two private NAT networks with different private IPs) would be an external connection. My guide for fixing that is here .
In a double NAT setup (or a setup where you have a wired router that gives out a set of IPs then a WiFi network that gives out a totally separate IPs) you’ll definitely get connection refused. You’d be seen as an external connection by your wireless router (or your wired router, this can go both ways) if one of the devices isn’t in bridge mode.
If you aren’t in relayed status then the IP to put in the browser is the *exact* same IP you used for your port forward. Not your external IP or anything else. The same one you used for the port forward to fix relayed status.
There is a final possibility but I think it’s really unlikely. It’s possible your Bobcat firmware hasn’t updated but *everyone* should have this as of around the 3rd or 4th of August but if you’re positive then you may want to power cycle the Bobcat and see if you can trigger a firmware update but definitely check everything else first. Hopefully that helps!
I am also having no luck communicating with my Bobcat. It is in relay mode. I setup up port forwarding as you laid out. In diagnostics it says the firmware is 7.22.2021. I rebooted it to see if it would trigger an update but no luck. Is this possibly because its stuck in relay mode and has no internet access?
That absolutely does sound too old. Relayed mode shouldn’t stop it from getting the firmware updates usually but it might slow it down quite a bit. Not having internet access at all is quite another thing though.
I’d honestly probably plug it into wired and see if you can get it to update as the firmware was released 8/3/2021 so yours is too old somehow!
I had 07.22 when I received it this past Thursday. Now, it’s running current firmware, but it has ERR CONNECTION REFUSED. Seems the Bobcat doesn’t have the webservice running or is just refusing connections on :80
Are you connecting by private IP address like 192.168.x.x? If you try to connect by your public IP address it won’t work without a port forward.
It doesn’t really make a lot of sense. Can you ping the miner with the “ping” command?
Microsoft Windows [Version 10.0.19042.1165]
(c) Microsoft Corporation. All rights reserved.
Z:\>ping bobcatminer
Pinging bobcatminer.jamesachambers.net [192.168.1.58] with 32 bytes of data:
Reply from 192.168.1.58: bytes=32 time=2ms TTL=64
Reply from 192.168.1.58: bytes=32 time=15ms TTL=64
Ping statistics for 192.168.1.58:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 15ms, Average = 8ms
In my case httx://192.168.1.58/ gets me the Bobcat utility in the web browser.
Are you on a different private network than your miner? Like are you connected with a different set of IP addresses than the Bobcat has? If you look in your Helium app what does it say your local / private IP is (should tell you in the diagnostics section)?
hey James,
I have an issue with my bobcat, I cannot connect to it using the private IP, nor can ping it.
I do have port forwarding set up, also binded the IP to the MAC address, but nothing, in the Helium app it says the miner is online (not relayed), has 7 witnesses, but no rewards in the last 48 hours.
Any help will be greatly appreciated. Thank you!
That is strange for sure. Are your cable/DSL modem and your WiFi device on separate networks?
What is your IP address vs. the miner? I’m guessing the miner has to be something like 192.168.0.x and you must be 192.168.1.x or something like that and you’re on different subnets.
If you aren’t in relayed status then you must have your port forwarding piece set up correctly. If you log into the Helium app what does it say your private IP is in there if you do the Bluetooth pairing and check the diagnostics? If you are connected wired and the Bobcat is wireless (or vice versa) you may need to switch your connection mode as I suspect you actually have two separate networks (they won’t be able to talk to each other, they’re separated even if it doesn’t seem like it).
If you had it the other way around you would be in relayed status despite having your port forward set up correctly. You basically avoided this by having the Bobcat be on the same network as whatever is assigning IP addresses on your network. Your wired computer probably isn’t and might be connected to the modem or something like that. Hopefully that helps!
And also when Ive try to go in Discovery Mode it says that “unable to initiate discovery mode, hotspot may be behind a relay and is not responding. check your router settings and try again” even though it is not relayed. Help me please
That means it’s not able to reach the hotspot right now. Is the light on the Bobcat actually green? When does the explorer say it last updated your status and you weren’t relayed? Mine says 33 minutes ago but I’m wondering if yours is stale.
You can try unplugging the power and plugging it back in (but you’ll have to wait a good 15 minutes after doing this) to see if it can get your connection reestablished to the network. If you’ve never been able to do this then it’s probably actually in relayed even if the explorer isn’t showing it.
Connecting to the diagnostics would help. You’ll probably have to do the Bluetooth pairing to talk to it and find out what is going on with it!
I definitely can’t explain it, everything definitely looks to be in order! Can you hit port 80 by public IP by chance? What about if you port forward port 80? (don’t leave this open if it works, at the very least anyone would be able to fool with your miner and click the functions that make it pause and stop mining for 30 seconds etc. or you may accidentally expose a web service somewhere on your network using a default password or missing a patch and get haxored)
I’ve got other things on the network that are functioning normally (a Synology NAS with Docker and services working on various ports). I would say that since :80 isn’t working on that IP, something’s wrong and it’s not worth potentially exposing the miner to the public. . Maybe the next OTA update might patch mine up. I’d reach out to Bobcat, but it’s been days since a ticket was opened for this issue. No reply yet.
I just got a reply back from them (Bobcat) from an email I sent them on Sunday (asking to update my unit).
Little did I know that it would update automatically.
I added my query of the diagnose page not working to the email reply back.
Will let you know what they say.
That should definitely not be necessary (the http:// already tells it port 80) but I wonder if you have browser extensions or security plugins trying to upgrade your connection to SSL (the Brave Browser does this too by default unless you disable “Upgrade connections to HTTPS” in the Brave Shield) then then :80 part may be hinting to it no, I for sure want you go to port 80 and stay in http.
It could also be some sort of web cache problem (either your browser’s cache or sometimes it’s network equipment cache above your computer like a router) and by qualifying with the :80 it’s forcing everything to resolve again since that won’t match the string you went to originally with the port left off the end.
That’s my best guess anyway but if it worked for you I bet it will work for others as well. Thanks again!
Hello guy, it’s possible to access this “Diagnoser Tool” outside the network somehow remotely using the router or something?
Hey Alex,
That’s a really good question. I really hope it doesn’t listen on the WAN IP since some of the options in there are to literally wipe the miner clean and sync from scratch (going to take a while). If it does then forwarding port 80 to the miner would let you access it as well as literally everyone else in the world. Make sure you understand that, if it lets you do this then make sure you turn it off after you’re done or remember that anyone has full access to your Bobcat miner!
My house has a VPN set up inside the router (lots of routers support this but it’s less common on consumer ones you’d find at Best Buy and more common on enterprise/business ones). What I mean exactly by this is that my router hosts it’s own VPN service basically as my public IP address and you connect on a VPN client to my public IP. Since it has that I can actually access anything on my local network from any device in the world that has an internet connection (including my phone) and that would be a good/secure way to set it up.
You could even use something like LogMeIn / TeamViewer or one of the gazillions of cloud screen sharing apps to do this. Chrome Remote Desktop as basically a part of your browser is a pretty accessible solution as well. If you install a cloud enabled one on your computer for example you can use your cloud remote desktop session and access any service you want including the Bobcat diagnoser from another computer, your phone through an app, etc.
Basically my recommendation would be to set up some kind of remote desktop for yourself (lots of free ones available for home users). You really don’t want to forward the Bobcat traffic to your local network through something like a port forward if they actually even do listen on the WAN IP for connections on port 80. Definitely the remote desktop setup (which can be free) is designed with security in mind for remote access and will let you do this better and so much more securely!
James, fantastic website/ guides. Super easy to use and understand, thank you for doing this. I am having an issue with my miner ever since the update 4 days ago (1.0.2.23S).
The miner had no issues and was synced/mining, then it went to a syncing status and got stuck at 99 percent. I went ahead and use the new feature, fastsync last night and it did clear up errors that were previously there.
This morning it’s still syncing and now says it needs attention. Below are the lastest logs, the only thing i noticed is that im behind on blocks. Any advice on a way forward?
timestamp "2021-09-14 15:55:55 +0000 UTC"temp0 "72 °C"
temp1 "71 °C"
temp_alert "alert"
ota_version "1.0.2.23S"
sync_status ""
miner
State "running"
Status "Up 10 hours"
Names
0 "/miner"
Image "quay.io/team-helium/miner:miner-arm64_2021.09.03.0_GA"
Created 1631590913
height
0 "25850 1001680"
1 ""
epoch "25850"
miner_height "1001680"
blockchain_height "1010832"
Hey David,
Thanks for the kind words! I think I can help. Yours actually looks a little bit different than some of the previous ones posted here. Your miner is blazing hot at the moment at 72 degrees C (161.6 F for SI challenged Americans like myself). You actually have a temperature alert/warning which means you’re at risk of lockups/crashes (or worse if it goes on for too long):
temp0 "72 °C"temp1 "71 °C"
temp_alert "alert"
Is this miner outside by chance? You aren’t far behind enough on blocks to do anything with it. It’s only about ~150 blocks behind and it needs to be a minimum of 400 for the Bobcat to even let you do anything to try to make it sync faster.
Your issue is it’s most likely locking up/crashing due to high temperatures or might even be throttling to keep itself from completely overheating/crashing out. It might just have the airholes blocked/obstructed but the temps are too high and are literally in the redzone / warning zone. Lowering your temps should fix all of it including the syncing!
James, great observation and feedback. The miner is inside on the third floor/attic space. It does get pretty hot and I’ve been turning off the AC to save on power. I will crank the AC and see what happens.
Insightful about the blocks as well, I calculated that I was 9,152 blocks behind. I subtracted the blockchain height with the miner height to get that answer, am I doing that wrong? If so if you have any resource on how to determine how far back I am it would be truly appreciated. Also, hope you don’t mind, but I’ve shared your site with my friends on discord :).
Hey David,
That sounds right! The attic gets pretty hot. You shouldn’t need to get the temperature down too much. I believe the warning starts at 70C which is pretty standard for electronics (some will take 80-90 but 70C is when a lot of things are really going to start cooking).
You’re right on the math. I missed that one number started with 1010 and the other was 1001 which is how I came up with 152. You did the calculation correctly by subtracting the miner height from the blockchain height to get the difference so I wanted to clarify that this was my mistake on this one and you are correct on the blocks!
In that case you actually can use the “fast sync” feature to catch up once you get the miner a few degrees cooler (definitely a few ways you can do this, even just putting a fan up there blowing directly on the miner would probably do it even with the warmer air because it will pull enough heat off the miner to get it out of the redzone). Once it’s back in the 60s the warning should drop off and you’ll stop getting throttled. I’ve used the fast sync a couple of times since it came out and it worked for me both times after about 30 minutes.
Mine was really struggling a few days ago staying synced and fell very far behind. Finally Bobcat released some firmware updates that seem to have stabilized it and then I used fast sync one more time and after about 30 minutes it will have you caught up!
timestamp "2021-09-14 17:50:40 +0000 UTC"temp0 51
temp1 50
unit "°C"
Thanks for the clarification on the math and owning the error! Also, you are brilliant. I fortunately can just put my miner in front of my AC and got the above results. The “needs attention” alert on the app went away so it’s a step in the right direction. I will now try the fast sync feature once more and report back on the result. I love when you troubleshoot something and actually get to a resolution, best feeling ever.
Hey David,
Oh wow, that turned out fantastic! I think you’re on the right track for sure here now. It makes sense that it was running at like 25-50% speed under throttling etc.
I look forward to hearing how it all turns out but I have a good feeling about it so far!
status ""gap ""
miner_height "timeout"
blockchain_height "1011114"
epoch "RPC"
temps are super cool now
timestamp "2021-09-14 20:55:10 +0000 UTC"temp0 42
temp1 40
unit "°C"
Here is the latest, it looks like my miner height timed out now, the bobcat miner is stuck on yellow. I gave it some time (a couple hours) and am wondering if I should unplug the miner and try the fast sync again. Any thoughts James before I try that?
Here are my current next steps for troubleshooting :
Power cycle then try Fast Sync again
Resync Miner
Reset Miner
Try these troubleshooting steps off the Helium site.
Hey David,
Yeah your plan looks good here. When I ran the fast sync the Bobcat Diagnoser page will actually kind of change and some parts will turn yellow. It will also tell you the miner light will turn yellow.
Before you do that though I would log into the Bobcat Diagnoser and see what the exact status you have is. Take a look at what block it is on now. Did the fast sync do anything? It might be getting close or it might have made zero progress in which case yeah it’s time for a power cycle and starting it again. It’s an alpha feature so it’s definitely not at all inconceivable it could have got stuck and needs another try before it’s going to work.
If you check it though and it actually has made a lot of progress and is getting close I would let it finish. The fast sync will only take you to within about 400 blocks of current then you still have to sync the “last mile” yourself (this is actually why they don’t let you use it if you aren’t at least 400 blocks behind, it literally wouldn’t do any good because the snapshot only goes up to that point).
(Oops, I realized you posted your status after I posted this and it has an error status, yes definitely reboot this one!)
2 hours seems too long though. I would check the status and if it hasn’t made any progress give the process another start here!
Just stopping in to give an update, I went ahead and rebooted and used fast sync again and the same thing happened, I was stuck on the same block height as before, which was “1001680”.
I went ahead and used the full re-sync feature and things seem to be moving towards the right direction, kinda. I checked my block height and it did change to “994345”, then I checked about an hour later and it’s “994475” currently. It’s at the very least moving so I will be patient and check in a few hours.
Hey David,
Thanks for the update! I have heard of such things. It lines up with the Bobcat release notes:
Apparently if your miner is not “healthy” to begin with the fast sync may not work and you did exactly the right thing. I’m guessing by healthy they mean that your miner is really far behind but that the copy of the blockchain you have is intact / uncorrupted.
It’s very possible invalid data got recorded when it was that hot as weird things (“undefined behavior”) can start to happen (and much stranger than this even) under those temperatures. Now that the temperatures are under control a resync should result in 100% valid data.
Let us know as it continues to move in the right direction / catches up!
Interesting about the importance about the temperatures, but also you brought up a pretty interesting point. Since it’s early I may go ahead and give this a shot and just restart if things get messed up again but, I think I should use the fast sync option now since my miner is “healthy”. The gap is currently 16,000 so it would be a good benchmark. Thank you for the consultation, I love the bouncing of ideas. I am going to give this a shot and will report back here the results.
Hey David,
Glad I could help! I think you can go ahead and use it now safely too. It is not typical for it to get corrupted like that and it actually caught me a little off guard when I first read your message but then I thought a little bit more about it and yeah anything can happen when the temps get that high (especially corruption/normally impossible things). It may have been like this for some time before it started having symptoms enough to notice.
Definitely let us know how it works out if you do for sure!
Don’t appear to be lucky with the fast sync, climbing slowly though.. at this rate ill be done in 8 or 9 days lol
"994345""994475"
"994577"
"994696"
"994720"
Yeah I agree that doesn’t look like the fast sync. My guess is it has to do with the notes saying it will only work for “otherwise healthy miners”. Since yours has never been sync’d fully yet since resetting maybe you don’t meet that criteria. I’m not sure because the features are only a week or two old and there’s not anything more to go on than what Bobcat tells us basically in their release notes that I constantly link to (this is where I get most of my info, straight from Bobcat’s release notes, I read every line). They’re also in “alpha” status (which is considered even earlier than “beta”) so it could have flat out been a bug that’s never supposed to happen for all we know.
It may just need to sync normally with this type of reset. At least it starts from a snapshot (and not from 0 even though the Bobcat diagnoser tells you it is going to start from 0 I have never seen it go 0, 1, 2, etc. syncing the chain and it always starts at some snapshot, the “blessed” snapshot for full syncs I believe). The sync speed varies too depending on who you are connected to etc. so definitely keep an eye on it but I would definitely let it finish since it doesn’t seem like we have much choice / much else we can do in this case! I don’t expect it to take 8-9 days fortunately as I think it should get picking up a little faster here over time!
This is definitely why I lean so heavily against telling people to do the full reset (it’s like getting it brand new and mine took 6 days to sync brand new but it doesn’t take as long these days because of improvements since I got my point) but I really don’t think you had much of a choice. Your status was completely crashed out into “error” and didn’t even have a blockchain status 2.5 hours after starting the fast sync. This can happen for sure and usually means the miner “container” was crashed out (this is what Bobcat refers to it as, it’s probably a docker container) but a power cycle usually gets it going. It didn’t in your case, which means something was legitimately wrong/broken on the disk/storage.
From Bobcat’s documentation we can assume this means your miner wasn’t “healthy”. Their documentation didn’t really mention though that if it’s not healthy it could completely crash out the miner where you have to fully reset it completely for it to get “healthy” again. Still, this looks like one of the rare appropriate cases to use the *full* reset because the miner was dead, dead, dead. Healthy in this context would refer to the files on your Bobcat’s local storage (namely the blockchain but also the Helium software).
How are you supposed to know if your miner is “healthy” or not? There’s currently no way of knowing other than your intuition if you think your miner is acting up and seems unhealthy to you. Mine acts up all the time (as in it will fall behind on sync or start turning yellow/red randomly) with the typical Bobcat firmware/falling behind on sync occasionally issues but I have never had to do a full reset or been prevented from using the fast sync feature (although maybe I should do a full reset, maybe that’s why mine acts up every week or two because it does need the full reset I’m trying to avoid).
We know these issues are widespread though because Bobcat admits to them in the patch notes and makes fixes to alleviate them *constantly*. They release a firmware update once or twice a week and have since I got my point and almost all of them have some sort of stability fix. You can see this for yourself as well here and everything that says “Bobcat Miner OTA” is a firmware update and if you click it you can see the notes. I’m guessing finding Bobcats that don’t occasionally stop syncing / go to yellow/red status for no apparent reason is quite rare.
Nothing in the status report tells us whether your copy of the blockchain / local files on the Bobcat are “healthy” or anything like that so hopefully Bobcat makes some improvements here to give us an idea of what kind of condition our miner’s disk and filesystem are in and whether those are “healthy”.
If it’s on WiFi right now it may sync faster on wired (my Bobcat’s wireless link even to APs 20-30 feet away is at best 300Mbit and usually closer to 100-150Mbit, fine for normal activity but syncing the whole chain it’s better to have more) but otherwise let’s let it finish and you should be good to go!
Hey James – You were right that it wouldn’t take 8-9 days, currently my gap is around 2272, “1010118” gap “2272”. Looks like things are going to be working here by this evening.
I wanted to ask, what should my height block be to be syncing? I think I read somewhere that it had to be at or above the blockchain height for it to be synced, is that accurate? A friend of mine just used the fast sync feature with success and he’s around 30 blocks behind, is that considered to be synced?
Hey David,
I’m so relieved that things did speed up, I actually felt bad that everything led to you having to do the full reset but I really 100% think it was for the best after doing some more research/learning today and going back over Bobcat’s notes. I really think it’s going to be better than ever once it finishes up here and I’m it’s looking as soon as this evening!
That is a great question about the sync status. I’m seeing conflicting answers about it too. Bobcat says this:
A fully synced miner will have a miner height that is larger than the blockchain height.I believe this is what you must have read as well. The only problem is that can’t be correct and doesn’t make any sense. The blockchain height is exactly what it sounds like, it is what block the network is on or the “real time” active block. Your miner can’t be ahead of it (unless it’s something like a mining point is always 1 block ahead because that’s the active block everyone on the network is always working on or a technicality like that).
If this is actually true, it can only be true because Bobcat’s own status display’s “blockchain_height” (or this variable might even come from the Helium client) has some kind of delay and it’s behind (meaning it’s a display issue). It can’t actually be true in reality, because the blockchain height is the current block that *all* miners are on.
Being several blocks ahead of a blockchain doesn’t even make sense to say. How would it still be “linked” together? What is filling those links in-between the block you are on and the block the blockchain is on? If it was linked to the chain it would be a part of the blockchain and a part of the height so they must be floating predetermined magic links and it just makes no sense when you break it down.
I also found this in the Bobcat Diagnoser user guide:
5. I see a negative gap in Sync Status. Is it normal?Yes. It means your miner is fully synced and ahead of the blockchain. This is normal.
Again Bobcat, no, this would not be normal if it was a reflection of what was happening in reality. How could your miner be blocks ahead of the entire blockchain? This has to be some kind of display/update issue but Bobcat seems pretty sure this is normal so we can be reasonably sure it’s normal for the Bobcat, but I am not sure the word normal means what they think it means. They seem to be right about this being normal behavior (basically Helium network lag) but with a complete misunderstanding of what is actually happening (or what is actually possible).
That’s okay, they are mining hardware manufacturers. No one said they were blockchain programming experts (remember, it runs the Helium core software/firmware internally as all Helium points do which takes care of most of those parts) so I’m not hating on them, but it’s a little embarrassing that this explanation breaks down with the most basic thinking through of what a blockchain actually is and how the blocks link together etc.
Maybe in these cases even what your miner thinks the blockchain height is ends up being several blocks behind what your active mining thread/process is actively working on (which would basically be a display issue due to lag/delays in what it thinks the blockchain_height is). It wouldn’t surprise me if the other miner models might see similar behavior (it may be a Helium thing) but I’m still waiting for my Nebra outdoor point to have something to compare the Bobcat with! It may be “normal” but is actually lag/latency and not at all what Bobcat is saying with your miner being multiple blocks ahead of the blockchain, that is not possible.
In my experience (and I believe I read this somewhere but I can’t find it after searching for some time) it’s something like within ~50 blocks is considered fully synced (at least as far as the Explorer is concerned, that might be different than what Bobcat shows in the Bobcat Diagnoser mining status for example). It might even be one of the hover over or descriptions on the Helium explorer or the app (which would explain why I can’t find a link) but it’s something close to that and there is some x amount of breathing room given before you fall back into “Syncing” status officially.
It depends who is measuring though as Bobcat has their criteria / Helium has it’s criteria but it’s probably your official Helium “status” we’re concerned about and that is the one that has some breathing room before it will start gossiping that you’re behind and in “Syncing” status. The official Helium syncing status is “gossiped” over the network so it can be super slow to update (just like getting out of relayed status) just as a heads up. Hopefully that helps!
Thank for your response, I agree with your logic that it doesn’t make sense for the miner block to pass the blockchain height.
status "syncing"gap "1964"
miner_height "1010983"
blockchain_height "1012947"
epoch "26130"
the gap is getting smaller but the progression slowed down. Do you think I should try the fast sync again or be patient? I would hate for the progress made thus far to be for nothing.
Hey David,
Yeah I definitely wouldn’t do the fast sync at this point on it, it’s way too close to risk having to start over. It’s definitely normal for it to slow down a little bit (or even moderately) as it gets closer. The reason is that a lot of the fellow miners you are connected to are also syncing and the closer you get to the newest block the fewer of your peers there will be that actually have the newest blocks because they themselves don’t have them yet either. Some of them may be 50 blocks behind, some maybe 500 behind, others still 10,000 or more etc. This cuts both ways as you also weren’t able to help them/others sync their newer blocks this whole time either.
If it feels like it’s *really* slowed down a lot the most I would do is power cycle it so that it gets a new set of random connections. It is a thing that sometimes you get “unlucky” and will be connected to a bunch of slow peers that aren’t sending blocks very quickly. If you’re in relayed status this is much more likely but it can still happen in regular normal mode syncing as well (it’s just all probabilities, you can theoretically get a set of 4 bad connections or 20 bad connections etc.).
I would definitely take no action though since I think you’re closer now than you were before the reset! Once it syncs and you get out of relayed status if it falls behind in the future you should be able to safely use the fast sync from now on now that the temperatures are stable. Doing a power cycle usually won’t set you back too far though and can definitely get things unstuck sometimes.
Have you done the “Net Speed” test on the miner? It’s not wireless and getting like a flaky 10Mbps connection to the attic is it? (The Bobcat’s WiFi card is terrible from what I can tell, I have 6 Unifi APs in my house with a central system (Unifi’s “Network” app) that measures all devices signal strength and it’s by far the lowest weakest signal device 100% of the time by a solid 10-20% even line of sight 20-30ft from an AP).
It’s definitely worth checking to make sure that the net speed test doesn’t say you’re getting a horrendous connection as that would 100% explain it. If it’s hard wired then this won’t help us of course but I thought I would mention it. The attic isn’t line of sight and WiFi gets hurt *bad* going through walls and on my house the attic floor (my ceiling depending on your perspective) is SUPER thick.
It would be smarter/wiser to use the “reboot” from the Bobcat Diagnoser menu instead of literally pull the power since there’s always a chance when you pull the power during a write it could corrupt the disk again. Since you don’t want to start over the safest thing to do is give it a gentle reboot with the Bobcat Diagnoser menu and it looks like it should finish within the next few hours or so here!
I haven’t done the net speed, below is the results for the first one. The miner is direct connected to an ethernet.
DownloadSpeed "63 Mbit/s"UploadSpeed "14 Mbit/s"
Latency "13.768562ms"
Current sync status
status "syncing"gap "1881"
miner_height "1011614"
blockchain_height "1013495"
epoch "26149"
The gap has closed a little but at this rate I don’t think it will ever catch up so I will consider the power cycle today. Another variable that I just thought of is the antenna, would this have anything to do with syncing? I don’t think it would but I purchased x2, 8dbi and x2, 5.8dbi. The antennas that have the 5.8 seem are both operating while the ones that have the 8dbi antennas don’t. Should I try swapping to the 4 dbi antenna the miner came with?
Hey David,
Unfortunately it’s the Bobcat breaking again. My Bobcat’s earnings have dropped to 36 cents per 24 hours (soon to be zero, the last little pittance of HNT hasn’t dropped off but will soon). It can’t stay caught up. My gap grew from 51 last night to 80 today:
{"status": "syncing",
"gap": "80",
"miner_height": "1013472",
"blockchain_height": "1013552",
"epoch": "26206"
}
We’re not the only ones.
These things break *all* *the* *time*. It has broke about once a week since I got it for months and months and months and it breaks for everyone else too best I can tell constantly. The Bobcat has to be the least reliable one by far (and the easiest one to get by far).
Look at this one.
Look how many people in that last one. Welcome to the Bobcat owner club! You will not a find a single person happy with it. At this point you can go back months and months in my comments here and see how many times it has broken for sure. This isn’t their first rodeo either and if you think my outlook was negative here wait until you see the wrath of reddit. Lots of them are straight up proclaiming the Helium network dead which is the melodrama from reddit you would expect. It is getting pretty tiring though for sure that they can’t just get/keep it working at all. They drop like flies and apparently people have been tracking the Bobcat’s all around their areas and they all go down all the time.
It’s their software/firmware. They always manage to get it barely working again somehow but it never lasts more than a week or so before the same problem start again. I honestly think they just put way too little RAM / CPU power on the Bobcat and really cheaped out. All the hotspot manufacturers I’m sure wish at this point they would have got an extra 1GB of RAM and not been so cheap. They are paying for it now by trying to get a relatively demanding application at this point (Helium) to run on SBCs/embedded computers with 1GB of RAM (or less) on board which is slow/underequipped even by SBC standards (I was buying bigger Pis than the manufacturers gave us with these hotspots literally 5-8 years ago).
All we can really do is wait for the next Bobcat firmware update that will probably get it working for a few days again before it will break yet again. Sorry for the negative outlook, but yeah this is like my 20th time and I honestly cannot wait for the Nebra to get here! It’s a Raspberry Pi that uses a SD card so I’ll definitely be able to break into that one and see what is going wrong (which Bobcat doesn’t really let us do). My guess is that we will see a firmware update pushed today or tomorrow for this.
***UPDATE: Actually, there was a firmware update pushed (don’t miss the comments at the bottom to see how many others are having issues right now):
Snapshot UpdatesThis release fixes some issues with snapshot loading, specifically with a Net Emissions related variable that was causing snapshot loading to fail and download cleanup issue that was leaving large partial downloads on disk.
Networking Improvements
This release also includes two networking improvements that should help Hotspots report the correct IP address to other blockchain participants.
We recommend Hotspot owners read the Helium Engineering Blog post as there were a lot of changes in this Helium miner update.
This sounds exactly like what you are getting hit with. That large downloads on disk one probably explains your crash a couple of days ago. There was a major required firmware update pushed to address exactly that. It says it would cause snapshot loading to fail which would have gave you your “error” crashed out mining container that wouldn’t start exactly like you got. That may be mystery solved on that one.
If you have the firmware update it may be safe for you to try the fast sync again. The snapshot loading issue is supposedly resolved (if yours has updated already, if it hasn’t it’s just as likely to crash as last time because it would mean you have the broken version still, it’s randomly scheduled for twice a day so that not all miners check at the same time).
Even if you end up having the can’t stay caught up issue like a lot of the rest of us are having (such as the reddit posts I linked to with other examples) you may as well take advantage of this time to catch it up as far as the fast sync will take you. The firmware update might even fix that too since it sounds like network syncing was also specifically addressed.
It looks like this might not even have been Bobcat’s fault (this time) and was actually a Helium issue/update. With this change, it may be worth power cycling your miner and see if you can trigger the firmware updates (works sometimes but definitely well under 25% of the time, it seems to depend on how close you are to your regularly scheduled check or if you missed one previously it will immediately check to make up for it).
If it doesn’t work after trying it once then you aren’t close enough to one and trying it 10 more times won’t work either, the only thing that might work is trying it after some period of time to see if it is close enough to give it to you. The best is if you missed a check for some reason (network connectivity, etc.) and have it queued up in there (Bobcat *has* confirmed that it will make up for missed checks, this change was made to address issues with people missing it and being stuck on old versions for a really long time and they updated/fixed it).
It’s not random chance based which is why rebooting it over and over won’t do anything. It’s really based on time and seems to maybe have a little wiggle room based on if you are within a certain period but even that is speculation (not confirmed) and it may just be if you haven’t “missed” one it’s not going to happen until the randomly scheduled time passes.
We don’t know what the timer is scheduled for so even if we have a gut feeling there’s some wiggle room there’s no way of seeing/knowing so anything we’ve observed so far almost certainly has some confirmation bias or other factors may be at play that we don’t understand / can’t see.
You’d want to wait at least an hour or two before trying to trigger it again because overdoing the power cycles definitely does a lot of damage (it takes a long time to reestablish all your network connection links, you lose even more syncing time, and other events running on timers may be missed as well) and has it’s own risks associated with it. UPDATE: Bobcat has also confirmed that they are pushing the firmware updates out in “batches”. This means the best thing to do is leave your miner plugged in and online!
Software power cycles (using the Bobcat Diagnoser reboot option) are dramatically safer than yanking the power from the wall as well since you don’t have the risk of power being lost in the middle of a write to storage. This happens all the time on computers and makes Windows have to run a chkdsk (Disk Checker), and potentially a fsck for Linux/Mac to clean up the mess. You won’t be able to do this on the miner (there’s no fsck option) and your only option will be to let it wipe it clean and reformat the disk blank and resync the blockchain.
Awesome Bonus Link: Mind the Gap! How to Fix a Huge Sync Gap from the Bobcat Blog. This confirms that it should be safe to use the fast sync to catch up. It even has a way to trigger it from the CLI using curl! Here’s a sample:
UPDATE #2: It’s starting to look like the update that fixed the snapshot issue might be partially causing the sync issues. It definitely seems like there’s still ongoing issues but we’ll see how it develops here throughout the day/tomorrow.
Thanks for info as always James, I didn’t get a chance to mess with the miner today but may have some time this evening/ tomorrow. I am going to go ahead and power cycle as it seems based on my snapshots the miner is starting to lag behind the gap.
(Yesterday)
status "syncing"gap "1881"
miner_height "1011614"
blockchain_height "1013495"
epoch "26149"
(Today/ right now) status
"syncing"gap "1886"
miner_height "1012234"
blockchain_height "1014120"
epoch "26168"
For the latest helium firmware update, is there any indication where I can see if its been applied or what version I am on? I believe that update you mentioned is 2021.09.14.0.
For now I am going to do a fast sync (if that doesn’t work)–> then a power cycle into a fast sync and see what happens. But of course first I need to figure out if I got the latest firmware update.
Also, this is crazyyyyy lol I also wish they didn’t cheap out —> “It’s their software/firmware. They always manage to get it barely working again somehow but it never lasts more than a week or so before the same problem start again. I honestly think they just put way too little RAM / CPU power on the Bobcat and really cheaped out. All the hotspot manufacturers I’m sure wish at this point they would have got an extra 1GB of RAM and not been so cheap. They are paying for it now by trying to get a relatively demanding application at this point (Helium) to run on SBCs/embedded computers with 1GB of RAM (or less) on board which is slow/underequipped even by SBC standards (I was buying bigger Pis than the manufacturers gave us with these hotspots literally 5-8 years ago).”
Welp great news, it seems like your deduction at putting two and two together worked. Identifying that the issue was related to the recent update, then advising on the fast sync again.
status "syncing"gap "44"
miner_height "1014128"
blockchain_height "1014172"
epoch "26226"
While the status is still showing syncing, I am sure it will be remedied here in a much shorter time. Do you happen to have a discord or anything like that? I’d love to stay in touch james… otherwise I will keep bugging you on here :)! .. and will 100 percent confirm if the miner comes out of its flatlined state.
Hey David,
That’s great news! It has been quite a ride. I don’t have a Discord set up yet but I’ve considered it but you can always find me anywhere here for sure!
It’s been quite a ride as you got caught up right in the middle of this firmware mess but I look forward to final confirmation that it’s 100% working!
challenge created, the miner is back! ty james!
Hooray, congratulations! I am also finally green again myself and all caught up.
There was a new update released to date to address some of this (just for completeness) here.
Thanks for your detailed notes you left throughout the process (will no doubt help others) and feel free to stop by any time. I’ll add some more content as well (I usually go on a content blitz and will publish a whole bunch all at once) so stay tuned and thanks again!
Can someone help?? My miner is showing Relayed and I went thru all of the steps (at least I think I did) and I can’t get it to completely sync.
{“timestamp”: “2021-09-13 23:54:46 +0000 UTC”,
“temp0”: “51 °C”,
“temp1”: “47 °C”,
“temp_alert”: “normal”,
“ota_version”: “1.0.2.23S”,
“sync_status”: “”,
“miner”: {
“State”: “running”,
“Status”: “Up 3 minutes”,
“Names”: [
“/miner”
],
“Image”: “quay.io/team-helium/miner:miner-arm64_2021.09.03.0_GA”,
“Created”: 1631577105
},
“height”: [
“26085 1009505”,
“”
],
“epoch”: “26085”,
“miner_height”: “1009505”,
“blockchain_height”: “1009896”,
“ports”: {
“44158”: “open”
},
“region”: “US915”,
“pubkey”: “112W3h1C4rUpjFp5F26xYw2o9xeXihMvWaDGmsgSAGgfQrTM28ku”,
“animal”: “boxy-goldenrod-huskie”,
“p2p_status”: [
“+———+——-+”,
“| name |result |”,
“+———+——-+”,
“|connected| yes |”,
“|dialable | yes |”,
“|nat_type | none |”,
“| height |1009505|”,
“+———+——-+”,
“”,
“”
],
“onboarding”: [
“”,
“********************”,
“General Manufacturing Info”,
“********************”,
“”,
“+————-+—————————————————-+”,
“| name | result |”,
“+————-+—————————————————-+”,
“| batch | bobcat_902 |”,
“| createdAt | 2021-09-02T20:27:33.577Z |”,
“|heliumSerial | null |”,
“| id | 263390 |”,
“| macEth0 | e8:78:29:53:94:d2 |”,
“| macWlan0 | 48:e7:da:65:22:15 |”,
“| makerId | 6 |”,
“|onboardingKey|112W3h1C4rUpjFp5F26xYw2o9xeXihMvWaDGmsgSAGgfQrTM28ku|”,
“|publicAddress|112W3h1C4rUpjFp5F26xYw2o9xeXihMvWaDGmsgSAGgfQrTM28ku|”,
“| rpiSerial | 7ac2c4d23eb55f59 |”,
“| updatedAt | 2021-09-10T22:40:37.944Z |”,
“+————-+—————————————————-+”,
“”,
“”,
“********************”,
“Manufacturer Info”,
“********************”,
“”,
“+——————+—————————————————+”,
“| name | result |”,
“+——————+—————————————————+”,
“| address |14sKWeeYWQWrBSnLGq79uRQqZyw3Ldi7oBdxbF6a54QboTNBXDL|”,
“| createdAt | 2021-02-24T22:16:39.379Z |”,
“| id | 6 |”,
“|locationNonceLimit| 1 |”,
“| name | Bobcat |”,
“| updatedAt | 2021-02-24T22:16:39.379Z |”,
“+——————+—————————————————+”,
“”,
“”
],
“errors”: “”
}
Hey Dennis,
I would for sure recommend to let it sync and stop pulling the power from it now because all of your diagnostic output looks correct (open NAT, port status open). It can take anywhere from 8-16 hours for your “relayed” status to stop showing in the explorer. It will have a really hard time catching up now unless you just let it sit and sync and then before too long here the relayed status should drop off the explorer here.
It’s only ~300ish blocks behind and honestly it’s probably from messing with the settings / removing the power / rebooting (3 minutes uptime on this output). That isn’t even far enough to use the “Fast Sync” feature (which requires a minimum of 400 blocks behind) so yours is considered too close to need to do anything dramatic (nor will it let you).
It’s also about how far we would expect it to be behind after a small amount of downtime. It can take a little bit of time to ramp up which is why I was suggesting leaving the power plugged in so it can ramp up and catch up. I honestly think you’re good to go here and just need to let it sit.
Your ports / NAT are open. You’re just waiting for the explorer and it’s a very long wait!
Hello James, first off i wanna say your information on your blogs are wonderful and your insight is greatly appreciated!!! I cannot connect to my bobcat diagnoser at all. I can confirm i am using the rite IP address of my miner, i am on the same local network, and the format i am entering is
httx://192.168.X.X/. Google chrome gives me this error ERR_CONNECTION_TIMED_OUT everytime i try.I do not want to open port 80 because of the security risk you have mentioned in the previous comments referring to a miner takeover (Please correct me if i am misunderstood). So how do i connect to my miner via google chrome (windows) with my private IP address instead of my public, as i assume this is what is causing the issue after reading what has previously been commented. Sorry if this is a silly question but any help would be appreciated!!!
Hey Kevin,
That’s very strange but I do have an idea based on a previous comment someone left that let them get around this for some reason. Can you try going to
httx://192.168.X.X:80/with the :80 appended on the end there? That got it for someone else that was having the same issue!tried this but no luck, no my understanding isnt this the way to do it if you have opened port 80. i have not done this, should i do this to be bale to use the diagnoser? i assumed opening the port would put my miner at more of a risk?
Hey Kevin,
Not unless you’re accessing it by your public IP address which if you open port 80 for your public IP address then anyone could access the diagnoser theoretically (hopefully the Bobcat is smart enough to not even listen on your public IP whether the port is opened or not, most programs are as this is a big security no-no, but I’m not totally sure if they did this or not but I certainly hope so given everyone has the same password you can’t change).
You don’t need to open ports for a private IP address because nobody can access a private IP address unless they are on your local network anyway. Are you trying to access it by your internet IP by chance?
The other possibility is you have two different networks. Is the miner on WiFi and your computer is wired in or vice versa? Does your wireless router hand out different IPs than your wired router? This type of a setup won’t work and is called double NAT. If your miner is wired then connect to it from a wired computer. If it’s wireless then connect from your phone or a WiFi device. Definitely double check those two things and it should hopefully get you going!
my bobcat is on wifi, and i am trying to access it via my windows 10 laptop on wifi, and they are both on the same wireless network. and i am using a unfi setup with no double NAT. i have gotten on to my other miners at my other locations where the wifi is alot more simple. I have somewhat of an idea what im doing when it comes to networking kinda haha. this is why i decided the use a whole unfi setup but at my home. sadly out of the 5 bobcats i have this is the only one i cannot connect to via IP.
Hey Kevin,
That context helps a lot! It also helps to know you only have the one unit doing it.
What happens if you take the miner somewhere else or put it on a different connection? Can you get it to it then?
We probably need to narrow down if it’s the miner or the network. If it’s the miner then this may be one of the rare occasions where it may be worth trying one of the “Reset Miner” options to reset back to factory defaults and let it set it up again from scratch. That’s because if you’ve eliminated the network then something must have gone wrong during the deployment of the miner and likely just setting it up again will fix it.
If it’s the network we need to take a closer look at your network. What is your laptops IP compared to the miner’s IP? Can you ping it?
thanks so much for your help so far no luck yet :/……how do you recommend pinging the minor from my laptop (windows 10)???… The minor and the laptop both have similar Ip addresses only differing by the last 4 numbers. and both devices are connected to the same unifi wireless access point on the same floor at my house and they are all on the same LAN…
after trying tp ping within windows cmd, it says destination host unreachable 4x then it loads to 4 packets sent / 4 packets received / 0 Lost…. it does this everytime i try running the ping command, im not sure if this is means its successfully communicating with my network or not.
the IP adress that is coming back through the cmd ping is not the one im entering in, it looks as if its coming from the public IP adress of the miner. my reasoning for this is it specifically says, reply from (ip address, which is not the private one i entered to ping) destination host unreachable 4x then it loads to 4 packets sent / 4 packets received / 0 Lost
Hey Kevin,
That means they probably have “Device Isolation” turned on in the Unifi network settings. That means devices on WiFi (or sometimes the entire LAN even) won’t be able to communicate with each other unless there are specific rules in place to allow it. This is common in enterprise/work networks for sure. It’s a security feature because 90% of the time wireless devices really do have no business talking to each other and nothing good usually comes from it (just bots scanning other devices on networks they infect, etc.).
This would need to be changed or the miner moved to a different network that isn’t in “isolation” mode and then your laptop would also need to be on that network to talk to it. Device isolation mode blocks pings, web traffic, everything from device to device even on the same LAN. You’re basically being locked out by security features if that makes sense!
thanks for your help, i believe you have identified the problem now i must find the remedy as i am still new to unifi. the unifi setup is actually my home i did this setup for a hobbyish approach but i can see this is going to be interesting. Do you have any idea on where to look to figure the rules needed within unifi to establish connection to the miner? i most certainly have full admin access to the network but nor full admin knowledge lol!
Hello James,
I’m getting relayed on 6 of my seven deployed miners. I’m using mifi and I have them all port forwarded with the bobcat diagnostic show the port open. However, I notice my “nat” settings in the diagnostic showed symmetric. Yours showed none. Any advice. My home unit works great!
Hey Kyle,
I think I may know what is going on. In my fixing relayed status post this came up and it was with someone using a 4G gateway. Here’s the exact thread.
Basically what is going on is that virtually all wireless (as in 4G, 5G, LTE, etc.) providers are often not actually giving you a real public IP. Typically you’re sharing a public IP with hundreds/thousands of other customers at once. They’re “cheating” and you likely actually are behind NAT right now already before it even gets to your router/hotspot unfortunately. They seem to all do this (apparently even internationally according to the thread I linked above).
In his case there was a super cheap (from an American perspective, $1.85/mo US I believe he said, good luck getting a US wireless provider to do anything for $1.85, it probably costs more than that to even expand your voice mail capacity) dynamic public IP addon service (meant primarily to address Xbox Live and PSN issues, which also need an open NAT for the best experience) he could get but he is in Europe.
(Sidenote: since very few of the presumably billions of wireless devices on any of these networks actually have real public IPV4 addresses (at least not one for each device unless they paid for a reservation, it’s NAT) if they actually did we would for sure not have enough IPv4 addresses for all of them. It’s probably also why cellphones outright prefer IPV6 and only use IPV4 as a last resort while computers often barely touch IPV6 and don’t require having it still to this day to do basically anything)
Who is your provider? Here’s some MiFi specific information about public IPs from AT&T here. AT&T you seem to need to get a static IP to get an open NAT which you don’t even want to know how much it costs (but if you do it’s in the link, and you need to switch to business class). Verizon also seems to be NAT. If you have one of these providers you may want to actually contact them and see if they have some kind of updated or cheaper option available than what they’re saying in these threads because those are truly outrageous.
Are you by chance outside of the US? You may have better options if you are (or if you live in part of the US that has some of the few surviving regional carriers that may still offer reasonable services/prices to non-business customers). I lucked out last time and they were!
Do you know credentials for Reboot option? It would be great to have that option, but without credentials it’s dummy option
Hey Oliwer,
Great question. I did not know that this had a username and password when I updated the guide because I didn’t follow through rebooting all the way and stopped after the first or second warning message during testing assuming that the reboot would actually complete all the way. Sorry about that!
The username is: bobcat
The password is: miner
I’m not sure what the point of putting this on there was if we can’t change it and they’re all the same. I guess maybe to prevent people who just stumbled onto it and don’t know what it is? At any rate that should take care of it (it’s also required for the resync option)!
Hi James, some great detailed info here! Thank you for that. I do have an issue though and perhaps you’d be able to give me some advise. Since 2 days ago everything flatlined. I logged in, saw that my miner “desynced” so to speak. Block height 1, epoch 1… that is all the way from being fully synced.
However, the main issue now is that I can not login to it from the Web, being on the same subnet and all, I also can not turn on the Bluetooth on it to run any diagnostics from my mobile app… so basically nothing at all… I’m lost to be honest. The lights switch from short timed yellow to long timed red. Rarely cyan.
My other bobcat is accessible and working…
Any thoughts? Any way to get in otherwise?
Thanks for your time in advance 🙂
Daniel
Hey Daniel,
Thanks for the kind words, and wow, that’s a wild one! It should also literally be impossible (allow me to explain).
When the miner syncs from scratch it doesn’t start from block 1. It downloads a bootstrap file basically and starts from there. Check out here to see some technical documentation.
That “snapshot” should be the starting point theoretically. It definitely sounds like the miner is crashed out (especially from the web interface and everything else being inaccessible).
The first thing I would do (if you haven’t already) is unplug it for about 60 seconds and plug it back in and then wait for a little bit to see if it triggers a firmware update to fix it. That firmware I linked to above was released right around the time you started having trouble and it may correct itself after a reboot.
It’s likely it will fix itself (Bobcat has always fixed bricked miners like this in the past with OTA updates eventually) but I do know a sneaky trick. Helium miners (all of them I believe, this doc is for the original Helium miner but RAK does this and Bobcat as well) come with a very old hidden feature that if you hold down the Bluetooth pairing button while booting you can revert to the previous firmware:
Now once it boots it will probably just update to the latest firmware again but we really just want to get it unstuck. Something must have gone wrong initially and retrying it may get it to succeed the next time (which is why I said just try rebooting it first and giving it 15-30 minutes or so and then check the status lights).
This happened to me as well not too long ago and my miner was bricked and completely inaccessible over the web interface / app for 2 days (this one was due to a containerd error that was fixed a couple of updates ago in my case) then magically fixed itself so I wouldn’t panic but it’s definitely annoying since I lost money from the downtime and you are too. If power cycling / the firmware revert trick doesn’t get it going then I would leave it plugged in still and connected to the internet (even if it’s showing red).
Definitely let us know what you find!
Thank you very much James for your reply and information! I shall try it as soon as I’m home from work today! 🙂
I’ll update once I have more news…
Have a great day!
Daniel
No problem at all, best of luck!
Hello,
I am having issues getting a browser to access the diagnostic tool. When I enter the ip address of the miner into any browser or device on my network I get a connection refused error “ERR_CONNECTION_REFUSED”.
Is this a known issue, and do you have any experience resolving something like this?
Hey John,
Great question! The first thing to check is to make sure you are connecting with a private IP address (usually 192.x.x.x or 10.x.x.x) and not your public IP as that will definitely get refused.
You also want to make absolutely sure you put a http:// in front of it. HTTPS will NOT work (it doesn’t have a certificate, won’t respond on the ports). An example of mine at home is:
httx://192.168.42.201/and that will bring up my Bobcat 300 Diagnoser Interface (says your point’s name right at the top like the screenshot).
If I try to connect with my public IP (the IP address you will get if you type what’s my IP in Google, 174.23.158.254 right now and usually looks nothing like a private IP) I will get connection refused. If you try that IP before my IP changes you’ll actually get taken to one of my remote access login pages. This is because at home my port 80 is forwarded to a different service on a different server. It’s also why it won’t work by public IP. You would have to forward port 80 to the Bobcat and then anyone would be able to access your miner’s interface because there is no password of any kind on this stuff (they could just click the most time consuming tasks over and over and basically do a denial of service (DOS) attack on your miner)!
Does your miner happen to be in relayed status? If so it’s possible that you have a WiFi network and a wired network that have separate private IPs. This will for sure break your miner and put it in “Relayed” yellow status and port forwarding will not work. Your connection in such a setup (this is called a double NAT setup, two private NAT networks with different private IPs) would be an external connection. My guide for fixing that is here .
In a double NAT setup (or a setup where you have a wired router that gives out a set of IPs then a WiFi network that gives out a totally separate IPs) you’ll definitely get connection refused. You’d be seen as an external connection by your wireless router (or your wired router, this can go both ways) if one of the devices isn’t in bridge mode.
If you aren’t in relayed status then the IP to put in the browser is the *exact* same IP you used for your port forward. Not your external IP or anything else. The same one you used for the port forward to fix relayed status.
There is a final possibility but I think it’s really unlikely. It’s possible your Bobcat firmware hasn’t updated but *everyone* should have this as of around the 3rd or 4th of August but if you’re positive then you may want to power cycle the Bobcat and see if you can trigger a firmware update but definitely check everything else first. Hopefully that helps!
I am also having no luck communicating with my Bobcat. It is in relay mode. I setup up port forwarding as you laid out. In diagnostics it says the firmware is 7.22.2021. I rebooted it to see if it would trigger an update but no luck. Is this possibly because its stuck in relay mode and has no internet access?
Hey D M,
That absolutely does sound too old. Relayed mode shouldn’t stop it from getting the firmware updates usually but it might slow it down quite a bit. Not having internet access at all is quite another thing though.
I’d honestly probably plug it into wired and see if you can get it to update as the firmware was released 8/3/2021 so yours is too old somehow!
I had 07.22 when I received it this past Thursday. Now, it’s running current firmware, but it has ERR CONNECTION REFUSED. Seems the Bobcat doesn’t have the webservice running or is just refusing connections on :80
Are you connecting by private IP address like 192.168.x.x? If you try to connect by your public IP address it won’t work without a port forward.
It doesn’t really make a lot of sense. Can you ping the miner with the “ping” command?
Microsoft Windows [Version 10.0.19042.1165](c) Microsoft Corporation. All rights reserved.
Z:\>ping bobcatminer
Pinging bobcatminer.jamesachambers.net [192.168.1.58] with 32 bytes of data:
Reply from 192.168.1.58: bytes=32 time=2ms TTL=64
Reply from 192.168.1.58: bytes=32 time=15ms TTL=64
Ping statistics for 192.168.1.58:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 15ms, Average = 8ms
In my case
httx://192.168.1.58/gets me the Bobcat utility in the web browser.Are you on a different private network than your miner? Like are you connected with a different set of IP addresses than the Bobcat has? If you look in your Helium app what does it say your local / private IP is (should tell you in the diagnostics section)?
hey James,
I have an issue with my bobcat, I cannot connect to it using the private IP, nor can ping it.
I do have port forwarding set up, also binded the IP to the MAC address, but nothing, in the Helium app it says the miner is online (not relayed), has 7 witnesses, but no rewards in the last 48 hours.
Any help will be greatly appreciated. Thank you!
Hey Liviu-Adrian,
That is strange for sure. Are your cable/DSL modem and your WiFi device on separate networks?
What is your IP address vs. the miner? I’m guessing the miner has to be something like 192.168.0.x and you must be 192.168.1.x or something like that and you’re on different subnets.
Usually the answer is to put one of the device in bridge mode which I talk about a lot in this article (more the comments for the bridge mode discussions) as well.
If you aren’t in relayed status then you must have your port forwarding piece set up correctly. If you log into the Helium app what does it say your private IP is in there if you do the Bluetooth pairing and check the diagnostics? If you are connected wired and the Bobcat is wireless (or vice versa) you may need to switch your connection mode as I suspect you actually have two separate networks (they won’t be able to talk to each other, they’re separated even if it doesn’t seem like it).
If you had it the other way around you would be in relayed status despite having your port forward set up correctly. You basically avoided this by having the Bobcat be on the same network as whatever is assigning IP addresses on your network. Your wired computer probably isn’t and might be connected to the modem or something like that. Hopefully that helps!
And also when Ive try to go in Discovery Mode it says that “unable to initiate discovery mode, hotspot may be behind a relay and is not responding. check your router settings and try again” even though it is not relayed. Help me please
That means it’s not able to reach the hotspot right now. Is the light on the Bobcat actually green? When does the explorer say it last updated your status and you weren’t relayed? Mine says 33 minutes ago but I’m wondering if yours is stale.
You can try unplugging the power and plugging it back in (but you’ll have to wait a good 15 minutes after doing this) to see if it can get your connection reestablished to the network. If you’ve never been able to do this then it’s probably actually in relayed even if the explorer isn’t showing it.
Connecting to the diagnostics would help. You’ll probably have to do the Bluetooth pairing to talk to it and find out what is going on with it!
I’m on the same IoT VLAN as the miner with my laptop – miner and Mac are on the same subnet (192.168.1.x).
I can ping it by hostname and IP.
PING bobcatminer (192.168.1.101): 56 data bytes64 bytes from 192.168.1.101: icmp_seq=0 ttl=64 time=3.316 ms
PING 192.168.1.101: 56 data bytes
64 bytes from 192.168.1.101: icmp_seq=14509 ttl=64 time=4.794 ms
% nc -z -v 192.168.1.101 80
nc: connectx to 192.168.1.101 port 80 (tcp) failed: Connection refused
% nc -z -v 192.168.1.101 44158
Connection to 192.168.1.101 port 44158 [tcp/*] succeeded!
Seems like the HTTPD service isn’t running on it.
Strange indeed.
I definitely can’t explain it, everything definitely looks to be in order! Can you hit port 80 by public IP by chance? What about if you port forward port 80? (don’t leave this open if it works, at the very least anyone would be able to fool with your miner and click the functions that make it pause and stop mining for 30 seconds etc. or you may accidentally expose a web service somewhere on your network using a default password or missing a patch and get haxored)
I’ve got other things on the network that are functioning normally (a Synology NAS with Docker and services working on various ports). I would say that since :80 isn’t working on that IP, something’s wrong and it’s not worth potentially exposing the miner to the public. . Maybe the next OTA update might patch mine up. I’d reach out to Bobcat, but it’s been days since a ticket was opened for this issue. No reply yet.
Hey Cory,
That’s understandable, definitely let us know if you hear from them and what the outcome is if you wouldn’t mind!
I just got a reply back from them (Bobcat) from an email I sent them on Sunday (asking to update my unit).
Little did I know that it would update automatically.
I added my query of the diagnose page not working to the email reply back.
Will let you know what they say.
I had the same issue and then tried
httx://192.168.X.X:80and the web interface loaded!Great tip!
That should definitely not be necessary (the http:// already tells it port 80) but I wonder if you have browser extensions or security plugins trying to upgrade your connection to SSL (the Brave Browser does this too by default unless you disable “Upgrade connections to HTTPS” in the Brave Shield) then then :80 part may be hinting to it no, I for sure want you go to port 80 and stay in http.
It could also be some sort of web cache problem (either your browser’s cache or sometimes it’s network equipment cache above your computer like a router) and by qualifying with the :80 it’s forcing everything to resolve again since that won’t match the string you went to originally with the port left off the end.
That’s my best guess anyway but if it worked for you I bet it will work for others as well. Thanks again!