Bobcat 300 Diagnoser Tool / Utility Guide (Helium Mining)

Bobcat 300 Diagnoser Utility
Bobcat 300* Diagnoser Utility

The Bobcat 300* Helium Miner finally has an interface you can use to be able to tell what it’s doing! Better yet, it lets you do a few things like conduct an internet speed test. You can even get a detailed breakdown of which block your miner is on!

In this guide I’ll break down how to access the web interface (easy) and run through some of the features that are available at launch (1.0). The utility started installing on Bobcat 300’s via OTA updates on August 3rd 2021. Let’s begin!

Hardware Used

Bobcat 300 Helium Miner
Bobcat 300 Helium Miner

The Nebra outdoor miner is a Helium miner designed to be installed permanently outdoors. It is powered by a Raspberry Pi Compute Module 3 and uses a eMMC chip for storage via a

Links: Amazon.com*

Finding IP Address

Hopefully you already know what the IP address of the Bobcat 300 is on the network as you will need to know that to get your miner out of relayed status and set up the “port forward”.

If you haven’t set up your port forward yet follow my Getting Out of Relayed Status Helium Mining Guide.

You can find your IP by logging into your router’s web interface / app or you can use the Helium app’s “Diagnostics” feature to get it as well!

Accessing Web Interface

Once you’ve determined your miner’s IP type it in your web browser like this:

http://192.168.X.X

It does not like https (port 443) and will only work with http (port 80).

Diagnoser Utility Options

Let’s cover what you can do using this utility at launch.

Temperature

Temperature is a serious obstacle for Bobcat 300 miners. If they are outside or in a very hot / poorly ventilated place they are well known to overheat. This feature helps you manage this and make sure your miner isn’t getting cooked.

If you choose the temperature option you will get the temperature of your unit (as temp.json in JSON format, useful if you are a developer) like this:

{
  "time": "2021-08-07 00:30:51 +0000 UTC",
  "temp0": 47,
  "temp1": 49,
  "unit": "°C"
}

This lets you keep an eye on your miner’s temperature. This one is inside an air conditioned building and it’s still nearly 49C or roughly 120F degrees but would be considered a “good” temperature reading!

Goldshell ASIC     Goldshell ASIC     Goldshell ASIC     Goldshell ASIC     Goldshell ASIC     Goldshell ASIC    

Sync Status (Update 9/3/2021)

New option added in 1.0.3 that is described as:

Introducing the sync status feature on the Bobcat Diagnoser, which answers the very simple yet important question of whether or not your Bobcat miner is properly synced to the network. Stop incessantly rebooting or waiting for the helium app to update the miner’s status. Now you’ll have access to real time information at the push of a button!

Bobcat Miner 9/2/201 Update

The Helium app really is agonizingly slow to update so this is a useful feature. Here’s an example of how it looks:

{
  "status": "synced",
  "gap": "-2",
  "miner_height": "995790",
  "blockchain_height": "995788",
  "epoch": "25674"
}

The “gap” is how many blocks behind you are, so I am 2 blocks behind with a gap of -2. The math adds up with the “blockchain_height” representing the height the entire network is at and “miner_height” representing the block height your miner is at / sync’d to.

Miner

The “Miner” choice gives you an extremely detailed breakdown of your miner. You want to be careful with this option because the menus warn you that this option slows down the Bobcat’s mining (it’s a resource intensive / high CPU operation).

Here’s what it looks like:

{
  "time": "2021-08-07 00:11:15 +0000 UTC",
  "temp0": "48 °C",
  "temp1": "50 °C",
  "temp_alert": "normal",
  "ota_version": "1.0.2.10",
  "miner": {
    "State": "running",
    "Status": "Up About an hour",
    "Names": [
      "/miner"
    ],
    "Image": "quay.io/team-helium/miner:miner-arm64_2021.08.02.0_GA",
    "Created": 1628292218,
    "Ports": [
      {
        "IP": "0.0.0.0",
        "PrivatePort": 1680,
        "PublicPort": 1680,
        "Type": "udp"
      },
      {
        "IP": "0.0.0.0",
        "PrivatePort": 44158,
        "PublicPort": 44158,
        "Type": "tcp"
      }
    ]
  },
  "miner_alert": "",
  "miner_desc": "",
  "pubkey": "1125RGgX5aXXXXX",
  "animal": "shallow-maroon-xxx",
  "region": [
    "US915",
    ""
  ],
  "height": [
    "24468    955285",
    ""
  ],
  "ports": {
    "1680": "closed",
    "44158": "open"
  },
  "p2p_status": [
    "+---------+------+",
    "|  name   |result|",
    "+---------+------+",
    "|connected| yes  |",
    "|dialable | yes  |",
    "|nat_type | none |",
    "| height  |955285|",
    "+---------+------+",
    "",
    ""
  ],
  "onboarding": [
    "",
    "********************",
    "General Manufacturing Info",
    "********************",
    "",
    "+-------------+----------------------------------------------------+",
    "|    name     |                       result                       |",
    "+-------------+----------------------------------------------------+",
    "|    batch    |                     bobcat_609                     |",
    "|  createdAt  |              2021-06-09T09:41:12.706Z              |",
    "|heliumSerial |                        null                        |",
    "|     id      |                       82510                        |",
    "|   macEth0   |                 e8:78:29:51:XX:XX                  |",
    "|  macWlan0   |                 48:e7:da:13:XX:XX                  |",
    "|   makerId   |                         6                          |",
    "|onboardingKey|112xxxx                                             |",
    "|publicAddress|1125RGg                                             |",
    "|  rpiSerial  |                  7cdf8xxx                          |",
    "|  updatedAt  |              2021-06-18T22:30:18.486Z              |",
    "+-------------+----------------------------------------------------+",
    "",
    "",
    "********************",
    "Manufacturer Info",
    "********************",
    "",
    "+------------------+---------------------------------------------------+",
    "|       name       |                      result                       |",
    "+------------------+---------------------------------------------------+",
    "|     address      |14sKxxx                                            |",
    "|    createdAt     |             2021-02-24T22:16:39.379Z              |",
    "|        id        |                         6                         |",
    "|locationNonceLimit|                         1                         |",
    "|       name       |                      Bobcat                       |",
    "|    updatedAt     |             2021-02-24T22:16:39.379Z              |",
    "+------------------+---------------------------------------------------+",
    "",
    ""
  ],
  "errors": null
}

I told you it was a lot of information!

We can actually see a lot of stuff that was previously not possible to see on your Bobcat like the enrollment key, various internal components and their serial numbers, etc.

Our P2P connection status (what the light represents on your Bobcat, green yellow etc.) is revealed in great detail as well. Look at this detailed breakdown here:

   "p2p_status": [
    "+---------+------+",
    "|  name   |result|",
    "+---------+------+",
    "|connected| yes  |",
    "|dialable | yes  |",
    "|nat_type | none |",
    "| height  |955285|",
    "+---------+------+",

Explorer

This is just a link to the Helium blockchain explorer. Not very useful!

Helium API

Connects to the Helium API and returns some information in JSON format about your specific miner. This is Helium-specific information that is tracked across all miners and not Bobcat specific.

Onboarding

This one is a little more interesting. It redirects you to your hotspot’s onboarding entry at onboarding.dewi.org. The results are given to you in JSON like the other commands.

I had not seen this site before but sure enough my onboarding entry appeared as expected!

Net Speed (Internet Speed Test) (Update 9/3/2021: Removed, Secret Link Available)

Finally a fun one! This one lets you test your Bobcat 300’s internet connection speed.

I ran the connection test (using WiFi) and this was my output:

{
  "DownloadSpeed": "27 Mbit/s",
  "UploadSpeed": "33 Mbit/s",
  "Latency": "16.586062ms"
}

Those results are actually terrible for my connection speed (gigabit fiber) but not unexpected for WiFi using G/N speeds.

At any rate this is very useful information to have whether you’re using wired / wireless. Having a tool like this lets you test the Bobcat’s connection specifically instead of running a general speed test on your computer. That only diagnoses your internet connection in general and not the Bobcat’s connection specifically so having this tool is a giant leap forward and worth using.

It does make your miner slow down / stop mining for about 30 seconds so don’t overdo it!

UPDATE 9/3/2021: In 1.0.3 this option is no longer in the list of options. It’s still available on the miner as speed.json. To access the option replace the IP in the URL below with your own miner’s IP address and go to:

http://192.168.X.X/speed.json

After a 20 second pause or so you will see the speed test (for now, they removed the menu option but the JSON request is still available/permitted):

{
  "DownloadSpeed": "25 Mbit/s",
  "UploadSpeed": "42 Mbit/s",
  "Latency": "9.81852ms"
}

Resync Miner (Update 9/2/2021)

This is the first really serious troubleshooting option we’ve been given for the Bobcat. It was added in 1.0.2 and the description given by hovering your mouse over the option is “It will delete the miner data, let the miner resync to the blockchain from scratch”.

Basically they’re saying this will wipe the local blockchain data from your miner and let it resync from scratch. You would only want to do this if your miner is completely stuck / corrupted.

Do you remember how long it took to sync your Bobcat when you got it? You definitely don’t want to have to go through that again unless your miner is completely broken and won’t sync and nothing else has worked / troubleshooting has led you to this conclusion.

This finally gives users the ability to clear and resync their Bobcat though which is a fantastic improvement from the launch 1.0.0 version!

Also requires a username / password:

Username:  bobcat
Password:  miner

Reboot (Update 9/2/2021)

This was a new option added in version 1.0.2. It’s as simple as it sounds and allows you to reboot the miner through the web GUI. Very handy!

This option will prompt you for a username and password. Here they are:

Username:  bobcat
Password:  miner

Blockchain Height

Blockchain height just gets the current height of the Blockchain from the official Helium API. This would be useful if you wanted to write a very simple script that wanted to compare if your hotspot’s block matched the current block on the blockchain.

A lot of the other available commands give the blockchain_height already (such as “Sync Status” and “Miner”) so it’s not extremely useful.

Bobcat OTA Update / Troubleshooting (Community Res)

Don’t get too excited about these ones. These are not at all what they sound like. They’re both just web links to community resources (at least at 1.0.0 release).

Fast Sync (Alpha) (Update 9/11/2021)

This new feature was added in 1.0.4. It’s meant to help catch you up quickly to the blockchain. I had the chance to use it today as Bobcat has had a lot of issues with hotspots falling out of sync and mine has been impacted.

The function takes about 30 minutes to sync and you should be at least 400 blocks behind to use it. From Bobcat:

Fast Sync

This is an alpha feature that gives healthy miners a sync boost, so they can quickly catch up to the top of the blockchain from the block they are syncing at.

IMPORTANT: this feature will NOT work for healthy miners whose gap to the blockchain is smaller than 400 blocks.

Bobcat Miner 9/10/2021 Post

As usual the credentials will be:

Username:  bobcat 
Password:  miner 

Conclusion

The whole time I’ve had my Bobcat it has been a “black box”. Who knows what it’s doing. You can’t log into it with SSH or using the serial console (set your speed to 1,500,000 baud (yes, 1.5 million) and the serial console “data port” does work over USB in something like Putty, this is some kind of security through obscurity baud rate) because all the accounts are either disabled or have a nologin shell.

There’s only a couple of useful “tools” in 1.0.0 release that actually make the Bobcat do anything but it’s so nice to be able to finally see and be able to do *SOMETHING*! I’d imagine that more menu options will continue to be added to the Bobcat 300’s new “Diagnoser” tool over time.

I definitely at least recommend running the “Speed Test” and checking your unit’s temperature to make sure everything is running well!

Other Resources

For the best place to exchange your Helium (HNT) to other tokens like Bitcoin (BTC) check out my Best Altcoin Mining Pools and Exchanges article

If you have a Nebra hotspot check out my getting root and SSH access to the Nebra guide

To understand what is happening with Ethereum mining and when it is ending see my Why GPU / Ethereum Mining Is Toast – Stop Buying GPUs article

133 thoughts on “Bobcat 300 Diagnoser Tool / Utility Guide (Helium Mining)”

    1. Avatar for James A. Chambers

      Hey tiky111,

      Having a gap means you are behind the blockchain (syncing). The number of the gap represents how many blocks behind you are. The correct gap to have if everything is working is a negative gap (-1) which is what shows for that value when you are fully synced.

      This isn’t normally a big deal as it will just catch up the gap but it depends on how long you’ve had it. Is the gap decreasing in size? If the gap is only increasing in size and doesn’t make any progress then it may be time for a power cycle or possibly to even reset the unit in the Diagnoser. It really depends on the rest of the status as you can get a gap normally just from the network lagging.

      If you want to post some diagnoser output I can probably tell you more specifically what is happening! Unfortunately just having a gap doesn’t really tell us too much as that is honestly pretty normal/routine to get those and it will resolve itself fairly quickly unless something else is wrong. Hopefully that helps!

    1. Avatar for James A. Chambers

      Region unknown? You may want to factory reset this if it will let you get that far. Otherwise you may want to contact the manufacturer. You aren’t trying to use something like a US915 point in the wrong country by chance?

  1. Avatar for D Van Camp

    I’m in need of help!

    I’ve got two Bobcat Miner 300s both of which are apparently very old but were brand new when I bought them last week (11/20/2021). The OTA version is 1.0.1.9 and the firmware is 2019.11.06.0 in one (I can no longer pair with that one and the other won’t allow me to Pair using BT at all). I was syncing for >6 days when the one I am running went Offline (light had been yellow for a couple of days but had been green at one point). I’ve rebooted multiple times to no avail. I can ping it locally and when I login using the IP I am presented with a webpage unlike anything I have seen anywhere (userid is ‘easylinkin’ and the title at the top is ‘g280 | Attitude Adjustment (unknown) | Load: 2.90 2.34 2.17’).

    Can anybody make any suggestions to get even one of these miners going?

    1. Avatar for James A. Chambers

      Hey D Van Camp,

      Oh wow, that is really, really old firmware from back when you could access the OpenWRT (or maybe it was DDWRT) firmware from the web interface. This is really really old from before the Bobcat Diagnoser tool (which is the new web interface) existed. I think back in that version the password is also easylinkin for that account as I ran into that when I was trying to get SSH access to my Bobcat on the current firmware (still a work in progress, I did finally crack the root password for my Goldshell ASIC miners though so I’ll get to the Bobcat eventually).

      Basically your version of the firmware only checks the Bobcat servers once every 24 hours. All you can do with that version is leave them plugged in and wait for them to update the firmware. In newer firmware it checks twice per day (so about once every 12 hours) but on your version I’m certain that is still only once per day. If it isn’t plugged in at the scheduled check time you often miss it and have to wait until it comes up again so I would just leave them both plugged in and online if possible.

      They will update themselves eventually but especially your version takes an agonizingly long time. Hopefully that helps!

      1. Avatar for D Van Camp

        James,

        Thanks so very much for that info. Yes, I automatically tried the easylinkin as the password and got in but have now changed it. I thought that what you brought up was my problem. It’s amazing that someone still had something this old that was unused! I wasn’t smart enough (at all ;-)) to ask any questions and bought two at a premium!

        If leaving them plugged in will help, do you think I could leave both plugged in in the same location just to get this update downloaded? I’ll be gone for a few days and thought it might resolve itself on both of them even though they would impact each other’s performance once they were up and running.

        I really do appreciate this forum as Bobcat support seems nonexistent!

        1. Avatar for D Van Camp

          Oops, I don’t think I can leave both of them online as I can only port forward 44158 to one IP! Oh, well, single thread it is!

        2. Avatar for James A. Chambers

          I wouldn’t worry about that part of it. It’s basically just a timer that runs on the Bobcat to check at a certain day. It’s random for each Bobcat so they don’t all check at the same times and swamp the servers.

          The relayed status will have no impact on the firmware updates whatsoever. It’s totally safe to just plug them both in until they update!

          1. Avatar for D Van Camp

            Thanks for that information! Unfortunately, when I got up this morning the currently running Miner had lost it’s connection to my network and thus the Internet. Not sure what is going on but not happy with my new Netgear cable/modem/router!

  2. Avatar for Tristan Choy

    So my bobcat miner has been relayed and I have no idea why. It was fine before, and on hotspotty it says its fine however now its just relayed….

    Awful really.

    it shows :

    "p2p_status": [
    "+---------+-------+",
    "| name |result |",
    "+---------+-------+",
    "|connected| yes |",
    "|dialable | yes |",
    "|nat_type | none |",
    "| height |1110657|",
    "+---------+-------+",
    "",
    ""

    Which should be fine right?

    My peerbook is like this:

    "| address | name |listen_ad|connecti|nat|last_upda|",
    "+-------------------------------------------+---------------------+---------+--------+---+---------+",
    "|/p2p/blahblahblach|perfect-pineapple-chi| 1 | 8 |non|237.765s |",
    "+-------------------------------------------+---------------------+---------+--------+---+---------+",
    "",
    "+---------------------------+",
    "|listen_addrs (prioritized) |",
    "+---------------------------+",
    "|/ip4/94.4.xx.xxx/tcp/44158|", (the correct IP
    "+---------------------------+",
    "",
    "+-----------------+--------------------+-----------------------------------------+-----------------+",
    "| local | remote | p2p | name |",
    "+-----------------+--------------------+-----------------------------------------+-----------------+",
    "|/ip4/172.17.0.2/t|/ip4/18.141.125.188/|/p2p/11245S2bqZ5n5ZhoCJMMD57kA2ebhxBPcBwN|custom-mango-monk|",
    "|/ip4/172.17.0.2/t|/ip4/18.223.200.123/|/p2p/1128k7YK3UfSh5qaAp37qe2jw3LaG6ycQUAA|tangy-gunmetal-ca|",
    "|/ip4/172.17.0.2/t|/ip4/5.30.200.139/tc|/p2p/112KmjHphM4cjk5wD9JhBPWjqw7TaBWoQrW3|real-heather-trol|",
    "|/ip4/172.17.0.2/t|/ip4/47.148.130.166/|/p2p/112UjBVaxzNzQK13wEKBBTdpFkyJYWAg4mCu|active-eggplant-q|",
    "|/ip4/172.17.0.2/t|/ip4/3.15.234.204/tc|/p2p/112tHmJW3Yb39YvfjPigQWq8DyNpHL13S4ta|glorious-goldenro|",
    "|/ip4/172.17.0.2/t|/ip4/79.100.24.90/tc|/p2p/11TH5xzG49xF5tecMA39PZWTKd6D2yZS6KBz|happy-tangelo-elk|",
    "|/ip4/172.17.0.2/t|/ip4/174.45.112.140/|/p2p/11vNgQpDaiEZ2NAvWKAF4efYt862YPyWonsb|cold-mango-lobste|",
    "+-----------------+--------------------+-----------------------------------------+-----------------+",

    Any ideas on whats going on and why it says relayed on explorer for the last week?

    Thanks in advance <3

    1. Avatar for James A. Chambers

      Hey Tristan,

      That does look correct from your report. The network has been pretty flaky lately with lots of issues/outages for the past couple of weeks. Still, if it has been a week that is quite a while.

      When was the last time you rebooted the miner? Sometimes this can get it going but a week is a pretty long time. Sometimes this can get these status changes through as it will pick a different set of peers to “gossip” your status to.

      Is the Helium explorer telling you it’s relayed or Hotspotty or both? I’d try rebooting it first and giving it a few hours. Your report looks “clean” for sure with a NAT type of “none” and open ports! It’s for sure a strange case though if it has been stuck in relayed for a week with a report looking like that. Definitely let us know how it turns out!

  3. Avatar for Alexander Patt

    Dear all,
    I’ve got a problem with the activation of one of my bobcat miners:
    Error message: “Onboarding key not found this time. Maybe Miner isn’t started”

    The miner is brand new not used yet!

    My OTA is the newest version.
    The miner connected since a couple of weeks and by ethernet (and configured by helium app for ethernet use).

    I almost tried to reboot, reset to factory defaults (by console) nothing works, always the same error.

    The miner 5$ report in the console shows this error: “Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?”

    Joining the helium network is not possible. The helium app shows the same error, that the miner is not started yet.

    I tried to contact the bobcat support several times but always get a standardised answere.

    Is there any solution for this problem?

    Thanks a lot in advance for your reply…

    Best, Alex

    1. Avatar for James A. Chambers

      Hey Alex,

      Welcome! When did you get this miner? I’ve heard of reports of people getting this sometimes for a day or two but most people said it was resolved after that here.

      It’s not totally clear if this is something they need to enter on the backend and they just haven’t done it yet or if there is some kind of delay. When you emailed Bobcat support did you provide them with your hotspot’s information? Sending them a copy/paste from the Bobcat Diagnoser’s “Miner (5s)” option should give them all the keys they need to fix this in the backend.

      You said this was only one of your miners. How many did you get if you don’t mind me asking? This is mostly just out of curiosity as this does seem to happen on some new miners and I’m curious if you got like, 3 miners or 30 miners etc.

      Traditionally when we’ve seen the “Cannot connect to the Docker daemon at unix:///var/run/docker.sock” it has ended up being that the Bobcat’s storage was corrupted and the “Reset Miner” option (which completely wipes the Bobcat’s storage including the Helium software and storage and downloads it from scratch) was the fix for it. It sounds like you have done this several times already so I believe the problem is actually with your onboarding key.

      This may resolve itself if Bobcat is just being slow to get the keys fixed on the Helium onboarding server (I think the onboarding servers have been having some outages as well) but I would send them an email with all of that information and maybe the subject “Miner needs to be onboarded – keys included” or something like that. It has your serial # in the information as well so they can verify it’s a real Bobcat on their end.

      Hopefully that helps and let us know how it turns out / how you resolve this!

  4. Avatar for Chris Fisher

    Hi James,
    Thanks for all the great info in the Q&A. I have a similar problem to the one below. The Bobcat was working fine for a month until the Fast Sync debacle but I got through that and was running fine for a week. Then the miner went offline. I notice in the Bobcat Diagnoser port 44158 was closed even though my Verizon router was saying the port was open. After going back and forth with Verizon, we opened all ports to Bobcat and got the Bobcat Diagnoser to show port 44158 is now “open”. It goes back to port “closed” after a while. Haven’t been able to get the green light. I am seeing the “RPC to ‘miner@127.0.0.1’ failed…as you discussed below. Is this going to require a miner reset? Does resetting the miner affect the HNT already mined or will it just keep the same wallet address? I’m at a loss for why port 44158 is open…then closed. This is what I’m seeing…
    {
    "timestamp": "2021-09-28 17:42:30 +0000 UTC",
    "temp0": "50 °C",
    "temp1": "50 °C",
    "temp_alert": "normal",
    "ota_version": "1.0.2.27",
    "sync_status": "",
    "miner": {
    "State": "running",
    "Status": "Up Less than a second",
    "Names": [
    "/miner"
    ],
    "Image": "quay.io/team-helium/miner:miner-arm64_2021.09.26.0_GA",
    "Created": 1632850384
    },
    "height": [
    "Node 'miner@127.0.0.1' not responding to pings.",
    ""
    ],
    "epoch": "Node",
    "miner_height": "pings.",
    "blockchain_height": "1030702",
    "ports": {
    "44158": "open"
    },
    "region": "NODE 'MINER@127.0.0.1' NOT RESPONDING TO PINGS.",
    "pubkey": "x",
    "animal": "scrawny-ox",
    "p2p_status": [
    "Node 'miner@127.0.0.1' not responding to pings.",
    ""
    ],
    "onboarding": [
    "Error: Usage information not found for the given command",
    "",
    "",
    ""
    ],
    "errors": ""

    Thanks for your help.

    1. Avatar for James A. Chambers

      Hey Chris,

      Thanks for the compliments and welcome! I think you’re definitely on the right track here and have diagnosed this really well and given great information here to work with.

      I think you’re spot on about what is happening here. I may be able to fill in some pieces here of what is going on here to make some more sense of it as well. The miner container is crashing / failing to start here.

      What is strange with yours that I haven’t seen before is that you have a miner_height of:

      "miner_height": "pings.",

      That’s a little bit funny honestly but it’s probably some kind of parsing error from unexpected output where the miner container is returning an error message instead of the output the Bobcat utility was expecting. It’s definitely for sure in bad shape though when unexpected text starts showing up where only numbers are supposed to be!

      Another thing that gives us some confidence in this diagnosis is this:

      "Node 'miner@127.0.0.1' not responding to pings.",

      127.0.0.1 means localhost or yourself basically. This type of connection never goes through your firewall or leaves the device. That tells us the problem is for sure on the device. The device can’t connect to another part of itself (the miner container) and since that type of traffic never leaves the miner we know this for certain and can rule out any networking/firewall for traffic to 127.0.0.1 because it’s a loopback connection to yourself that will never touch the real networRk or be seen by it.

      As far as the port opening and closing that is likely the mining container trying to start and then crashing somewhere in the process. This may terminate the “listening” thread when the process crashes which could explain some of the port behavior as well. This would repeat over and over as it will try to restart the miner container over and over periodically (it has a cooldown period before tries but it does retry) but it doesn’t sound like yours is going to come back up on it’s own if it hasn’t by now.

      It sounds like it has been like this for some time. The last couple of times we saw this both people had completely crashed out status’s just like yours and after doing the reset they immediately got a status back. Syncing was painful but it sounds like if this is been an issue it’s time to do it for sure. I would choose the option to wipe the miner because this is a pretty bad crash when this happens. One of the other people had a bug that completely filled their storage to 100% with failed snapshots so often times when it’s this crashed out it really is best to wipe it clean and let it redownload as something has gone wrong internally in the storage that we don’t want to keep around.

      Hopefully that helps and definitely let us know what you find!

  5. Avatar for Dawid

    Hello James
    Thank you for great article. I have my miner relocated from my friends house to my home and it seems “stuck”… It was working properly (but poorly) before and now I get:
    "| name |result |",
    "+---------+-------+",
    "|connected| no |",
    "|dialable | no |",
    "|nat_type |unknown|",
    "| height |1017880|",
    "+---------+-------+",

    I’ve open up the ports 44158, 443 and 22.
    I can do “Net speed test” directly from diagnoser tool so it seems to be connecting to the internet…
    May it be because i don’t have a public IP and my IP seems to be pointing to my provider located in another city? Or maybe my internet is too slow? (16MBps download)

    1. Avatar for James A. Chambers

      Hey Dawid,

      Great question! That definitely doesn’t look reachable, absolutely. Your initial suspicion is correct. Without a public IP address you can’t be reached or “dialed”. It will always be in relayed status without a public IP without port 44158 forwarded basically.

      This one actually looks worse to me than that though. You actually have “|connected| no |”, in yours which sounds like it’s not connected to the network at all. I take it you are in red status? Can you post your full status (with anything weird like unit serial #s or Helium IDs etc taken out)?

      Can you elaborate on what you mean by not having a public IP / public IP is pointed to a provider in another city? I am guessing by that you mean that you have a shared public IP (meaning that it doesn’t point directly to you, it is shared with others across your ISP and doesn’t go to any 1 particular person). This is called Network Address Translation (NAT) and if this is the case then it is pretty common (and getting more and more common) and we’re on the right track. If I’m misunderstanding/misinterpreting your setup let me know as some of those tiny details can make a difference sometimes.

      Internet speed looks okay at 16MBps, at least the speed test doesn’t seem to explain any of it. These are errors/misconfigurations it looks like to me from your short version status report here. Let’s take a look at the rest of the config. We don’t need the Helium serial # parts and all of those at the bottom, but just the top and middle has some more info we could look at than the quick status report!

      If it’s just in relayed status you can usually get some kind of public IP option from your ISP (has been relatively cheap for others so far that have let me know it worked out for them) but it definitely seems like maybe more than that here. Keep in mind that “public IP” and “static IP” are different things. A “static IP” is a public IP that never changes and they’re usually relatively expensive.

      There is usually a cheaper option to have a “dynamic IP” that does change but is still a public IP assigned to just you. It’s the public IP (one that goes only to your router which then gets forwarded to your miner) that is the important part for getting out of relayed / being reachable. At any rate we’ll see what else we can figure out here!

      1. Avatar for Dawid

        James, thank you for your reply.
        I should have written more about my situation, my bad.
        It has been showing in app/explorer as “offline” (only once – left alone for many hours magically appeared as “syncing” but then it went offline again and since then no luck).
        Yes it seems I’m behind a NAT (some of my friends have bobcats and they work fine behind NAT, but are “relayed”)
        I’ve used “fast sync” a few times and it seemed to work, my miner was gaining on the block count, but after that – just stuck on the same block (this gave me the idea that it actually connects to the blockchain, but maybe I was wrong and the numbers “miner_height” aren’t to be trusted after fast sync?)
        It seems I wrote nonsense about the speed test because 16MBps is my upper limit, the real – bobcat measured – download speed is 7mbit, upload 2mbit, latency 354 ms (I’m using kinda crappy satellite internet for now – I live in a rural, mountainy area). Maybe those are too low for helium blockchain?
        Didn’t want to do a full resync since in a week or two im going to put this miner in a city on 10th floor where its going to stay for a longer time. But since I have it at home right now – just wanted to check what is my reach area with a new 8dBi antenna on the roof (for possibly future miners) – but so far no luck getting it started here.
        Today i’m checking out alternative router at home (to check if it wasn’t source of problems) and at the evening I’m going to get a GSM/LTE modem for a few days to check different internet connection.
        But,
        just now situation has changed…
        the bobcat diagnoser page says “Onboarding key not found this time. maybe Miner isn’t started” on top and the miner report looks like this:
        "temp_alert": "normal",
        "ota_version": "1.0.2.25",
        "sync_status": "",
        "miner": {
        "State": "running",
        "Status": "Up About an hour",
        "Names": [
        "/miner"],
        "Image": "quay.io/team-helium/miner:miner-arm64_2021.09.16.1",
        "Created": 1632399116 },
        "height": [
        "26468 1022036",
        ""
        "epoch": "26468",
        "miner_height": "1022036",
        "blockchain_height": "1023160",
        "ports": {
        "44158": "open" },
        "region": "EU868",
        "pubkey": "",
        "animal": "",
        "p2p_status": [
        "RPC to 'miner@127.0.0.1' failed: timeout",
        "" ],
        "onboarding": [
        "RPC to 'miner@127.0.0.1' failed: {'EXIT',",
        " {{badmatch,{error,too_many_retries}},",
        " [{miner_keys,keys,1,",
        " [{file,\"miner_keys.erl\"},{line,87}]},",
        " {miner_cli_info,info_onboarding,3,",
        " [{file,\"miner_cli_info.erl\"},{line,268}]},",
        " {clique_command,run,1,",
        " [{file,\"clique_command.erl\"},{line,87}]},",
        " {blockchain_console,command,1,",
        " [{file,\"blockchain_console.erl\"},",
        " {line,14}]}]}}",
        ""
        ],
        "errors": ""

        Looks like that last “fast sync” massacred my boy. What do you think? Resync or reset?

        1. Avatar for James A. Chambers

          Whoa! Yeah that was what I was afraid of. You diagnosed this pretty well. Let’s take a quick look:

          “RPC to ‘miner@127.0.0.1’ failed: {‘EXIT’,”,
          ” {{badmatch,{error,too_many_retries}},”,
          ” [{miner_keys,keys,1,”,
          ” [{file,\”miner_keys.erl\”},{line,87}]},”,
          ” {miner_cli_info,info_onboarding,3,”,
          ” [{file,\”miner_cli_info.erl\”},{line,268}]},”,
          ” {clique_command,run,1,”,
          ” [{file,\”clique_command.erl\”},{line,87}]},”,
          ” {blockchain_console,command,1,”,
          ” [{file,\”blockchain_console.erl\”},”,
          ” {line,14}]}]}}

          Yikes. Basically that is the Bobcat failing to connect to itself (127.0.0.1). This won’t happen unless it’s super, super dead. We have seen something similar once before and it was from the fast sync killing it. Their error was a little different but the status is crashed out even slightly worse than that one. This one is missing the onboarding keys is what it’s trying to tell us (the encryption keys that secure the miner and tell Helium it’s allowed to be on the Helium network).

          The funny thing is it really isn’t too far behind on syncing. It’s doing fine on syncing somehow (guess it doesn’t need an onboarding key to sync). It’s 1,124 blocks behind in this screenshot which with an onboarding key it seems like you would still catch up pretty fast. I don’t even think you’re going to be relayed. Your NAT status and ports are reading open now in this report. You just don’t have an onboarding key so the miner can’t actually do anything other than try to sync. It looks like it’s trying to fix itself but that message says it has reached the maximum number of retries to match the key. Most likely your copy of the key is corrupted/missing on the miner.

          This is another one of those cases where yes I would indeed do the full reset on this one (the one that says it will wipe your miner and set it up from scratch, that is indeed what we want in this situation because we want the miner to be onboarded again so it gets new onboarding keys). This should onboard your miner and get the keys set up properly on the miner again. I do not believe you have a choice here but honestly other than the onboarding everything is looking pretty good. From what I’ve seen you won’t even be in relayed status once you fix the onboarding from that latest report. I think when you do the full reset it will take a while to sync but then you’ll be good to go once and for all

          1. Avatar for Dawid

            geez, I was afraid that “reset miner” would wipe out everything from miners memory and that I would have to wait another few days for it update blockchain, yet just after an hour after resetting my status looks like this:
            "State": "running",

            "miner_height": "1012400",
            "blockchain_height": "1023761",
            "ports": {
            "44158": "open"

            "p2p_status": [
            "+---------+---------+",
            "| name | result |",
            "+---------+---------+",
            "|connected| yes |",
            "|dialable | yes |",
            "|nat_type |symmetric|",
            "| height | 1012400 |",

            It is a few thousand blocks earlier than before (like fast sync never happened), but man, it looks pormising.

            Thank you for your patience and support

            1. Avatar for James A. Chambers

              Hey Dawid,

              Wow, that is the fastest I’ve ever heard of. I think you’re probably good to go as that’s looking fantastic! I think you will be out of relayed as well like we were hoping. Everything’s looking good!

Leave a Comment

Your email address will not be published. Required fields are marked *

Type here..

Exit mobile version