Update: A quick update on our investigation into Seed Nodes. We've seen some significant improvements in Witness counts over the last day or so and have been making incremental progress (including deploying an upgrade to configuration last night). We are seeing improved stability, but think there is still more to be done here, so will continue to monitor and improve.
The team is spending the morning investigating what they suspect is the cause of witness rates being lower than they should be. We have no updates on the root cause at this time but it is a known issue and we are focused on making sure the seed nodes can scale to meet the needs given the size of the network.
The real solution here is Light Hotspots but we'll have to take a few days to fix this now before going back to that crucial work. We restarted seed nodes yesterday thinking it was a simple fix of local disks getting too full but have noticed that witness rates are getting close to the prior depressed quantity of witness counts.
For context, when a Hotspot participates in Proof of Coverage and notices a neighbor beaconing, they need to submit their Witness report to the Challenger. In order to find the Challenger on 620k+ node network, they often need to use the seed nodes (think global address book like DNS) to translate the Challenger's address to an IP and port. These seed nodes (40 of them deployed around the world) are operated by Helium, Inc., on behalf of all Hotspot, Validator, Router, ETL, and Node operators. They are the first entities on the network that each node connects to in order to find out about the other nodes they could connect to and are an important glue for the network.
For visibility, we monitor these dashboards:
More updates to come as we continue to investigate.
Source: Official Helium Community Discord