Low PoC Receipt Rate
As reported in the community, Helium are investigating the fact that Proof of Coverage Witness activity has gone down quite a bit. In the past this has been associated with the network’s seed nodes being overloaded as the number of hotspots continues to grow. They are looking into the seed nodes and other potential culprits of this issue and will have an update in a few hours. They are hoping a small bit of firefighting here will help us get through the next few weeks when seed nodes are less important to Light Hotspots and their ability to deliver witness receipts directly to Validators.
Updates from the Official Helium Discord Community below:
Yesterday at 4:47 PM
Things appear to be improving as we have done a rolling restart of all the seed node servers (there are many of them spread across quite a few datacenters). We’re still investigating the root cause and whether or not it makes sense to add on more seeds over the next few weeks as we get closer to Light Hotspots.
Yesterday at 5:53 PM
We’re aware that things aren’t completely back to normal. The investigation continues.
Yesterday at 8:20 PM
Just a short update here. We've just added 6 new seed nodes and are looking to add several more if needed.
Yesterday at 11:36 PM
We are bringing 8 more seed nodes online within the hour..
April 6, 2022
Today at 1:49 AM Another update on how things are going and what we're doing next. We've increased the seed pool by 14 nodes and it doesn't appear to have done much to support the network's needs.
What we're doing next is going to add 8 more nodes to bring us to a total of 64 seed nodes. We're also going to reduce the number of connections each node needs to support. This is maybe a bit counter-intuitive but our expectation is that a greater number of nodes that serve fewer Hotspots each will allow the network to resume normal operations. (Rough math: 64 x 15000 connections serves 960k Hotspots and other chain following nodes).
Today at 1:49 AM
Also, for folks who are unclear what Seed Nodes are and their role in the network. Seed Nodes give Hotspots routing information to the rest of the network (other Hotspots, Validators, etc.) by sharing their "address book" of other nodes on the network. Think of them like DNS nodes for libp2p. For example, if your Hotspot is trying to submit a witness receipt and it needs to dial a Challenger it doesn't know about, it asks a Seed Node. They're important for us until Light Hotspots are launched and they're operated by the Nova Labs team on behalf of the network.
Today at 3:59 AM Things appear to have stabilized with our changes but we will continue to monitor overnight. Most of the team is looking at 12 hours of firefighting so they’re going to take a few hours of rest here. We will evaluate any further changes in the morning.