At about 1:30am Pacific all blockchain_etl runners crashed at about block 1088130. While the API was still accepting transaction the backing ETL was not there to submit them and update their status for the API.
No submitted transactions were lost during this outage.
A fix was applied to both the StakeJoy and Helium APIs at about 4am Pacific.
Folks running blockchain_etl will want to install release https://github.com/helium/blockchain-etl/releases/tag/1.1.121 to recover. The release notes include instructions in case many restarts have caused the ledger to run away from the database.
The Helium and StakeJoy ETL/APIs are catching up with the blockchain and pending transaction status will start updating again in the next 60 minutes or so as it gets closer to the head of the chain again.
Comments