"A wizard is never late..."

The same applies to my witness updates. This is my weekly monthly regularly scheduled irregular one. My last post was four months ago, and my last proper witness update announced an upcoming HardFork - so clearly the schedule is working as designed. ;-)

Did you notice Hive having hiccups on July 9th and 10th? No? Good. That means the chain did its job even while having a genuinely bad day. A few witnesses noticed a lot, some node operators stared at very confusing logs, and I got a good excuse to make a post.

What happened

On Thursday, July 9th, at 18:54 UTC, the network had a brief hiccup - the kind where a block arrives a moment too late and, for a few seconds, the network sees two competing head blocks. Micro-forks like this happen routinely and normally end quietly, within seconds.

This one did not end quietly.

Some context first. Every hived keeps a fork database - a small in-memory set of recent blocks it might still need for switching between competing forks. Its depth is simple: head block minus last irreversible block, plus one. Thanks to one-block irreversibility working as well as it does, that is usually two blocks, often just one. OBI became a victim of its own success here: the healthier the chain, the smaller the window.

Now the bug. When a peer offered a block older than that window, hived did not treat it as "old news, ignore". It treated it as invalid - the same verdict reserved for forged signatures - and applied the standard punishment: disconnect the sender, plus every other peer that advertised the same block. At a fork moment, "peers advertising the same recent block" describes most of your peer list.

And here is the best part: the rejected blocks were not even fork leftovers. On one of the seed nodes we examined, every single rejected block was byte-identical to a block already sitting on the irreversible chain. Perfectly honest, just terribly late: one peer had fallen behind and kept re-relaying blocks that everyone already had - in half an hour, that added up to roughly 12,000 disconnects.

Peer counts collapsed to zero, over and over. Disconnected nodes fell behind, reconnected, innocently offered their now-stale head block, and got kicked again. Each storm fed itself for about half an hour until nodes fell back to sync mode and reconverged on the majority chain - the protocol healed itself while many operators were still typing grep ;-)

One more bug made it worse. The goodbye message a node sends when it disconnects a peer embedded the entire rejected block as the reason. A message that large could not be decoded by the receiving node, so the real disconnect reasons stayed invisible in the logs - one node counted over 6k undecodable goodbye messages.

The storm came back twice the next day, triggered by new hiccups:

storm 1   2026-07-09  18:54-19:25 UTC   worst minute: 9 of 20 blocks
storm 2   2026-07-10  05:21-05:46 UTC   worst minute: 10 of 20
storm 3   2026-07-10  12:34-13:15 UTC   632 of 900 slots produced

How bad was it, really?

Time for data - all of it straight from the chain via HAF. Helpfully, the chain keeps its own attendance list: a producer_missed virtual operation records every missed slot together with the witness that was scheduled for it.

Transparency FTW!

A normal day on Hive has 28,800 three-second block slots, of which the network typically misses 60 to 160 (a loss of 0.2-0.6%, usually near the low end). July 9th missed 230 slots. July 10th missed 557, and even that worst day still delivered 98.1% of its blocks on time. The two days together missed 787 slots; a normal two days lose about 130.

What does a missed block actually mean for you? Not lost transactions. Pending transactions simply wait and get included in the next produced block, a few seconds later. The longest gap between consecutive blocks during the whole event was 15 seconds. The witness participation rate - a moving 128-slot window that every node tracks - briefly dipped to about 61%; normally it hovers near 100%.

The storms did not treat everyone equally. Nine of the top 20 witnesses missed between 57 and 89 blocks each over the two days - about 2-3% of their scheduled slots. The other eleven missed ten blocks or fewer each. Where a node sat in the p2p topology mattered, and so did the node itself: faster machines with better connectivity reconnected and caught up sooner. The witness schedule does not care about any of that - it keeps assigning slots to a witness that temporarily cannot hear the network, and each such slot becomes a three-second hole in the chain.

The network stayed alive the whole time: 494k transactions processed on July 9th and 490k on July 10th, both well within the range of the surrounding days.

The API layer had a rougher ride. HAF, the Hive Application Framework most Hive API nodes are built on, keeps its own record of reversible blocks and replays fork switches into the database - and all that forking exposed a latent bug exactly there: after a fork switch, irreversibility could advance past a block missing from HAF's tables, at which point hived shuts down cleanly, by design. Several HAF nodes went down this way - five during the July 10th midday storm alone - and needed a restart. So if your favorite API node suddenly stopped responding and you had to switch to another one, that was why. The bug is haf#333, reported by @mahdiyari, fixed within a day in haf!973.

The fix: v1.28.7

Released the next day, Friday, July 10th. Two changes:

• A block below the fork database window is now treated as what it almost always is: an honest block from someone slightly behind. It is classified as unlinkable and goes through the normal reconciliation path (re-sync with that peer) instead of triggering mass punishment. With this change, the July 9th hiccup would have resolved quickly, like every other micro-fork does.
• The goodbye message now carries only the block number and id instead of the whole block, so it always stays readable - and a node that receives an unreadable goodbye no longer trips over it.

Genuinely invalid blocks - bad signature, failed validation - still get the punitive treatment.

The network healed itself each time, but the code needed humans. Huge thanks to Eric, who diagnosed the mechanism, authored the fixes, and shipped the release the same day, and to @blocktrades, @abit, and all the witnesses who spent those two days reading p2p logs, testing patches on live nodes, and upgrading.

Details for the curious: hive!2064 and hive!2065, plus the HAF fix linked above. No hardfork, no change to consensus rules - p2p and node-local behavior only.

Upgrade, please

• Running a witness or a seed node? Please upgrade to v1.28.7 soon. The fix works like herd immunity: a patched node stops kicking its honest neighbors, but unpatched neighbors can still kick you. Every upgraded node makes the next storm smaller; enough of them, and there is no storm at all.
• Running a HAF API node? HAF 1.28.7 carries the same hived plus the fork-handling fix.
• Everyone else - regular users, exchanges: nothing urgent. Upgrade at your own pace.

As I write this, 10 of the top 20 witnesses run v1.28.7, one is on 1.28.6, and 9 are still on 1.28.3. In the top 100, 66 are still on 1.28.3. I understand why - 1.28.3 served us well for months, and moving off it is not free. But this bug stays invisible until the network has a bad day, and we just had two of those.

The practical part: coming from 1.28.6 the upgrade is a drop-in binary swap - stop, replace, start. Coming from 1.28.3, which statistically speaking means you, it requires a replay. That is exactly what the downloads below are for: the exchange snapshot gives you a jump start (despite the name it works fine for witness and seed nodes too - just comment out the plugins you do not need).

Downloads and resources

As usual, https://gtg.openhive.network has you covered:

• hived and cli_wallet v1.28.7 binaries, built on release day
• exchange node snapshot: hive-1.28.7-20260711-exchange.tgz (6G) with an example config
• block_log and block_log.artifacts, if you prefer to start from the beginning (of time), but I don't recommend that. In the vast majority of cases you already have a block_log from a previous run, and even if you don't, syncing from scratch would likely be almost as effective.

Official sources:

• GitLab: https://gitlab.syncad.com/hive/hive/-/tags/1.28.7
• Docker Hub: hiveio/hive:1.28.7

My witness node and the seed node at seed.openhive.network:2001 (one of hived's default seed nodes) were running these fixes even before v1.28.7 was tagged - reviewing patches has its perks.

A word about uptime

Cloud providers usually define their service levels per calendar month, measured in minutes of downtime per month.

Let's be maximally unfair to Hive and count every missed block slot as downtime. The last 30 days, including the two worst days we have had in a long time: 99.69%. Counting it the fair way - was the chain accepting and confirming transactions? - there was no downtime at all. The worst thing that happened to a transaction was waiting an extra dozen seconds.

For scale, from the world of centralized services:

• In 2018, a 43-second network partition at GitHub turned into 24 hours and 11 minutes of degraded service (their post-mortem is a great read). Microsoft completed its takeover of GitHub five days later - make of that what you will. These days you cannot even check how they are doing: GitHub stopped publishing aggregate uptime numbers on its status page, so a community project reconstructs them from the incident feed instead.
• In 2021, a configuration change took Facebook, Instagram and WhatsApp off the internet - DNS and all - for about five and a half hours.

Hive's grand total for both storm days: 787 skipped block slots - the transactions meant for them simply confirmed a few seconds later - and every storm burning itself out in about half an hour with no human intervention. Our attendance list lives on-chain, where nobody can stop publishing it. I am not saying we are better. I am just leaving the numbers here ;-)

Governance

Your governance votes expire when unused for a year, and witnesses only hold the position for as long as you keep them there. Take a moment, review your witness and proposal votes, and adjust them. Preferably toward witnesses who keep their nodes up to date.

Trivia

• Block 108,000,000 was produced on July 10th at 02:36 UTC by @therealwolf - right between storm one and storm two. A perfectly round number, and everyone was too busy watching peer counts to celebrate it.
• Out of my 2,741 scheduled slots across those two days I missed exactly one, and I cannot even blame the storm. That was me going "oh, let me quickly try if... Oh."

One block at a time. Ideally, all that were scheduled.