TL;DR: A low-probability scenario caused a temporary slowdown of the network. This is a known possibility and intended behavior of the Tenderbake consensus algorithm.
On January 1st, around 14:13 UTC, the Tezos Mainnet experienced a major slowdown: ~52 minutes elapsed between block levels #3,019,850 and #3,019,852, due to block #3,019,851 requiring 18 rounds to reach consensus. Network endorsing power recovered immediately on the following block, producing a slower round 0 block. After #3,019,853, block time stabilized to the expected 30 second time between blocks.
Following the incident, we’ve examined how this slowdown came to happen, and why the network failed to reach a quorum on earlier rounds.
Today, we are confident that no bug in either the Tenderbake consensus algorithm or its implementation by the Octez baker was behind this incident. The results of our investigation point towards a particularly unlikely, yet known, scenario.
Below, we will detail how it unfolded. But first, a short summary:
-
Block production did not stop, Tenderbake kept running: it took 43 minutes for the network to agree on the next blockchain head, as a result of #3,019,851 being finalized at round 17. But, proposals were made in all previous rounds at that level. Remember that Tenderbake entails a trade off in choosing network safety over network liveness.
-
Network was perfectly safe: the behavior observed fits within a known slow consensus scenario for Tenderbake: the endorsing stake was split roughly even between bakers which had locked their endorsement for the Round 0 block and those who did not, without reaching the 67% endorsing power requirement.
-
Liveness was slow, but kicking: it took 17 extra rounds for one of the locked bakers to re-propose the locked payload. It immediately got sufficient endorsements for the chain to move on.
-
A very late block proposal was the trigger: the scenario was triggered by a block proposal arriving unusually late within the expected time slot. The reasons for this delay are still under investigation. However, it was less pressing than assessing whether the Octez baker implemented Tenderbake correctly.
-
An unlikely yet known (and tested!) scenario: the particulars of this slow consensus scenario (detailed below) make it a quite unfortunate incident. Moreover, similar scenario were already thoroughly tested both in the Octez baker’s test suite and using Functori’s Mitten simulation toolkit.
-
Not related to baker crashes: we had first observed reports of baking nodes crashing due to a lack of available storage space. This working hypothesis, mentioned in our early messages on Sunday, turned out to be a false lead. It is not, a priori, connected to the slow consensus scenario that unfolded.
So, what happened?
In focus: the consensus committee for #3,019,851
To understand what happened, we dug into the committee for #3,019,851, and analyzed its interactions before it reached a quorum. We relied on Teztale, a live consensus-introspection tool, currently being developed at Nomadic Labs. We started building Teztale to inform our Incident Response Team in situations like this.
As an aside, the description that follows is fairly technical and requires some understanding of how Tenderbake works. If you are not familiar with the concepts or the terminology, we highlight a few resources:
-
The Lima protocol’s Consensus documentation entry.
-
A look ahead to Tenderbake (blog post).
-
Tenderbake’s Baker as a StateMachine (blog post).
-
A Solution to Dynamic Repeated Consensus for Blockchains (research article).
Round 0
Let’s plunge into the consensus committee for #3,019,851. The histogram above tracks the reception of the block proposal at round 0, and its associated preendorsements and endorsements. Here, the X axis tracks the elapsed time since round rights were enabled, and the Y axis plots the number of delegates seen (pre)endorsing this proposal.
The rights to propose a block at round 0 at level #3,019,851 were
enabled at 2023-01-01T14:14:29Z – that corresponds to the origin of
the plot. The baker holding those rights proposed a payload, whose
block hash was BLZL7AtZKP21eQKEESymTCJYvfV5fpuh6ZZQXffTwg4QstzfxoD.
However, this block proposal was not seen (and validated!) by our node
running Teztale, or any of the nodes we have access to, until
14:14:54Z – denoted by the red vertical bar in the histogram. That
is, the block payload could only be (pre)endorsed 25 seconds after
rights were enabled. This is awkward, as most bakers usually produce
and propagate blocks significantly faster than this.
Given a 30 seconds round duration, Round 0 was scheduled to finish at
14:14:59Z. This left only 6 seconds for the network to reach
consensus on this block. It was clearly not the case: preendorsements
and endorsements for this round arrived too late to our node, as the
histogram shows.
Let’s focus on the (pre)endorsements in this round. In this particular committee, 229 bakers (out of the 240 that had endorsing rights) managed to inject a preendorsement within this period. Then, 122 bakers (accounting for 43.83% of the endorsing power) observed in time that a pre-quorum was reached, and proceeded to endorse that block – witnessed by the violet bars in the histogram.
This action “locked” them to the endorsed payload content for all successor rounds at level #3,019,851. This means they would only endorse re-proposals of the original block payload.
Given that 43,8% of endorsing power is obviously more than 33% but far from the required 67%, the rules of Tenderbake specify that the only way to move forward in this scenario is when one of the “locked” bakers re-proposes this round proposal’s payload content.
That is, when it is their turn to do so.
Rounds 1 – 16
Here is when Fate kicks in. Unfortunately, all bakers with proposing rights between Rounds 1 and 16 were part of the 104 bakers (amounting to the 56.17% of the stake) that had not locked to the payload of round 0!
The probability of arriving at this scenario, given the stake distribution in play, was 0.5617^16 = 0.0098191%. That is, in roughly 1 out of 10000 possible committee configurations, the right to propose a block in the next 16 rounds would be given to a baker which had not locked its commitment to the payload proposed at Round 0. And this was the case.
Indeed, in every round between Round 1 and Round 16, the designated delegate proposed a block. Yet, neither of them were locked, so they proposed a fresh proposal with a new payload instead.
They were all doomed to fail: their proposals would never be endorsed by the locked delegates and it was not possible to reach the minimal endorsement requirements without them.
The network was both safe and live. Yet, the dice had rolled a bad hand, and there was regrettably nothing to do but wait for fairer winds.
Round 17 and beyond
Finally, it was the turn for
tz1gUNyn3hmnEWqkusWPzxRaon1cs7ndWh7h
to propose a block at Round 17. This delegate was one of the delegates
locked on the payload proposed at Round 0, and correctly re-proposed
it. The resulting block proposal,
BLfpYc4HvRpyacP6f75hpTgRJjzzpfyQRaxzNQys5gByEdr6T24, was produced
with timestamp 2023-01-01T14:56:59Z, as expected.
Most delegates in the committee endorsed the block proposal in time, as seen in the histogram above, resulting in 6,982 out 7,000 endorsed slots. Having reached consensus on the head of the chain, it became #3,019,851 and the network finally moved on.
Note that the next block #3,019,852 was produced at Round 0. Still, the “time between blocks” #3,019,852 and #3,019,851 is around 5 minutes because the timestamp of the former corresponds to the end of the last round of the latter, namely the end of Round 17. Its successor, another Round 0 block, stabilized time between blocks to 30 seconds.
Lessons and take-aways
The foremost conclusion is that what we experienced is just Tenderbake at work.
We are relieved to see the implementation of Tenderbake working as expected in a slow consensus scenario. Still, we regret having experienced the worst slowdown in block propagation since the adoption of Tenderbake on Tezos Mainnet:
-
#2244609, the first block of the Tenderbake era required 15 rounds to reach consensus, as some delegates needed time to adapt to the migration.
-
#2244610, the second block in the Tenderbake era, required 13 rounds.
-
#2490369, the first block of Jakarta, required 12 rounds – due to an expected longer migration due to patching contracts during stitching.
Looking ahead, we consider possible optimizations targeting scenarios like the one described above. For instance, Tenderbake could be less aggressive when increasing round duration. Another possibility is to allow bakers which are not locked to still consider late preendorsements for older payloads, instead of just discarding them. This would allow them to re-propose a block consistent with the potentially locked payload, leading to faster agreement.
When we first implemented Tenderbake, we gave more priority to the safety of the implementation over heuristics to improve liveness. Today, as we propose to further reduce the time between blocks, these optimizations become more relevant, and we look to implement them in the near future.