TL;DR: Based on meticulous testing, Nomadic Labs and Trilitech propose reducing Tezos’ Layer 1 block time from 15 to 10 seconds. This blog post walks you through the test setup and results, including a hardware recommendation for bakers.
(EDITED 27-03-2024 to update minimal recommended hardware specs)
Block time is an important and highly visible parameter for a blockchain.
It determines how often incoming transactions are added to the chain, and on Tezos it impacts the user experience in two noticeable ways: latency and finality.
Latency is how quickly a new transaction is first applied by the network. The shorter the block time, the shorter the wait before moving on with what you were doing, and the smoother the user experience.
Finality is the time it takes for a transaction to be considered irreversible. The shorter the time to finality, the smaller the window for malicious actors to attempt double-spending or other fraudulent activities. Faster finality makes a blockchain more reliable and useful for mission critical and high-value transactions.
Tezos’ ‘Tenderbake’ consensus algorithm achieves finality after two blocks, and the current block time of 15 seconds hence equals a time to finality of around 30 seconds. In comparison, the time to finality on Ethereum is ~15 minutes (64+ blocks), and on Solana it’s 12 seconds (32 blocks).
10 second blocks: secure, stable, decentralized
Tezos launched with a Layer 1 block time of 60 seconds in 2018, and developer teams have since worked on optimizing the network to enable shorter block times. In 2021 block time was reduced to 30 seconds, and in 2023 it was further reduced to the current 15 seconds.
In an upcoming “P” protocol proposal, we include a reduction to 10 seconds, enabling a time to finality of 20 seconds.
Why not just step on the pedal and lower block time to, say, 1 second?
As with most parameters – block size being another example – it’s not quite that simple. Each blockchain network works differently, but they share common challenges in that they must balance speed with other factors, notably:
- Security. Transactions are sufficiently verified before being accepted by the network.
- Stability. The blockchain runs reliably, with outages close to non-existent. Also called liveness.
- Decentralization. Broad distribution of validators prevents colluding actors from selectively censoring transactions.
While most blockchains more or less align on the first two, decentralization is often treated differently by various blockchains with diverging philosophies and assumptions.
To achieve higher speeds, some blockchains define a limited set of validators that secure the network. Others have high requirements in terms of hardware, internet connection, and minimum stake, effectively creating an economic barrier.
The Tezos ecosystem prioritizes open participation and a low barrier of entry for bakers (validators). The goal is to have bakers of different geographical regions and economic abilities participate in securing the network, which encourages decentralization and makes the network more inclusive.
Recommended hardware
With recent optimizations of the network, we are confident that a block time of 10 seconds will not harm security, stability or decentralization.
The conclusion is based on thorough testing, performed with strict performance requirements, high load on the network (full blocks), and with added delays simulating Ledger Nano S signing and unstable network conditions.
In fact, the test was performed successfully with a block time of 8 seconds, and 10 seconds was chosen to provide extra safety margin. For a deep dive into our testing methodology and the optimizations that made the reduction possible, see sections below.
Based on our results, we recommend the following minimum hardware specifications, which are viable for all and performant enough to bake at the target block time.
- 3 CPU cores
- 8GB RAM + 8GB swap (or 16GB RAM)1
- 100GB SSD storage (or similar I/O performance)
- A low latency reliable internet connection
As the work to reduce latency and time to finality on the Tezos network continues, we expect further reductions in Layer 1 block time, but low hardware requirements remains a goal.
And now, let’s dive into the details behind the reduction to 10 seconds. Heads up, it’s going to get technical.
The road to 10 seconds
We start with a brief history of Tezos network optimizations and block time reductions.
In 2021, a reduction from 60 to 30 seconds was made possible with tweaks to the consensus algorithm (Emmy*) in protocol Granada.
In 2022, protocol Ithaca replaced Emmy* with Tenderbake, a brand new consensus algorithm with deterministic finality, which opened the door to further optimizations. Ithaca also introduced a ‘light check’ validation of manager operations (such as transfers and contract calls) allowing nodes to propagate blocks faster through the network. It also introduced mempool optimizations. The light check approach was later improved in protocol Kathmandu, as part of the pipelining project.
In 2023, protocol Lima extended the light check concept to other types of operations (consensus, voting, etc.). Building on these advancements, the block time was reduced to 15 seconds in the Mumbai protocol.
More recently, in the Nairobi protocol, consensus operations propagation was again improved to further accelerate the consensus process.
At this point, identifying the obstacles for further block time reduction was not straightforward, and we deemed it necessary to improve our methodology with reproducible large-scale experiments.
Below, we cover the setup, the results, identified obstacles and improvements, and next steps.
Methodology
The purpose of our experiments is to examine whether a given block time is “safe”, and if not, identify bottlenecks. We define a safe block time as one where network performance lives up to these criteria:
- at least 99% of blocks are baked at round 0
- no blocks are baked with a round higher than 1
Framework
To get meaningful data about the consequences of a reduced block time in our simulations, we need to mimic Tezos Mainnet as closely as possible and load the chain with heavy traffic.
For this we have developed the framework Tzimulator, which is capable of executing large-scale experiments within a controlled environment.
The tool uses real-world data in the form of historical operations from Tezos Mainnet. It takes a snapshot of the chain as a starting point, and executes the subsequent two weeks of operations on top of it.
- A cluster of Tezos nodes is started
- The snapshot from Tezos Mainnet is imported
- Using the historical operations, an injector node keeps a high level of proposed operations in the mempool, ensuring full blocks (at the gas limit) throughout the test
- The test is executed for a specified number of operations or for a specified time.
Architecture
The Tzimulator framework is built on top of Kubernetes (also known as K8s), an open-source system for automating deployment of containerized applications.
Thanks to this highly automated framework, we can efficiently run different realistic simulations while tweaking various parameters. For testing block times, we have also run simulations on different hardware architectures.
All the experiments use machines with:
- 6.5Gb of RAM memory,
- 100Gb persistent SSD storage,
- Gigabit network with simulated network delays
Meanwhile, the CPUs have been configured differently to reflect different hardware requirements. The experiment was carried out with 4 CPU cores of the following architectures:
- n1-standard-4 (2Ghz),
- n2-standard-4 (2.6Ghz)
In addition to the above parameters we can adjust the following:
- Signing delay: Simulates signing times similar to slow hardware devices such as Ledger Nano S, remote signers, etc.
- Network delay: Simulates network congestions, with ad-hoc delay ranging from 20ms to 150ms.
- Mempool load: Defines target number of operations in the mempool at all times.
- Block time: The targeted time between blocks. Various block times were tested (15s, 10s, 8s, 5s) with variations of the above parameters and different versions of Octez to determine the safety of each configuration.
We can also configure which data are collected for a given simulation run. For every node and baker we can recover standard Octez logs (daily-log). For a comprehensive overview and analysis of node behavior, Octez Metrics can be enabled to feed a Prometheus database with a Grafazos dashboard. Finally, our consensus inspection tool Teztale can be used for analyzing consensus performance.
Differences with Mainnet
Presently, the framework diverges from Mainnet in a few ways.
Node and baker configurations are uniform, with all instances using identical hardware, signing hardware, and docker images.
The framework also uses a reduced number of bakers (up to 250) and nodes (up to 250) compared to Mainnet’s larger count (around 400 bakers and 5000 nodes). All nodes operate within the same cluster, and network delay is instead simulated using a custom ad-hoc solution.
Regarding the network topology, the framework has a simpler setup compared to the Mainnet network. It uses the default P2P configuration, with each node connected to ~50 other nodes. The resulting graph for 250 nodes is not strongly connected, and some hops are needed for operations from the injection to reaching the farthest node.
The workflow: Identifying and fixing bottlenecks
Pinpointing bottlenecks and implementing solutions was carried out as an iterative process:
- Choose block time and other parameters.
- Conduct an experiment with the network under full load.
- Analyze data collected from an internally developed profiler, Octez Metrics, and Teztale.
- Identify potential bottlenecks based on analysis of the collected data.
- Develop an optimization and conduct additional experiments using the same parameters and the newly implemented improvement.
- If the optimization demonstrates improved results, merge it into the Tezos codebase, and re-initiate the process.
The baseline
We started out with a baseline experiment: a network configured with the parameters below, notably a blocktime of 15 seconds which is equivalent to current Mainnet.
- Node version: Octez v18.1
- Block time: 15 seconds
- Simulated signing delay: 1 second
- Random network delay: 20-150ms
- Number of nodes (each with an associated baker): 250
- Target mempool operations: 2500
- Hardware:
- 6.5Gb RAM
- 4 CPU cores
- n1-standard-4 machine with Intel Broadwell CPU platform
- PersistentSSD disk type
The summary report for the experiment shows that out of the 363 levels, only one level was found to be at round 1 where the rest were at round 0.
We observed that quorum – 66% attestation required for completing a block level – was reached in just over 11 seconds on average. Average time for block application (validation time + application time) was 6 seconds.
Average time elapsed since block timestamp, high network load (full blocks) | |
Stages of a round | Baseline results |
Block validated: ready for consensus | 4.51s |
Block fully applied: new chain head | 6.02s |
Pre-quorum reached: first consensus vote complete | 7.47s |
Quorum reached: second and final consensus vote complete | 11.14s |
The graph below (‘Attestation Reception Delay’ from Teztale’s Level page) shows the result from a single, random block level during the experiment. It aligns with the summary, but also reveals that the (pre-)attestation receptions have a noticeable difference between the first and last occurrences.
Overall, our observations confirmed that the network was safe with a 15 second block time. On average, quorum was reached after 11.14 seconds, with a maximum time of 12.46 seconds.
However it also confirmed that improvements would be required in order to lower block time to 10 seconds.
Implemented improvements
Providing a stepwise walkthrough of every performed experiment and subsequent implementation of improvements would make this blog post_ very_ long.
Rather, we present an overview of all improvements implemented through iterations of the process described above. We also present the results of an experiment run with all improvements implemented.
The improvements consist of changes to the Octez baker and the Octez node’s mempool handling. A good example of an improvement with a significant impact is the pre-emptive forging implemented into the Octez baker.
Previously, the baker proposing the block after the current one would wait until the end of the current level to begin the process. But that is not necessary.
Proposing a block can be split into three parts: forging, signing and injecting. Forging and signing are the most time consuming parts, but they can be started before the block level has started.
Most blocks have “idle” waiting time for bakers after quorum is reached, and pre-emptive forging lets the next baker begin forging the next block during this time, so it is ready for injection at the start of the new level. The result is earlier propagation through the network.
With 15 second block time, this change led to quorum being reached ~2 seconds faster – a significant improvement.
Additional baker improvements:
- Operations selection. Bakers are prevented from wasting time on validating a manager operation that would always fail due to insufficient remaining gas in the block.
- Operations request mechanisms. Bakers no longer fetch complete mempools from peers, which can delay consensus operations when the mempool is large.
- Consensus disk writes. The number of on-disk writes when registering consensus votes are reduced.
- Optional state disk writes. Consensus state is no longer written to disk at each block.
- Attestation injection time. Attestations are sent immediately after pre-quorum has been detected, without waiting for block application.
Mempool improvements:
- Advertisement computation. A change to how the mempool data structure is filled, which saves a lot of computation.
- Advertisement priority. Avoids re-advertising non-consensus operations that have been advertised for a previous block.
- Classification of operations order. Optimizations of mempool search reduces time spent by putting more time consuming elements last.
Lastly, the number of cycles with metadata kept on disk for full and rolling nodes has been reduced from 6 to 2. The result is a lighter storage allowing faster access.
Result of improvements
Below we present the results of an experiment using a development version of Octez (v20-dev) with the above improvements enabled. We used the same hardware specifications and parameters as in the baseline experiment.
The perhaps most important metric – the time required to reach quorum, allowing the network to progress to the next block – decreased significantly, from ~11 to ~4.5 seconds.
Average time elapsed since block timestamp, high network load (full blocks) Before Octez improvements (v18.1) and after (v20-dev) |
||
Stages of a round | v18.1 | v20-dev |
Block validated: ready for consensus | 4.51s | 1.08s (-76%) |
Block fully applied: new chain head | 6.02s | 2.53s (-58%) |
Pre-quorum reached: first consensus vote complete | 7.47s | 2.88s (-61%) |
Quorum reached: second and final consensus vote complete | 11.14s | 4.55s (-59%) |
The ‘Attestation Reception Delay’ graph shows the result for a single, random block level during the Octez v20-dev experiment. It aligns with the summary and shows that (pre-)attestations now arrive quicker and more consistently around the average time.
Based on these results, we are confident that it is safe to reduce Tezos’ block time to 10 seconds, once the improvements are implemented in Mainnet nodes and bakers.
Our experiments showed even an 8 second block time to be safe, but the more conservative 10 seconds was chosen for the upcoming “P” protocol proposal for extra safety margin.
Hardware recommendations (again)
The following hardware specifications match the ones used for our experiments. They are provided here as a guideline, but bakers are advised to perform their own testing.
- 3 cores, 2 needed by the node and 1 needed by the baker (arm64 or amd64/x86-64)
- 8GB of RAM + 8GB of swap or 16GB of RAM1
- 100GB SSD storage (or similar I/O performance)
- A low-latency reliable internet connection
While it is possible to run the baker setup on a cloud platform, it may not be cost effective over the long term. Instances with specs similar to following (or higher) would work:
- n1-standard-4 (2 GHz) machine with Intel Broadwell CPU platform
- 6.5Gb RAM allocated to baker and node processes
- 100GB Persistent disk SSD disk type
Impact on Mainnet
As mentioned initially, lowering the block time improves the latency of Tezos Layer 1, resulting in a smoother experience and faster finality.
Some parameters are affected by the change:
Gas per block. Since the gas limit per block is correlated with block time, a 10 second block time means the gas limit will be lowered from 2.4M to 1.73M gas units. As there are more blocks pr. minute, throughput is unchanged. The hard gas limit per operation is left unchanged at 1M gas units.
Consensus traffic. Increasing the number of blocks per minute increases bandwidth consumption from consensus operations. Experiments showed an increase from 200kB/s to 250kB/s on a network running at full load, which is well within acceptable limits.
Cycle length. Increasing the number of blocks per minute (from 4 to 6) also impacts the cycle size as each cycle will have more blocks (from 16384 to 24576).
For nodes running a rolling history mode, we’ve implemented a change that counteracts the increase in storage due to the extra blocks. Historically, such nodes have been required to store two weeks of blockchain data (5-6 cycles) for security purposes, but the deterministic finality of the Tenderbake consensus algorithm allows us to safely reduce it to 3-6 days (1-2 cycles).
Hence, while the number of blocks is increased by 50%, the storage footprint for rolling nodes is reduced by ~75%. Meanwhile, archive nodes and full nodes, which store all the chain’s history, will see an increased disk footprint with a 10 second block time.
Next steps
We will continue to run experiments, identify bottlenecks and implement improvements to further improve overall performance of Octez and enable even lower block times.
A factor worth pointing out is the signing delay introduced by older hardware devices, such as the Ledger Nano S. Experiments with added delays corresponding to these hardware devices yielded the following results:
Machine type | Signing device | Minimal safe block time |
n1-standard-4 (4-core vCPU @ 2GHz, 8GB RAM assigned) |
Ledger Nano S | 8 |
Ledger Nano S+ | 7 | |
n2-standard-4 (4-core vCPU @ 2.6GHz, 8GB RAM assigned) |
Ledger Nano S | 6 |
Ledger Nano S+ | 5 |
The results indicate that a further reduction of block time (by 1 second), could be achieved confidently, if the majority of bakers currently using Ledger Nano S were to adopt the Ledger Nano S+, or another equivalently faster signing solution.
Also worth noting is the difference in performance between the two machine types. Unsurprisingly, the ‘n2’ machine’s faster CPU enables significantly lower block times. However, hardware specifications must always be considered in the context of making sure Tezos has a low barrier for participation to encourage decentralization.
Keeping Tezos decentralized
As our experiments show, Tezos is able to have a Layer 1 block time of 10 seconds while maintaining low hardware requirements.
A quick way to enable further block time reductions would be to increase requirements for hardware. This is not an uncommon approach for blockchains looking to boost performance.
Ethereum, for example, has higher official hardware requirements than Tezos – at least 16GB RAM and a more performant CPU. A chain like Solana is able to produce blocks at a very high pace, but relies on significantly higher official hardware requirements, including 12 CPU cores @2.8 GHz, 256GB of RAM, 2 SSD disks up to 1TB, and 1 Gbit/s internet connection with 10Gbit/s preferred.
For now, we maintain a more conservative target for our work on Tezos. This is because higher barriers for participation may affect decentralization negatively – the economic cost of the hardware is one factor, with access to the required internet bandwidth being another.
As a reminder, decentralization is not just a principled ideal. Ultimately, decentralization remains the only way to enable true censorship resistance – a core purpose of blockchains.