+A
A
-A
  1. Research & development
  2. > Blog >
  3. TPS evaluation for Tezos

TPS evaluation for Tezos

in-depth
19 April 2022
Mark Karpov (Tweag) and Nomadic Labs
share:

In this post we are going to try to answer the question “what is the throughput of Tezos on the Mainnet” by examining the subject from different angles. First, we must note that there is no standard methodology for measuring the throughput of a blockchain. In general, people want to know the number of transactions per second (TPS) that a blockchain can process. There are, however, a few questions that need to be answered first:

  • What is the configuration of the network that is being examined? Is it a minimal network or a larger network where latencies are increasingly important?
  • What does it mean for a transaction to be “processed”? Does it mean that it is included in the next produced block or perhaps in a decided block?
  • Is the TPS value limited by the protocol constants/parameters or technical limitations of the system?
  • What kind of transactions are we dealing with? In most systems that support smart contracts the makeup of the stream of transactions that is being processed is going to directly affect the maximal TPS number. The situation is simpler with e.g. Bitcoin where all transactions are identical from that point of view.

The last point is an important one because, all things being equal, TPS of a blockchain can change over time depending on the transactions users tend to perform. Let’s take a closer look.

Analyzing transactions

In Tezos, all transactions can be divided into two groups depending on their destination:

  1. Transactions that have an implicit contract as destination.
  2. Transactions that have a smart contract as destination.

In the first case the cost (both in gas and in terms of processing power) is constant no matter the amount being transferred and other parameters. In the second case, however, the cost and the performance of such a transaction is going to depend on the code of the smart contract in question.

It follows, that in order to determine the TPS, one needs to know the proportions of different kinds of transactions in the typical block. The best way to achieve that is to analyze the history of performed transactions. We choose to use the Tezos indexer to do that. The tool stores Tezos’s transaction history in a Postgresql database, which we can query and analyze. The result is a JSON file such as this one:

$ cat 2022-01-01-to-2022-02-28.json
{
  "regular": 5026035,
  "origination": 16905,
  "contract": {
    "KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton": 1611069
  }
}

Here, we can see that in the first two months of 2022 there were 5026035 transactions that had an implicit contract as their destination and the most popular smart contract at the time was KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton (Hic et Nunc NFTs) with 1611069 calls. For now, we include only smart contracts that are involved in at least 10% of all transactions, but this threshold can be changed. Having this data, we can run gas estimations for regular transfers and smart contracts calls in order to calculate the average transaction cost.

Gas-based estimation

It is possible to perform a TPS estimation based on the maximal allowed gas consumption per block. Transactions in a block cannot exceed hard_gas_limit_per_block, which is a protocol parameter; its value (on the master branch) is 5200000. With Tenderbake, in the best case scenario a block will be decided at round 0, therefore, according to the current value of minimal_block_delay, the maximal block production rate is 1 block per 30 seconds. Next, we can use the average transaction cost that we have obtained through analysis of transaction data to determine TPS as:

tps = hard_gas_limit_per_block / (average_transaction_cost * minimal_block_delay)

Our utility (which is part of Tezos and can be built by running make build-tps-deps && make build-tps) can perform this estimation:

./tezos-tps-evaluation-gas-tps -a average-block=src/bin_tps_evaluation/average-block.json
[14:26:25.243] Starting test: tezos_tps_gas
[14:26:27.956] Reading description of the average block from src/bin_tps_evaluation/average-block.json
[14:26:28.061] Originating smart contracts
[14:26:28.285] Waiting to reach the next level
[14:26:57.514] Average transaction cost: 2900
[14:26:57.514] Gas TPS: 60
[14:26:57.536] [SUCCESS] (1/1) tezos_tps_gas

We can see that the gas-based estimation of TPS on Mainnet is currently 60.

For comparison with blockchains like Bitcoin that do not support smart contracts, here is a gas-based estimation for regular transactions only:

./tezos-tps-evaluation-gas-tps
[08:40:21.806] Starting test: tezos_gas_tps
[08:40:21.807] Gas TPS estimation
[08:40:24.382] Using the default average block description
[08:40:24.499] Originating smart contracts
[08:40:24.723] Waiting to reach the next level
[08:40:54.642] Average transaction cost: 1421
[08:40:54.643] Gas TPS: 122
[08:40:54.667] [SUCCESS] (1/1) tezos_gas_tps

Benchmark-based estimation

It is also possible to lift limits imposed by the protocol in the form of constants and parameters, as well as hard-coded values in order to determine the purely technical TPS limit by running a specialized benchmark. This approach requires us to make some choices regarding the setup:

  • We are going to examine the smallest self-sufficient network possible: a node + a baker. This way we will be able to more directly judge raw performance of the node and the baker, but not of a distributed network. The former is much simpler and more predictable than the latter. Also, needless to say, performance of a distributed network depends directly on the performance of the node.
  • A transaction is considered processed if it is successfully injected and included in the next produced block, i.e. it has been successfully applied and validated.
  • The stream of transactions is modeled on actual historical data.

The TPS benchmark is run with the tezos-tps-evaluation-benchmark-tps command. It spawns a network comprising a node, a baker, and a client. The network will use the same constants and parameters as the Mainnet. By default 10 blocks will be produced, but this can be changed by supplying the blocks-total command line option. The total number of applied operations in these blocks will be divided by the total time spent producing the blocks and the resulting value will be presented as the empirical TPS. The benchmark is also capable of calculating de facto TPS of injection—the number of transactions actually injected, which is useful in judging the results.

./tezos-tps-evaluation-benchmark-tps -a average-block=src/bin_tps_evaluation/average-block.json -a lift-protocol-limits
[20:42:19.167] Starting test: tezos_tps_benchmark
[20:42:19.167] Gas TPS estimation
[20:42:21.761] Reading description of the average block from src/bin_tps_evaluation/average-block.json
[20:42:21.858] Originating smart contracts
[20:42:22.082] Waiting to reach the next level
[20:42:51.262] Average transaction cost: 2900
[20:42:51.262] Gas TPS: 60
[20:42:51.293] Tezos TPS benchmark
[20:42:51.293] Protocol: Alpha
[20:42:51.293] Blocks to bake: 10
[20:42:51.293] Accounts to use: 30000
[20:42:51.293] Spinning up the network...
[20:43:01.039] Originating smart contracts
[20:43:01.861] Waiting to reach the next level
[20:43:27.301] Using the parameter file: /run/user/1000/tezt-185969/1/parameters.json
[20:43:27.301] Waiting to reach level 3
[20:43:57.054] The benchmark has been started
[20:49:03.482] Produced 10 block(s) in 306.43 seconds
[20:49:04.091] BLSh3JB76aHp2Zg37zrv35kr6XNu8Nti9hiGELriEWNJXHoEQN9 -> 5699
[20:49:04.431] BM2MMEawEUvKZm25LoB81rGEm2MtQtdkaXzAeuJACsbcZcsuuD2 -> 5651
[20:49:04.759] BLRN2oxtgpep1D3oCiD6NGo9GPha6Z2S69LiTa8SPskHLdjN1Ag -> 5762
[20:49:05.095] BMD3U3mv8EM9ZKYsK6sxFob5ZHHCYRUZiSn6LJVgvnp8eMaV8bY -> 5822
[20:49:05.411] BLCZf4mHGaawCCuJZLF2yFXwhg1tn32BbBezYm4dnik1mfDQ4wK -> 5892
[20:49:05.684] BLpPYyppLmp4enYopGBWNHvhGLzBFuWaFSJmEqkR1QQK1stWbPt -> 3868
[20:49:05.996] BKmqoWrZgevEmmd7RXaEX5umsBQaW3cnHJRK4AJq5uzEXzaryeE -> 5721
[20:49:06.303] BMGpJV4HtYjzfeq5HLJmvN9LFpPoPeuBeEdGdwYsZ57rc7gyvnQ -> 5810
[20:49:06.610] BLYCC8jRAb4xAQAYzZixC4PSYLacFtFHAV2C6jWi43obZL9aXVw -> 5537
[20:49:06.918] BLAb72vxpN5tnGoq2gbLP1ce8XfAqEruWgbkheyKdiUmMC5LuDt -> 5417
[20:49:06.918] Total applied transactions: 55179
[20:49:06.918] Total injected transactions: 55452
[20:49:06.918] TPS of injection (target): 1000
[20:49:06.918] TPS of injection (de facto): 180.96
[20:49:06.918] Empirical TPS: 180.07
[20:49:06.962] [SUCCESS] (1/1) tezos_tps_benchmark

We can see that on my laptop the empirical TPS result for the minimal network is 180.

The goal of the TPS benchmark is to give a high-level estimate of the TPS value that the system is capable of. It can be used to catch TPS regressions, but not to find where exactly the bottlenecks are.

The empirical TPS is significantly affected by the hardware on which the benchmark is run and other factors, such as the amount of logging that is performed. For example, passing -v is likely to result in lower empirical TPS values. This is why it is important to run this kind of benchmark on a dedicated runner that has predictable performance.

The empirical TPS should normally be very close to the de facto TPS of injection. If it isn’t, then it means that the system cannot keep up with the injection rate, i.e. the bottleneck is in the system. Otherwise the bottleneck is in the injecting code, as is the case in the log above.

Automatic daily runs of the TPS benchmark

Every day the TPS benchmark is run and the following results are registered:

  • Gas TPS
  • Results of running the TPS benchmark with protocol limits:
  • De facto TPS of injection
  • Empirical TPS
  • Results of running the TPS benchmark with protocol limits lifted:
  • De facto TPS of injection
  • Empirical TPS

Regressions for these values are detected and recorded using the same framework as the one Tezt long tests use.

Current limitations and future work

Since every smart contract is unique, we cannot automatically provide support for all smart contracts that might become popular in the future. Therefore, some work will be necessary in order to add more smart contracts to the benchmark.

Another area for improvement is making the injecting code more efficient so that it can reach higher rates of injection.