TL;DR: We found a critical liveness bug in Tezos’ Smart Rollups shortly after the activation of the Paris protocol. This issue does not affect the security of Tezos mainnet and Etherlink — assets are safe! It does affect however users ability to withdraw funds from Etherlink to Tezos Layer 1. This issue is fixed by ParisC, an emergency bugfix protocol for Tezos mainnet. Octez v20.1 includes ParisC as a user-activated upgrade. This means that Octez nodes running v20.1 will automatically activate ParisC on block #5,898,241.
Shortly after the activation of the Paris protocol upgrade on June 4th, we found a critical liveness bug with Tezos’ Smart Rollups. The issue originates from the business logic in the Tezos economic protocol which deals with cementing rollup commitments. In particular, the bug prevents cementing commitments that were published but were not cemented before the activation of Paris.
This issue does not affect the security of Tezos mainnet and of Etherlink — assets are safe! It does not challenge either the liveness of Tezos Layer 1. It does affect however users ability to withdraw funds from Etherlink to Tezos Layer 1.
Our assessment concluded that this issue can only be addressed by patching the Paris protocol currently live on Tezos mainnet as soon as possible. We have prepared an urgent bugfix protocol, ParisC, to restore users’ ability to move assets between Etherlink and Tezos Layer 1 seamlessly.
Octez v20.1 was released earlier
today. It
Includes ParisC as a user-activated upgrade (UAU) at level
#5,898,241 — not expected before June 25
2024, 03:11:25 UTC. This means that nodes running Octez v20.1 (and
higher) will activate ParisC,
PsParisCZo7KAh1Z1smVd9ZMZ1HHn5gkzbM94V3PLCpknFWhUAi, just before
block level #5,898,241.
Note that Octez versions prior to v20.1 do not include the ParisC protocol, nor its associated binaries. Bakers and node operators running incompatible versions will not be able to activate ParisC and will see themselves unable to participate in Layer 1 consensus after the network upgrades.
We acknowledge that this critical issue, and the urgent measures required to address it, are not isolated events. Regrettably, they come after needing to deploy a user-activated protocol-override for ParisB a few weeks ago, and other recent security patches and bug-fixes to Octez and the Tezos protocol.
We share the community’s concern and we will address this situation in the near future. We will adjust our development and testing processes in upcoming protocol proposals so as to meet the standards that both the community, and our own history as contributors to the core-development of Tezos, demands.
We are confident, however, that the measures we outline in this post constitute the best way forward at the moment, and that they prepare the ground for the successful launch of Etherlink.
Note also that ParisC does not affect the timeline for the activation of Adaptive Issuance and Staking. These features will activate at the beginning of cycle 748 — on block #5,849,089.
Octez v20.1 also includes optimizations in the baker binary and other minor fixes and improvements to the Smart Rollup node. We refer to the Octez releases page for further detail and we will concentrate in the rest of this announcement on the bug (and its fixes) which motivates the need for ParisC.
Bug analysis: commitment cementation does not adapt correctly to Paris migration
A Tezos Smart Rollup is a dedicated external processing unit that outsources computation outside the domain of Tezos Layer 1. Each rollup can define their own business logic. Etherlink, for example, is an EVM-compatible layer 2 blockchain on top of Tezos Layer 1. All Tezos Smart Rollups share a common architecture and common infrastructure enshrined in the Tezos economic protocol. The common infra includes an inbox and an outbox for verifiable communication between Tezos Layer 1 and rollups and, crucially, a common business logic for fraud proofs — aka refutation games.
Rollup commitments are part of the common architecture as well. Commitments are a special kind of message which are posted by rollup nodes on Tezos Layer 1. They represent a claim that the interpretation of all inbox messages included in a sequence of Layer 1 blocks of predefined length (called the commitment period), when applied on the state of a parent commitment, led to a given new state of the rollup, by performing a given number of valid execution steps in the rollup’s business logic.
Commitments are optimistically trusted but can be challenged in
refutation games. Cementing a commitment makes it final and
unchangeable, placing it outside the scope of new refutation
games. Cemented commitments can then be used to prove the existence of
outbox messages allowing the execution of transactions on behalf of
the rollup. Etherlink relies on such outbox messages to implement tez
withdrawals from the rollup to Tezos Layer 1. The economic protocol
makes use of a specific
constant,
smart_rollup_commitment_period_in_blocks to identify the next
commitment to cement.
However, the business logic as implemented in Paris failed to handle the modification of the commitment period caused by the reduction of minimal block time to ten seconds in effect since the activation of Paris. As a result, the protocol cannot retrieve the last cemented commitment and prevents any recently published commitment from being cemented.
As a consequence, the impact on Etherlink users is critical, because this issue effectively prevents them from withdrawing tez from Eterlink to Tezos Layer 1.
Any call to the pre-compiled Etherlink withdraw
contract
that is more recent than the latest cemented commitment cannot be
claimed on Tezos Layer 1.
Under normal circumstances, it takes two weeks for a withdrawal order to be completed. However, due to this bug the latest cemented commitment is the one included on #5,645,527 on May 21. Consequently, no withdrawal request happening after that date can be successfully claimed until this issue is fixed and more recent commitments can be cemented. This doesn’t mean that funds are lost — funds are safe —, but that they cannot be made available on Layer 1 until the successful deployment of ParisC.
Fixing commitment cementation in ParisC
Disclaimer This subsection overviews the necessary changes in ParisC required to address the bug described just above. It assumes some technical understanding of the internals of the Tezos economic protocol implementation. Tezos is an open source project and every change to the implementation of the economic protocols distributed with Octez can be tracked on the tezos/tezos Gitlab repository. Still, describing the changes here contributes to reassure interested parties as to the sole purpose of the user-activated upgrade.
Technical understanding of the implementation of Smart Rollup’s
business logic aside, fixing this bug is quite straightforward. The
key necessary changes in commitment cementation are circumscribed to
the following patch applied to
sc_rollup_stake_storage.ml,
in
tezos/tezos!13747:
let* old_lcc, old_lcc_level, ctxt =
Commitment_storage.last_cemented_commitment_hash_with_level ctxt rollup
in
- let sc_rollup_commitment_period =
- Constants_storage.sc_rollup_commitment_period_in_blocks ctxt
+ let* last_proto_activation_level, last_commitment_period =
+ Sc_rollup_storage.previous_protocol_constants ctxt
+ in
+ let new_lcc_previous_protocol =
+ Raw_level_repr.add old_lcc_level last_commitment_period
in
let new_lcc_level =
- Raw_level_repr.add old_lcc_level sc_rollup_commitment_period
+ if Raw_level_repr.(new_lcc_previous_protocol < last_proto_activation_level)
+ then new_lcc_previous_protocol
+ else
+ let sc_rollup_commitment_period =
+ Constants_storage.sc_rollup_commitment_period_in_blocks ctxt
+ in
+ Raw_level_repr.add old_lcc_level sc_rollup_commitment_period
in
(* Assert conditions to cement are met. *)
let* ctxt, (new_lcc_commitment, new_lcc_commitment_hash), dangling_commitments
The cement_commitment function now computes the blockchain level
associated with a commitment using the appropriate constant, depending
on whether the commitment was published before or after the activation
of the current protocol. This approach requires knowing the previous
value of the smart_rollup_commitment_period_in_blocks constant, and
the activation level of the current protocol (here, ParisB2).
We have hard-coded these two values for the two networks affected by
this bug (mainnet and ghostnet) together with the necessary changes
required to enable the migration from ParisB to ParisC. These are
implemented in
init_storage.ml:
+ let parisb2_activation_level, previous_commitment_period =
+ if Chain_id.(chain_id = Constants_repr.mainnet_id) then
+ (Raw_level_repr.of_int32_exn 5726209l, 60)
+ else if Chain_id.(chain_id = Constants_repr.ghostnet_id) then
+ (Raw_level_repr.of_int32_exn 6422529l, 60)
+ else
+ (* Setting [paris2b_activation_level = Raw_level_repr.root] ensures
+ that we will never consider an uncemented commitment as posted
+ during the previous protocol. *)
+ (Raw_level_repr.root, 1)
+ in
We will generalize this bugfix in an upcoming Q protocol upgrade proposal to ensure that future changes to block time-related constants are handled more gracefully by Smart Rollups.
Finally, we have also modified assert_commitment_period in
sc_rollup_stake_storage.ml. This
function is used when a commitment is submitted to Tezos Layer 1 to
ensure its publication is within the correct commitment period. It was
also affected by the same issue as cement_commitment above and the
new value for the length of the period was used for block levels
preceding the activation of the new protocol.
let* pred, ctxt =
Commitment_storage.get_commitment_unsafe ctxt rollup pred_hash
in
+ let* last_proto_activation_level, last_commitment_period =
+ Sc_rollup_storage.previous_protocol_constants ctxt
+ in
let pred_level = Commitment.(pred.inbox_level) in
(* Commitments needs to be posted for inbox levels every [commitment_period].
Therefore, [commitment.inbox_level] must be
[predecessor_commitment.inbox_level + commitment_period]. *)
- let sc_rollup_commitment_period =
- Constants_storage.sc_rollup_commitment_period_in_blocks ctxt
+ let last_proto_expected_level =
+ Raw_level_repr.(add pred_level last_commitment_period)
+ in
+ let expected_level =
+ if Raw_level_repr.(last_proto_expected_level < last_proto_activation_level)
+ then last_proto_expected_level
+ else
+ Raw_level_repr.add
+ pred_level
+ Constants_storage.(sc_rollup_commitment_period_in_blocks ctxt)
in
let* () =
fail_unless
- Raw_level_repr.(
- commitment.inbox_level = add pred_level sc_rollup_commitment_period)
+ Raw_level_repr.(commitment.inbox_level = expected_level)
Sc_rollup_bad_inbox_level
in
return ctxt
Do not hesitate to reach out to us on our usual channels for further questions about the bug and how the ParisC protocol addresses the issue.
Call to (collective) action: ensuring a smooth activation of ParisC
As we mentioned above, in order to restore Etherlink withdrawals,
Paris C will be deployed as a user-activated upgrade (UAU). This
means that Octez nodes running Octez v20.1 (and higher) will
automatically enforce the activation of Paris C,
PsParisCZo7KAh1Z1smVd9ZMZ1HHn5gkzbM94V3PLCpknFWhUAi, just before
block level #5,898,241 — not expected
before June 25 2024, 03:11:25 UTC. The exact time will depend on
network health, as it is the case with any protocol upgrade.
A UAU is a measure of last resort whose success requires greater ecosystem coordination than regular protocol upgrades, and within a tighter schedule.
Bakers and node operators should urgently upgrade their infrastructure to Octez version v20.1 in order to ensure the success of the UAU. Those operating earlier versions will not be able to activate the patched protocol and would see themselves unable to participate in Layer 1 consensus afterwards.
The operational procedure for bakers is akin to a regular protocol activation. With sufficient time before block level #5,898,241 bakers should:
- upgrade their installation of Octez to v20.1,
- make sure to relaunch v20.1
octez-nodeand v20.1octez-baker-PtParisB(Paris B baker binary), and other optional Paris B binaries, e.g.,octez-accuser-PtParisB); and, - launch v20.1
octez-baker-PsParisC(Paris C baker) and other optional Paris C binaries, e.g.octez-accuser-PsParisC.
ParisC binaries will be idle before the UAU level, and will take over once ParisC has successfully activated. Conversely, ParisB binaries will work until the UAU level, and will become idle later. You can safely stop them after the activation of ParisC.
There is no risk of double baking or double attesting blocks by running Paris B and Paris C baking software at the same time, as these are different protocols with two different hashes – despite being virtually identical except for the fixes described above.
Again, as is the case with any protocol upgrade, bakers must make sure that they are not running two baker binaries for the same protocol at any time. Otherwise, there is indeed a risk of double signing block and consensus operations.
We invite bakers and node operators to test their setups in a
dedicated pariscnet test network, which
will start soon, before upgrading to Octez v20.1 on Tezos mainnet. The
UAU will also be deployed on the
Ghostnet test network on block
level #6,729,729, not before June
20 2024 03:24:28 UTC.
This urgent upgrade also affects dev tooling and the infrastructure stack on top of the Octez suite and the Tezos protocol. We have reached out to several teams, and we will be working closely together to ensure the ecosystem tooling is ready in time.
Note also that ParisC does not affect the timeline for the activation of Adaptive Issuance and Staking. These features will activate at the beginning of cycle 748 — on block #5,849,089.
Do not hesitate to reach out with questions and concerns on this bug, ParisC and how to prepare for the UAU — we are available on our usual channels.
Let’s work together to ensure a smooth activation of ParisC!