diff --git a/in-progress/8025-publishing-tx-objects/design.md b/in-progress/8025-publishing-tx-objects/design.md index 3d740d67..3cd78e43 100644 --- a/in-progress/8025-publishing-tx-objects/design.md +++ b/in-progress/8025-publishing-tx-objects/design.md @@ -23,98 +23,30 @@ We have epochs, which are divided into $E$ slots, which are measured in $S$ seco An epoch has a committee, which is responsible for proposing/validating blocks in the pending chain. -The committee is sampled from a broader validator set based on a seed that gets set when the first proposal in the *previous* epoch is submitted. +The committee is sampled from a broader validator set based on a seed that gets set when the first proposal in the _previous_ epoch is submitted. The committee changes every epoch. + ![](./setup.png) ### Proposal process overview TODO - ### Committee Signature Scheme -- Optimistic BLS - -To see how we get to this, look at [Pleistarchus](https://github.com/AztecProtocol/aztec-packages/issues/7978). - -### What is published, when - -A proposer will collect signatures from the committee, and then publish (within a single L1 transaction) the content in blobs and his "proposal" to the rollup contract: - -- CallData - - proposal: - - numTxs - - txsHash (a commitment to first nullifiers of all transactions in the proposal) - - kzgHashes (a commitment to the versioned KZG hashes which include all the transactions of the proposal) - - inHash - - GlobalVariables - - blockNumber - - slotNumber - - timestamp - - coinbase - - feeRecipient - - gasSetting (`fee_per_da_gas`, `fee_per_l1_gas`) - - Proposer sig (ECDSA) - - Attestations: - - aggregate BLS sig - - bitmap for missing signatures -- Blobs - - Tx1 - - max fee - - note hashes (from private) - - nullifiers (from private) - - l2ToL1Messages (from private) - - note encrypted logs (from private) - - encrypted logs (from private) - - unencrypted logs (from private) - - public call request 1 - - contract address - - call context - - msgSender - - storageContractAddress - - functionSelector - - isDelegateCall - - isStaticCall - - args - - public call request 2 - - ... - - Tx2 - - ... - -> Most of the global variables could be directly populated by L1, but providing it makes it a lot clearer what the reason of failure is if such a one is encountered. Also the `inHash` can be populated by L1. - -#### DA Oracle Changes - -When the data is to be submitted as part of the same transaction as the block proposal to the L1 contract, we have two variations possible: +For the following changes it is assumed that [Pleistarchus](https://github.com/AztecProtocol/aztec-packages/issues/7978) will be used. +This means that we will be using an optimistic approach for the sampling, proposer and signature validation. -1. Remove the data availability oracle in its whole, and simply use the versioned KZG hashes directly -2. Save the versioned KZG hashes in the data availability oracle +### Changes to contracts -Solution 1 is significantly cheaper as it don't update storage. Furthermore, by requiring that the versioned KZG hashes are part of the proposal provided, we can show the exact KZG hashes using just the transaction and the commitment to the proposal. +The contracts will be altered such that instead of receiving block `header`s and bodies with tx-effects (state diffs), we will now be receiving `proposal`s which include parts of what was previously in the `header` but not all of its components. -#### Rollup Contract Changes +The proposal will be of the structure as follows, notable they DO NOT include an archive or references to the state: ```python -struct ChainTip: - block_number: uint256 - slot_number: uint256 - - -struct State: - pending_tip: ChainTip - proven_tip: ChainTip - - -struct ProposalLog: - hash: bytes32 - slot_number: uint256 - archive: bytes32 - - struct GasSettings: feePerDaGas: uint256 feePerL2Gas: uint256 @@ -131,17 +63,89 @@ struct GlobalVariables: struct Attestations: signature: AggregateBLS - missing: uint256 # support up to 3*256 members in committee + missing: uint256 # Supports committee size of 256 members -struct Proposal: +struct ProposalHeader: num_tx: uint256 txs_hash: bytes32 kzg_hashes: bytes32 in_hash: bytes32 global_variables: GlobalVariables proposer_sig: ECDSA - attestations: Attestations + + +struct CallContext: + msg_sender: AztecAddress + storage_contract_address: AztecAddress + function_selector: bytes4 + is_delegate_call: bool + is_static_call: bool + + +struct PublicCallRequest: + target: AztecAddress + call_context: CallContext + args: DynArray[bytes32, ] #How we are dealing with the max number of args? + + +struct PrivateTxEffects: + note_hashes: DynArray[bytes32, MAX_NOTE_HASHES_PER_TX] + nullifiers: DynArray[bytes32, MAX_NULLIFIERS_PER_TX] + l2_to_l1_messages: DynArray[bytes32, MAX_L2_TO_L1_MSGS_PER_TX] + note_encrypted_logs_hashes: DynArray[bytes32, MAX_NOTE_ENCRYPTED_LOGS_PER_CALL] + encrypted_logs_hashes: DynArray[bytes32, MAX_ENCRYPTED_LOGS_PER_CALL] + unencrypted_logs_hashes: DynArray[bytes32, MAX_UNENCRYPTED_LOGS_PER_CALL] + + +# Need a better name for this one! +struct TxObject: + max_fee: uint256 + private_tx_effects: PrivateTxEffects + public_call_requests: DynArray[ + PublicCallRequest, + MAX_PUBLIC_CALL_STACK_LENGTH_PER_TX + ] + + +# Will be published in blobs. Capped by blobs +struct ProposalBody: + txs: DynArray[TxObject, ] + +``` + +As noted earlier, in [Proposal process overview](#proposal-process-overview), the proposal will eventually either be pruned or proven. To propose, the sequencer is publishing `ProposalHeader` and attestation as calldata and `ProposalBody` as blobs within a single L1 transaction. + +> Most of the global variables could be directly populated by L1, but providing it makes it a lot clearer what the reason of failure is if such a one is encountered. Also the `inHash` can be populated by L1. + +#### DA Oracle Changes + +When the data is to be submitted as part of the same transaction as the block proposal to the L1 contract, we have two variations possible: + +1. Remove the data availability oracle in its whole, and simply use the versioned KZG hashes directly +2. Save the versioned KZG hashes in the data availability oracle + +Solution 1 is significantly cheaper as it don't update storage. Furthermore, by requiring that the versioned KZG hashes are part of the proposal provided, we can show the exact KZG hashes using just the transaction and the commitment to the proposal. + +**Conclusion**: The data availability oracle dies. + +#### Rollup Contract Changes + +```python +struct ChainTip: + block_number: uint256 + slot_number: uint256 + + +struct State: + pending_tip: ChainTip + proven_tip: ChainTip + + +struct ProposalLog: + hash: bytes32 + slot_number: uint256 + archive: bytes32 struct FeePayment: @@ -149,7 +153,7 @@ struct FeePayment: amount: uint256 -EPOCH_LENGTH: constant(uint256) = 48 # pulled out of my ass +EPOCH_LENGTH: constant(uint256) = EPOCH_LENGTH proposals: public(HashMap[uint256, ProposalLog]) @@ -157,22 +161,25 @@ state: public(State); def __init__(): - proposals[0] = Proposal(hash = GENESIS_PROPOSAL_HASH, slot = 0, archive = GENESIS_ARCHIVE) + proposals[0] = ProposalLog( + hash = GENESIS_PROPOSAL_HASH, + slot = 0, + archive = GENESIS_ARCHIVE + ) -def propose( - proposal: Proposal, - proposer_sig: ECDSA, - attestations: Attestations - ): +@blobs(ProposalBody) # This is my ugly way of saying is in blob +def propose(proposal: ProposalHeader, attestations: Attestations): ''' Notice that no signatures are checked, all of that is optimistic + The signatures will be of the `ProposalHeader` ''' if proposal.num_tx == 0: assert len(tx.blob_versioned_hashes) == 0 assert proposal.kzg_hashes = empty(bytes32) else - assert hash([h for h in tx.blob_versioned_hashes]) == proposal.kzh_hashes # Replaces DA oracle + # Replaces DA oracle + assert hash([h for h in tx.blob_versioned_hashes]) == proposal.kzg_hashes gv = proposal.global_variables @@ -187,17 +194,11 @@ def propose( assert gv.block_number == self.state.pending_tip.block_number + 1 assert proposal.inHash == INBOX.consume(gv.block_number) - proposal_hash = sha256(proposal, proposer_sig, attestations) - - self.proposals[gv.block_number] = ProposalLog(proposal_hash, gv.slot_number) + self.proposals[gv.block_number] = ProposalLog(sha256(proposal, attestations), gv.slot_number) self.state.pending_tip = ChainTip(gv.block_number, gv.slot_number) -def challenge_proposal( - proposal: Proposal, - proposer_sig: ECDSA, - attestations: Attestations - ): +def challenge_proposal(proposal: ProposalHeader, attestations: Attestations): ''' Implement challenges for: - Bad proposer @@ -251,145 +252,74 @@ def submit_next_epoch_proof(proof, archive: bytes32, fees: FeePayment[EPOCH_LENG - TODO: - What are we going to do with stuff like the outhashes? They depend on the execution, so we cannot really do it at the time of the proposal if that is. So we would practically need it to happen with the entire epoch at this point. Might just be another tree to make it cheaper. It gets kinda funky with finding the block number because it depends. -### Forced inclusion of transactions - -- Need to fix some notation on what is the "full" proposal etc. -- My syntax is not really valid vyper but I just enjoy writing it that way, sorry not sorry. - -The idea: - -- We have a commitment `txs_hash` that is a l1 friendly merkle tree of the first nullifiers in a proposal. -- To do forced inclusion, we require that a specific `nullifier` is in one of these trees at some point in time. - - If not, then the proof will simply fail and one will end up eventually submitting a new proposal that have it to extend the chain - - If the committee stalls forever, we need an option to go "based" and propose and verify the proof at the same time. - -We will first showcase, how you can force the committee, and then afterwards the changes needed to perform the based operation. The reason behind this is mainly that I have not outlined the based yet. - -```python -struct ForceInclusion: - nullifier: bytes32 - include_by_slot: uint256 - included: bool - - -struct ForceInclusionProof: - proposal: Proposal - forced_inclusion_index: uint256, - block_number: uint256 - membership_proof: bytes32[] - - -forced_inclusions: public(HashMap[uint256, ForceInclusion]) -forced_inclusion_tip: public(uint256) -forced_inclusion_count: public(uint256) - -FORCE_INCLUSION_DEADLINE: immutable(uint256) - - -def __init__(deadline: uint256): - self.FORCE_INCLUSION_DEADLINE = deadline - +### Changes to circuits -def initiate_force_include(tx, proof, block_number_proven_against): - ''' - To be used by a user if they are getting massively censored by - the committees. - ''' +#### Rollup circuits - assert block_number_proven_against <= self.proven_tip.block_number +Circuit changes for the rollup circuits have a few points of interest: +- [The transaction validity conditions](#transaction-validation) +- The introduction of `txs_hash` +- Posting the objects instead of effects as the body - archive = self.proposals[block_number_proven_against].archive - assert proof.verify(archive, tx) +We will ignore the validity conditions here since that is described later. - self.forced_inclusions[self.forced_inclusion_count] = ForceInclusion( - nullifier = tx.nullifiers[0], - include_by_slot = get_current_slot() + self.FORCE_INCLUSION_DEADLINE - ) - self.forced_inclusion_count += 1 +For the `txs_hash` the idea is relatively simple - build a merkle root (SHA256) of the `tx_hash` for every transaction in the proposal. +As briefly mentioned earlier, this is to more easily support forced inclusion, and prove that a tx was included in a specific block. +The practical implementation would be very much like the `out_hash` or `txs_effects_hash`. +For the objects, we can keep the private state diff as is, but need to have the private kernel insert the enqueued public calls as well. +No state changes from the AVM should make their way into this, so the public kernel should be altered to not push things around. -def show_included(fip: ForceInclusionProof): - ''' - Convince the contract that a specific forced inclusion at `forced_inclusion_index` is - indeed included. - ''' - assert fip.forced_inclusion_index < self.forced_inclusion_count - nullifier = self.forced_inclusions[fip.forced_inclusion_index].nullifier - - assert self.proposals[fip.block_number].hash == fip.proposal.hash() - assert fip.membership_proof.verify(nullifier, fip.proposal.txs_hash) +### Changes to the node - self.forced_inclusions[fip.forced_inclusion_index].nullifier.included = True +Since the L1 will no longer know about the complete state-diffs, the node will be unable to simple download these and apply them. This means that the node will instead get a hold of the `ProposalHeader` and `ProposalBody` from L1. It could then apply these proposals on top of its current state to progress. Essentially to sync all of the history on top of the genesis state by re-execution. This model is very similar to what is done in Ethereum or really most blockchains. Consider looking at https://forum.aztec.network/t/collaborative-based-rollups/5227 for a more thorough walkthrough. - self.progress_forced_inclusion_tip() +When applying the history through re-execution, the node will need to also use the same version of the AVM (Aztec Virtual Machine) as was used when the proof was originally submitted to the rollup contract, practically meaning that the node will need multiple versions and how to distinguish between "when" they were used. Keeping track of the versions and when they were used is handled by the Gerousia and Apella design, but the actual versions need to stored separately as they are only referred through version id and verification keys on the rollup. +Note, that since every epoch is to be proven, there will be a state root `archive` after each of those that a user can use as a "root of trust" and check against. This is also useful when validating snapshots received. -def progress_forced_inclusion_tip(): - cache = self.forced_inclusion_tip - for i in range(cache, self.forced_inclusion_count): - if not self.forced_inclusions[i].included: - return - self.forced_inclusion_tip = cache +The node will have its own view of the current `archive` and state based on the proposals that it have applied itself. +> Remember, that one can always get someone else to perform the job of re-executing (look at infura/alchemy and Ethereum), but that comes with its own concerns. -@override -def submit_next_epoch_proof(proof, archive: bytes32, fees: FeePayment[EPOCH_LENGTH]): - super.submit_next_epoch_proof(proof, archive, fees) +#### Forks (re-orgs) - forced_tip = self.forced_inclusions[self.forced_inclusions_tip] +Separately to this change with re-execution, the node will need to handle forking and re-orgs as well. This is not really specific to this proposal, but have not really been discussed for the point of view of the node. When we above pointed to the current `archive` based on the proposal it have applied, we say so as there could potentially be multiple proposals which have been published (for the same block number / at the same height). - if forced_tip.included_by_slot != 0: - assert forced_tip.included_by_slot > proposal_hashes[-1].slot, 'force' +Since we push the L1 to figure out what is the real, the node will need to take the `ProposalLog` from the contract into account when it is applying the state changes. +To have the view of the proven chain, the node must simple apply all of the `ProposalLog`'s that are stored in the `proposals` and have a block number smaller than or equal to the `proven_tip`. +Everything beyond the `proven_tip` is still "up for discussion". +This means that we need to support the ability to "revert" full blocks, since they might have been applied at the start of an epoch, but you then later figure out that it was no good, e.g., not proven in time, +In this case, you want to go back to the proven chain, throwing the current pending chain away, and then start new pending proposals. - -def submit_proof_with_force(proof, archive, fips: ForceInclusionProof[]): - ''' - To be used by the proof submitter if they are including forced inclusions. - If the forced inclusions are not needed for their block, but is for a later deadline - they can use the old, and someone else can include it later when required - ''' - - super.submit_next_epoch_proof(proof, archive, fees) - - for fip in fips: - self.show_included(fip) - - forced_tip = self.forced_inclusions[self.forced_inclusions_tip] - if forced_tip.included_by_slot != 0: - assert forced_tip.included_by_slot > proposal_hashes[-1].slot, 'force' -``` - -### Blob circuits - -### Rollup circuits +The PXE also need to have an idea around if things have been forked out, since we don't want it to believe that it have plenty of notes, but only figure out when it needs to spend those that it don't. +One way to deal with this, could be that the PXE will check that its "new" notes and nullifiers are indeed in the `archive` whenever a proof lands on L1. ### Prover interactions -### Keeping track of chain state - -How do nodes keep track of the pending archive since they are not published to L1? - ### Private kernel verification -Validators need to verify the private kernels +Validators need to verify the private kernels, and should only attest to proposals where all are valid. ### Transaction validation -If a transaction is "valid" if and only if it can be included in a block (and subsequently an epoch), and the proof of that epoch can be verified on L1. +A transaction is "valid" if and only if it can be included in a block (and subsequently an epoch), and the proof of that epoch can be verified on L1. Nodes need to "validate" transactions: i.e. check if a transaction is valid. -It *must* be possible to determine if a transaction is valid without executing it. +It _must_ be possible to determine if a transaction is valid without executing it. -Therefore, transactions *must* be valid iff it has a private kernel proof that successfully verifies. +Therefore, transactions _must_ be valid iff it has a private kernel proof that successfully verifies. #### Changes needed There are many cases today where a transaction is invalid, and cannot be included in a block, e.g.: + - global variable mismatches (chainId, version, etc.) - transactions reverting in public setup - the transaction was not included before its "max block" - duplicate nullifiers +- invalid sibling paths - etc. The protocol must gracefully handle these cases, and instead of having the transaction be invalid, it should allow the transaction to be included, but with a "failed" status. @@ -398,6 +328,15 @@ In the event of a "failed" transaction, the transaction will appear in the block Further, the transaction's fee will be set to zero. +**Invalid sibling paths**: +As the sequencer is the one providing membership paths for the base rollup, it must not be possible for him to deliberately provide bad paths, thereby making the tx "invalid" and make it have no effect. +To address this, we can add another check to each of our membership or non memberships, to ensure that the paths provided were not utter nonsense. +Remember that failure to prove inclusion is not equal non-inclusion. +This check is fairly simple, if it is a membership check where an index was provided, and it fails, the sequencer must show what the "real" value was, and that it differs. +If it is a membership without a provided index, and it fails, a non-membership must be made. +If it is a non-membership we must prove that it was in there. +Essentially the sequencer is to do an `xor` operation, with membership and non-membership - one of them must be valid if he is not lying. + ### Proving phases