Weight Meter
Background
It is always critical to ensure the resourced consumed by each block is limited to prevent DOS attack vector. Smart Contracts such as EVM and pallet-contract uses Gas Metering to ensure the block can be constructed within the limit. However Gas Metering have ~30% overhead (according to some number I read on internet) and therefore in order to achieve maximum performance, benchmarking & weights are used by FRAME.
In short, FRAME offers library to run benchmarks on extrinsic to measure the computation time for each extrinsic and perform linear regression analysis aims to create formula that (over)estimate the computation time for a given extrinsic parameters. The formulas are then used to generate weight functions to calculate the weights for each extrinsic call. pallet-executive and pallet-system is then use the weights to ensure the block construction time will fit in the requirement time (2s for Polkadot / Kusama).
Benchmarking & Weights works fine for FRAME pallets as they are all have a relative low execution upper bond. So for many cases where weight estimation is hard, it could just overestimate and refund unused weights. However it won't work nicely for complex DeFi platform such as Acala.
The Problem
There are many constraints on the benchmarking & weight approach:
- Linear regression analysis is used so other kinds of computation model will not have accurate result
- The weight function cannot access storage
- We can overestimate using max parameter value and refund unused weights
- It is common the logic will have many different branches and each branch have different computation complexity
- It is recommend to avoid having too many different branches so we can have benchmarks for each branch. But sometimes that just not doable
- Cannot deal with dynamic code execution. i.e. when execution path involves user supplied smart contracts
- Does not deal with hooks well
- It is recommended to have hooks have constant time, but again this is simply not possible for Acala
Proposed Solution
In order to find a right balances between two available solutions (gas meter / benchmarking & weights), we could use a Weight Meter.
The problem of Gas Meter is the high overhead which is caused by it instrument each individual instructions.
The problem of Benchmarking & Weights is it is very hard to accurately benchmarking a complex call as a whole because it could have a big number of branches. One example is DEX swap, which involves multiple token transfer, and each transfer may be handled by pallet-balances, or orml-tokens, or a ERC20 call into module-evm.
So instead of benchmark the whole extrinsic call, which is hard, we can breakdown the extrinsic call into smaller methods (which is a good coding practice anyway), and benchmark each individual method. A Weight Meter will be used to metering the weight usage of each method to come out the final weights for the call. Weight estimation, similar to gas estimation, will be used to help user to come out the appreciated weight for the call.
The existing benchmarking frameworks is similar to integration test and what we want is more like unit test where we can mock our parts and test individual parts easily.
Technical Details
- Research & study the current benchmarking code and weights setup
- Feasibility study and PoC
- Create detailed implementation plan
- Develop new benchmarking framework hopefully reuse the existing one as much as possible
- Benchmark one method instead of one call
- Benchmark with mocked Trait instead of Runtime
- Develop Weight Meter library to measure and enforce weights
- Use SignedExtension to add weight_limit to extrinsic
- Use procedural macro to annotate weights and apply Weight Meter
- Develop CLI tool to generate Weight Functions from benchmark result
- Develop RPC to estimate weights by dry run transaction and report used weight
- Refactor ORML & Acala pallets so they can be benchmarked with the new framework
- Update ORML & Acala pallets to use the new framework
- Provide scripts to automatically benchmark & generate weight functions
Weight Meter
Background
It is always critical to ensure the resourced consumed by each block is limited to prevent DOS attack vector. Smart Contracts such as EVM and pallet-contract uses Gas Metering to ensure the block can be constructed within the limit. However Gas Metering have ~30% overhead (according to some number I read on internet) and therefore in order to achieve maximum performance, benchmarking & weights are used by FRAME.
In short, FRAME offers library to run benchmarks on extrinsic to measure the computation time for each extrinsic and perform linear regression analysis aims to create formula that (over)estimate the computation time for a given extrinsic parameters. The formulas are then used to generate weight functions to calculate the weights for each extrinsic call.
pallet-executiveandpallet-systemis then use the weights to ensure the block construction time will fit in the requirement time (2s for Polkadot / Kusama).Benchmarking & Weights works fine for FRAME pallets as they are all have a relative low execution upper bond. So for many cases where weight estimation is hard, it could just overestimate and refund unused weights. However it won't work nicely for complex DeFi platform such as Acala.
The Problem
There are many constraints on the benchmarking & weight approach:
Proposed Solution
In order to find a right balances between two available solutions (gas meter / benchmarking & weights), we could use a Weight Meter.
The problem of Gas Meter is the high overhead which is caused by it instrument each individual instructions.
The problem of Benchmarking & Weights is it is very hard to accurately benchmarking a complex call as a whole because it could have a big number of branches. One example is DEX swap, which involves multiple token transfer, and each transfer may be handled by pallet-balances, or orml-tokens, or a ERC20 call into module-evm.
So instead of benchmark the whole extrinsic call, which is hard, we can breakdown the extrinsic call into smaller methods (which is a good coding practice anyway), and benchmark each individual method. A Weight Meter will be used to metering the weight usage of each method to come out the final weights for the call. Weight estimation, similar to gas estimation, will be used to help user to come out the appreciated weight for the call.
The existing benchmarking frameworks is similar to integration test and what we want is more like unit test where we can mock our parts and test individual parts easily.
Technical Details