-
Notifications
You must be signed in to change notification settings - Fork 167
Open
Labels
feature:benchmarkneeds-discussionNeeds discussion before proceedingNeeds discussion before proceedingscope:fillScope: fill commandScope: fill commandscope:fwScope: Framework (evm|tools|forks|pytest)Scope: Framework (evm|tools|forks|pytest)type:feattype: Featuretype: Feature
Description
Assuming that timing Engine API payload execution becomes the canonical method to benchmark opcodes, it'd be valuable to be able to report the absolute average time taken to execute a single opcode from a benchmark test. For example, how long does it take (on average) to execute an ADD
versus an ADDMOD
?
If the --opcode.count
functionality, cc:
- chore(fill): decide on an "opcode count"
t8n
interface #1852, and, - feat(t8n): Support evmone's
--opcode.count
option #1956
becomes standard int8n
tools, then we could include this information in state (for the single state test tx) and blockchain (for every block in the fixtuer) in test fixtures in order to calculate and report the relative average time taken directly inconsume
.
The worst case block execution time for a crafted benchmark test targeting a particular opcode is obviously a valuable result and almost equivalent; but knowing the average time taken per opcode could be easier as a basis for discussion.
Considerations:
- The number of, resp. gas used by, other opcodes executed within the block. Can we normalize the result by taking this into consideration? Perhaps it's negligible?
- We could add this information to every test fixture, but perhaps it only make sense for benchmark test fixtures? (Only benchmark tests target a specific opcode adequately for the result to make sense). Perhaps there are other use cases though?
- ...
Perhaps there's a better way? We should check how single opcodes have been benchmarked on a relative basis in the past.
Metadata
Metadata
Assignees
Labels
feature:benchmarkneeds-discussionNeeds discussion before proceedingNeeds discussion before proceedingscope:fillScope: fill commandScope: fill commandscope:fwScope: Framework (evm|tools|forks|pytest)Scope: Framework (evm|tools|forks|pytest)type:feattype: Featuretype: Feature