MAINNET 3.0 (#4843)

* Asc message execution - requery message bytecode after each message execution (#4710)

* Requery bytecode

* cargo fmt

* fix call stack inconsistency (#4709)

* Improve async message checks (#4706)

* Improve async message checks

* Change checks for async messages

* Add unit tests

* Fix ledger change to take into account cancelled message balance change (#4715)

* Take again the speculative changes after async message cancellation

* use .apply() to merge the two LedgerChanges

* Fix: we cannot combine two ledger changes with apply

* avoid cloning the changes

* Remove comment

* Fix async msg same slot (#4718)

* fix open rpc spec (#4716)

* Add eliminated_new_messages in eliminated_msg

---------

Co-authored-by: Modship <yeskinokay@gmail.com>

* Update interface_impl.rs (#4728)

* Fix potential ledger keys boundaries issue (#4731)

* Consistent expiry period for async message and block operations (#4722)

* Asc message execution - requery message bytecode after each message execution (#4710)

* Requery bytecode

* cargo fmt

* fix call stack inconsistency (#4709)

* Improve async message checks (#4706)

* Improve async message checks

* Change checks for async messages

* Add unit tests

* Fix ledger change to take into account cancelled message balance change (#4715)

* Take again the speculative changes after async message cancellation

* use .apply() to merge the two LedgerChanges

* Fix: we cannot combine two ledger changes with apply

* avoid cloning the changes

* Remove comment

* Fix async msg same slot (#4718)

* fix open rpc spec (#4716)

* Add eliminated_new_messages in eliminated_msg

---------

Co-authored-by: Modship <yeskinokay@gmail.com>

* Consistent expiry period for async message and block operations

* Update message validity for expiration

* Minor comment fix

---------

Co-authored-by: Leo-Besancon <lb@massa.net>
Co-authored-by: Modship <yeskinokay@gmail.com>

* Update ci.yml

* fmt

* Fixe issue with fees (#4734)

* fix fees

* Update test_rewards execution test

* Update test with updated reward formula

* Update ci.yml

* revert test change

* Rm dbg and create const value

* fmt

---------

Co-authored-by: Leo-Besancon <lb@massa.net>

* Fix amount remaining to slash 2 (#4733)

* Fix amount_remaining_to_slash_2 calculation

* Rename slashed_coins and calculation to improve readability

* Fix total_slashed_coins to return value

* Fix wrong LedgerChanges::Delete handling (#4737)

* Fix bad LedgerChanges::Delete handling

* Avoid too much code duplication

* Update ledger_db.rs

* Fix check

* Add runtime condom middleware (#4741)

* Use CondomLimits struct
* Apply limits to wasm modules
* Add max_runtime_module_exports
* update runtime and serde_yaml
* Fix outdated UT

---------

Co-authored-by: Leo-Besancon <lb@massa.net>
Co-authored-by: Jean-François <jfm@laposte.net>

Signed-off-by: Jean-François <jfm@laposte.net>

* Update rust version to 1.81 (#4750)

* Update rust version to 1.81

* Cargo clippy & fmt pass

* Cargo clippy & fmt pass 2

* Cargo clippy & fmt pass 3

* Remove commented code

* Update actions/checkout to v4 (#4751)

* Update actions/checkout to v4

* Use dtolnay rust action instead of deprecated action-rs

* Use dtolnay rust action (2)

* Use dtolnay rust action (3)

* Update JamesIves action for ci doc

* Add typos config file (#4753)

* Add typos config file

* Fix typo errors

* Add typos to github CI

* Cargo fmt pass

* More typo fixes

* Fix slot index position status (#4736)

* Fix SlotIndexPosition from Future to Past in case slots_since returns an error

* Update ci.yml

* Validate that hd_cache_size >= snip_amount (#4732)

* Improve handling of HistorySearchResult for fetched executed_ops/denunciations (#4735)

* Improve handling of HistorySearchResult

* Fix CI

* Update ASC trigger comment (#4740)

* Update ASC trigger comment

* Add comment for ledger entry deletions

* fmt

* Fix join error message (was inconsistent with: VM controller thread)

* Cargo fmt pass

* Revert "Fix join error message (was inconsistent with: VM controller thread)" (#4756)

* Update tonic to version 0.12.3 and jsonrpsee to 0.24 (#4759)

* Update tonic to version 0.12.3 and jsonrpsee to 0.24

* Cargo fmt pass

* Test compilation fixes

* Set prost as a workspace dependency (will be easier to later upgrade

* Fix massa-api tests

* Cargo fmt pass

* Update jsonrpsee to 0.24.6

* Remove mut for builder

* Move dev local dependencies (commented)

* Change massa-sc-runtime & massa-proto-rs branches

* Early set operation id (#4758)

* Fix message for ExecutionManager (#4757)

* Add missing default gas cost for some abis (#4761)

* Add missing default gas cost for some abis

* Add xtask check gas cost definitions

* Update massa-sc-runtime in Cargo.lock

* Update massa-sc-runtime in Cargo.lock - round 2

* Add eclude list in xtask check_gas_cost_definitions

* Cargo fmt pass

* Add additional verification for v & s value in evm_signature_verify (#4762)

* Add additional verification for v & s value in evm_signature_verify

* Cargo fmt pass

* Add additional verification for v & s value in evm_get_pubkey_from_signature

* Cargo fmt pass

* Isolate some tests

* Cargo clippy fixes

* Reject signature if signature.s is high order

* Remove wasm_gas_costs.json and used a constant value for wasm operator (#4763)

* Remove wasm_gas_costs.json and used a constant value for wasm operator

* Cargo fmt pass

* Update massa-sc-runtime dependency

* Cargo clippy fix

* Add SC recursion limit (#4729)

* Asc message execution - requery message bytecode after each message execution (#4710)

* Requery bytecode

* cargo fmt

* fix call stack inconsistency (#4709)

* Improve async message checks (#4706)

* Improve async message checks

* Change checks for async messages

* Add unit tests

* Fix ledger change to take into account cancelled message balance change (#4715)

* Take again the speculative changes after async message cancellation

* use .apply() to merge the two LedgerChanges

* Fix: we cannot combine two ledger changes with apply

* avoid cloning the changes

* Remove comment

* Fix async msg same slot (#4718)

* fix open rpc spec (#4716)

* Add eliminated_new_messages in eliminated_msg

---------

Co-authored-by: Modship <yeskinokay@gmail.com>

* Add initial code for recursion limit

* Latest runtime

* Run CI on PRs based on mainnet_2_3

* fmt

* Fix config and add UTs

* Update scenarios_mandatories.rs

* Review comments (CI for all branches starting with "mainnet_" + comment)

* Update ci.yml

* Remove manual increment / decrement in interface implementation

* fmt + update sc_runtime + fix warning

* Update test

* Update constants.rs

* Updated execution config for tests

* Updated usize -> u16 for recursion counter and limits

* Update test comments

* Add comments regarding the needs of this limits

* Update sc-runtime branch

---------

Co-authored-by: Modship <yeskinokay@gmail.com>

* Update executed denunciations in StateChanges.apply method (#4764)

* Update executed denunciations in StateChanges.apply method

* Cargo fmt pass

* Consistent reset logic (#4765)

* Limit event msg size & event number per operation (#4768)

* Limit event msg size & event number per operation

* Fix typo

* Add unit tests for event data size & count

* Cargo fmt

* Cargo clippy fices

* Create event after all checks

* Cargo fmt

* Create event before using it

* Update jsonrpc to 0.24.7

* Update Cargo.lock

* Fix generate_event_wasmv1

* Remove the node_testnet_rewards_program_ownership_proof command

* Remove boostrap whitelist by default (#4773)

* Delete bootstrap_whitelist.json

* Add bootstrap_whitelist file for tests

* Use TempFile instead of hardcoded whitelist test file

* Update version to MAIN.2.5

* Update inaccurate error message (#4778)

* Update inaccurate error message

* Cargo fmt

* fix typos in comments

Signed-off-by: Jean-François <jfm@laposte.net>

* Fix execution_trace_nested TU (#4779)

* Fix execution_trace_nested TU

* Update comment

* Fix mainnet_2_3 CI

* Removed launch_wasmv1 gas costs (#4781)

* Fix async message updates and add unit test (#4766)

* Fix async message updates and add unit test

* Cargo fmt pass

* Cargo clippy fixes

* Cargo fmt pass

* Deferred calls (#4723)



---------

Co-authored-by: AurelienFT <aurelien.foucault@epitech.eu>
Co-authored-by: Damir Vodenicarevic <damipator@gmail.com>

* Test: Fix test_fetch_message (deferred_calls merge)

* deferred calls const gas cost

* Metrics: add network version metrics

* fix empty help msg

* Allow transfers to SC addresses (#4789)

* Allow transfers to SC

* Add unit test for 4 scenarios

* Use coherent PublicKeyFormat in parse_slice (#4783)

* Fix incorrect address hashing + add TU for evm_get_address_from_pubkey (#4785)

* deferred calls denominator

* Add massa event cache crate (#4769)

* Add massa event cache crate

* Add event cache controller into massa execution

* Cargo fmt

* Add event cache config in masa-node

* Minor fixes

* Cargo clippy fixes

* Cargo clippy fixes

* Add limits & security checks

* Add controller + manager

* Cargo fmt pass

* Fix check/clippy with --all-targets

* Better event cache clear at startup && better filtering

* Rename to config to max_call_stack_length

* Improve event cache filtering

* Avoid lock contention in controller::get_filtered_sc_output_events

* Improve comment

* Add query limit

* Add tick delay in event cache writer thread

* Use per address / operation id / is_error counters

* Cargo fmt

* typos fixes

* Cargo clippy fixes for tests

* Cargo fmt

* Add mock expectations + impl

* Cargo clippy for TU fixes

* Use MAX_EVENT_PER_OPERATION constant

* Unit test the filter optimisations

* Add more doc

* Cargo clippy fixes

* Use ..Default::default in TU

* Cargo clippy fix

* Use scope

* Use scope 2

* Remove tick_delay + directly mem::take struct

* Add tu for counter removal

* Add KeyKind in KeyBuilder

* Wait for condvar in wait_loop_event

* Removed unused lib

* Condvar wait fix

* Truncate event message in case of error

* fetch_message now returns NoInfo if message_id cannot be found (#4792)

* fetch_message now returns NoInfo if message_id cannot be found

* Add not for returning HistorySearchResult::Present

* Improve massa db reset doc (for dev) (#4796)

* Update massa-sc-runtime dependancy

* Fix tests (get_interface_version defaults to 1 for tests)

* Check before deleting the event cache dv (#4798)

* Fix entry count massa module cache (#4793)

* Update entry_count value

* Reset db instead of panic

* Fix deferred_call_changes in active history + fmt

* Always reset DB cache on new ModuleCache

* Do not panic if fails to destroy HD cache

* Update bootstrap timeouts (#4800)

* Update bootstrap timeouts

* Fix typos

* Remove StateChanges::apply moethod (#4801)

* Remove StateChanges::apply moethod

* typos fixes

* Add TU for delete_datastore_entries function (#4802)

* Add TU for delete_datastore_entries function

* Cargo clippy fix

* Features/tu async message reset 1 (#4804)

* Add send_message_then_reset_bytecode TU

* Code cleanup

* Minor updates

* Update address for sandbox mode

* Fix features for massa-sc-runtime (#4818)

* Fix db sanity check (#4820)

* FinalState(is_db_valid): add check for deferred calls keys

* fix address deserialization

* Fix: use Slot::from_bytes_key instead of SlotDeserializer

---------

Co-authored-by: Leo-Besancon <lb@massa.net>

* Reset operation id before executing async message (#4826)

* Feature/delete datastore entries tu 2 (#4806)

* Improve unit test for delete_datastore_entries

* Cargo clippy fixes

* Use end_prefix in get_entire_datastore_raw

* Fix event cache estimation for slot filters (#4819)

* Update event_cache.rs

* Add comment on unwrap_or(0) and unit test

* fix typo

* clippy fixes

* Fix versioning + Started + network restart (#4824)

* open rpc update (deferred calls) (#4830)

* Change max event data size from 512 to 1024 (#4829)

* Add versioning to mainnet update 2.5 (#4711)

* Asc message execution - requery message bytecode after each message execution (#4710)

* Requery bytecode

* cargo fmt

* Create MIP "MIP-0001-ASC-BugFix"

* Add versioning to asc execution context

* Asc message execution - requery message bytecode after each message execution (#4710)

* Requery bytecode

* cargo fmt

* Improve versioning

* fmt

* Update ci.yml

* Add ledger changes versioning

* Add runtime module versioning

* Add send_message versioning

* Add address category versioning

* Add Fix eliminated msg versioning

* add versioning test-exports mip

* fmt

* Add ledger delete_entry versioning

* Add versioning for Consistent expiry period

* Update speculative_async_pool.rs

* Add versioning to fees fix

* Add versioning for Fix amount remaining to slash 2

* Add versioning for LedgerChanges::Delete

* cargo fmt + check pass

* Add Condom Middleware versioning

* fmt

* Add versioning to max_recursive_calls_depth

* Fix deadlock

* Add versioning to Fix potential ledger keys boundaries issue

* Add versioning to gas costs

* Add versioning to Add additional verification for v & s value in evm_signature_verify

* Update runtime dep

* Add comment relative to versioning of Update executed denunciations in StateChanges.apply method

* Add versioning to Limit event msg size & event number per operation

* Fix versioning for interface implementation tests

* Fix test for max_event size

* Add versioning to Early set operation id

* Review comment: propagate error message and better match

* Review comment: clean out match

* Review comment: Add helper function to query versioned condom_limits in context

* Add comment

* Add dummy info for MIP_info

* Change .unwrap() to .expect() for get_block_timestamp

* Disable deferred calls abi if exec_comp = 0

* deferred calls const gas cost

* Add versioning for fix async message updates and add unit test

* Api : deferred call endpoint versioning

* Metrics: add network version metrics

* fix empty help msg

* Rollback deferred calls abi bail

* update runtime

* Allow transfers to SC addresses (#4789)

* Allow transfers to SC

* Add unit test for 4 scenarios

* Add versioning to Allow transfers to SC addresses

* Use coherent PublicKeyFormat in parse_slice (#4783)

* Fix incorrect address hashing + add TU for evm_get_address_from_pubkey (#4785)

* Add versioning to Use coherent PublicKeyFormat in parse_slice

* Add versioning for Fix incorrect address hashing

* cargo fmt + check

* Add versioning to massa event cache

* Fix versioning of massa event cache

* Reset cache if version becomes active (#4791)

* Reset cache if version becomes active

* add &mut + cargo fmt

* Update massa-execution-worker/src/execution.rs

Co-authored-by: Modship <yeskinokay@gmail.com>

* Update execution.rs

* Update hd_cache.rs

* Avoid taking a lock

* Rename + add info!() on cache reset

---------

Co-authored-by: Modship <yeskinokay@gmail.com>

* In HD DB reset(), Do not panic if fails to destroy HD cache

* Fix async message execution order in slot with no active version (#4799)

* Update the order of execution of async msg in  slot

* Remove debug print

* current_mip_version to get_status (#4803)

* Fix versioned TU

* metrics block version (#4832)

* add metric for current_active_version

* Metrics : remove counter for version 0 when current ver > 0

---------

Co-authored-by: modship <yeskinokay@gmail.com>
Co-authored-by: Sydhds <sydhds@gmail.com>

* update package to 2.6

* Use event counter for event limitation per op / asc / deferred_call (#4840)

* Use event_count_in_current_exec for event limitation

* Review fixes, and remove versioning of origin_operation_id order in prepare_op

* cargo clippy and clean bypass

* Datastore keys queries (JSON RPC / GRPC) (#4809)


---------

Co-authored-by: modship <yeskinokay@gmail.com>

* Allow bypass event limits (#4844)

* Allow certain router addresses to bypass event limits

* improve comment

* Apply review suggestions

* Apply review suggestions

* cargo fmt

* Directly put callstack as param for bypass_event_limitation

* Fix bypass event limitation (#4848)

* NewSlotExecutionOutputs filters (#4834)

* Deferred_call : cancelled gas (#4851)

* Deferred_call : remove expect when refound storage cost to sender

* deferred_calls : remove sub gas call when cancel it + clean metrics

* remove DEFERRED_CALL_TOTAL_REGISTERED from codebase

* fix grpc reflection

* unidirectional stream (#4853)

* limit error message length

Signed-off-by: Jean-François <jfm@laposte.net>

* fix UT

Signed-off-by: Jean-François <jfm@laposte.net>

* fmt

Signed-off-by: Jean-François <jfm@laposte.net>

* Add event cache db documentation (#4859)

* add metric for event cache and other missing metrics (#4863)

* add metric for event cache

* Fix sub metrics

* Add metrics for massa-db-worker change history sizes

* Add lru_cache_memory_usage metric

* Disable db metrics for tests

* Refactor db metrics & lock handling

* Review comment

* Add active history events len

---------

Co-authored-by: Leo-Besancon <lb@massa.net>

* Fix: thread_count sanity check in deferred_calls (#4867)

* Deferred call / async msg context error (#4869)

* Add massa-node and massa-client disclaimer (#4875)

* Add basic placeholder disclaimer

* Update panic message

* Update community charter and add license

* cargo fmt

* Update typos

* improve error message in case we are not in a tty

* fmt

* fix typos

* Ensure ASC msg ordering is deterministic (#4873)

* ensure_asc_msg_ordering is deterministic

* Revert "ensure_asc_msg_ordering is deterministic"

This reverts commit 3d114cc.

* Handle ASC ordering fix versioning

* Optimize take_async_batch

* add comment

* refactor: review comment + avoid double deref

* Simplify asc message fetching and deletion

* improvements from review comments

* cargo clippy fix

* Several bootstrap and rocksdb improvements (#4866)

* return SlotTooOld error in case of cache miss

* Increase default final_history_length

* do not fill cache

* Update rocksdb dep to 0.23

* Disable rocksdb cache in DB and on exposed iterator_cf

* Update massa_db.rs

* Add other memory related RocksDB options

* Change rocksdb, add jemalloc for linux, and update db options

* feature gate jemalloc

* cargo fmt

* cargo clippy fixes

* Rename, cleanup & use read cache only when necessary

* stream transfer history (#4861)



Co-authored-by: sydhds <sydhds@gmail.com>
Co-authored-by: Leo-Besancon <lb@massa.net>

* clippy

* update proto-rs branch

* fix duplicate

* test(ci): run tests with minimal required features, exclude gas_calibration

Signed-off-by: Jean-François <jfm@laposte.net>

* fix openrpcc spec

* add infos to GetAddressDatastoreKeysRequest

* Fix 4894 (#4899)

Fix #4894

* Mip massa client (#4898)

* cargo clippy fix

* Revert removal of Cargo.lock

* Remove versioning of MIP-0001 (#4908)

* Remove versioning of MIP-0001

* Remove block reward versioning (review comments)

* rename fetch_msgs_v1 to _opt, and cargo fmt

* Refactor api get_status versioning

* Remove API versioning

* Remove versionning from GRPC

* replaced dbg! with debug!

* Update get_interface_version impl

* Revert removal of `execution_component_version` in execution context

* pub(crate) instead of pub

* Update version to MAIN.3.0 (#4909)

* increase default value `readonly_queue_length` to 100 (#4910)

* Cargo update massa-sc-runtime

* Update CI triggers

* add deferred calls to transfers (#4907)

* add deferred calls to transfers

* Update massa-api-exports/src/execution.rs

Co-authored-by: Damir Vodenicarevic <damipator@gmail.com>

* Safe address handling

* Update execution.rs

* Update Cargo.lock

* cargo clippy fix

* Update Cargo.lock

* Update grpc new_slot_abi_call_stacks

* Update grpc new_slot_transfers

---------

Co-authored-by: Damir Vodenicarevic <damipator@gmail.com>

* Add comment for V0 event limit

* Update Cargo.lock

* CI: Target fixed cargo-nextest version

* Typos: exclude BlockIds

* Sync with latest massa-sc-runtime and massa-proto-rs on main

---------

Signed-off-by: Jean-François <jfm@laposte.net>
Co-authored-by: Leo-Besancon <lb@massa.net>
Co-authored-by: Sydhds <sydhds@gmail.com>
Co-authored-by: Damir Vodenicarevic <damipator@gmail.com>
Co-authored-by: Sydhds <sylvain.delhomme@gmail.com>
Co-authored-by: Jean-François <jfm@laposte.net>
Co-authored-by: AurelienFT <aurelien.foucault@epitech.eu>
Co-authored-by: Peterjah <ps@massa.net>

Author: 7 people

.github/workflows/ci.yml

@@ -4,7 +4,7 @@ on:
   push:
     branches: [main, staging, trying]
   pull_request:
-    branches: [main, 'mainnet_*']
+    branches: [main, 'main_*', 'mainnet_*']
     types:
     - opened
     - reopened
@@ -224,8 +224,8 @@ jobs:
           version: "23.x"
           include-pre-releases: false
           repo-token: ${{ secrets.GITHUB_TOKEN }}
-      - run: cargo install cargo-nextest --locked
-      - run: cargo nextest run --retries 10 --profile ci --all-features
+      - run: cargo install cargo-nextest --locked --version 0.9.97-b.2
+      - run: cargo nextest run --retries 10 --profile ci --features=execution-trace,dump-block
       - run: cargo clean
       - run: cargo test --doc
       - uses: codecov/codecov-action@v4