docs(l1): adding healing documentation (#4708)

**Motivation**

During our development we developed the healing algorithm based on some
assumptions, this adds documentation for it.

**Description**

- Uploads documentation relating to healing in snap sync and account
deletion in ethereum.

---------

Co-authored-by: Tomás Grüner <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Javier Chatruc <[email protected]>

Author: 4 people

.gitignore

@@ -74,3 +74,6 @@ GEMINI.md
 
 # RKYV
 *.bin
+
+# Documentation Graphs
+!docs/internal/l1/healing/*.svg

docs/SUMMARY.md

@@ -24,6 +24,8 @@
   - [Databases]()
   - [Networking](./l1/fundamentals/networking.md)
   - [Sync modes](./l1/fundamentals/sync_modes.md)
+  - [Snap sync internals](./internal/l1/healing.md)
+    - [Can an account disappear from Ethereum's state trie?](./internal/l1/delete_accounts.md)
   - [Pruning]()
 
 # Ethrex for L2 chains

docs/internal/l1/delete_accounts.md

@@ -0,0 +1,29 @@
+## Can you delete accounts in Ethereum? Yes
+
+### How it happens
+
+Ethereum accounts are broadly divided into two categories:
+- Externally Owned Accounts (EOA): accounts for general users to transfer eth and call contracts.
+- Contracts: which execute code and store data.
+
+Creating EOA is done through sending ETH into a new address, at which point the account is created and added into the state trie.
+
+Creating a contract can be done through the CREATE and [CREATE2](https://eips.ethereum.org/EIPS/eip-1014) opcode. Notably, those opcodes check that the account is created at an address where the code is empty and the nonce is zero, but **it doesn't check balance**. As such, a contract can be created through taking over an existing account.
+
+During the creating of a contract, the `init_code` is run which can include the [self destruct opcode](https://eips.ethereum.org/EIPS/eip-6780) that deletes the contract in the same transaction it was created. Normally, this deletes an account that was created in the same transaction (because contracts are usually created over empty accounts) but in this case the account already existed because it already had some balance. This is the only edge case in which an account can go from existing to non-existing from one block to another after the Cancun fork.
+
+### How we found it
+
+Snap-sync is broadly divided into two stages:
+- Downloading the leaves of the state (account states) and storage tries (storage slots)
+- Healing (reconciling the state). 
+
+Healing is needed because the leaves can be downloaded from disparate blocks, and to "fix" only the nodes of the trie that changed between nodes. [In depth explanation](https://www.notion.so/lambdaclass/Healing-Algorithm-Explanation-and-Documentation-269b9462471380e4a275edd77c8b5dc5?source=copy_link).
+
+We were working under the assumption that accounts were never deleted, so we adopted some specific optimizations. During the state healing stage every account that was "healed" was added into a list of accounts that needed to be checked for storage healing. When healing the storage of those accounts the algorithm requested their account states and expected them to be there to see if they had any storage that needed healing. This lead to the storage healing threads panicking when they failed to find the account that was deleted.
+
+During the test of snapsync mainnet, we started seeing that storage healing was panicking, so we added some logs to see what account hashes were being accessed and when where they healed vs accessed. Exploring the database we saw that the offending account was present in a previous state and missing in the next one, with the corresponding merkle proof matching the block state root. Originally we suspected a reorg, but searching the blocks we saw they were finalized in the chain. 
+
+The account state present indicated an account with nonce 0, no code and no storage but with balance. We didn't have access to the account address, as the state trie only stores the hash of the account address so we turned to another strategy to find it. Using [etherscan's API](https://docs.etherscan.io/api-endpoints/accounts#get-internal-transactions-by-block-range) allowing to search internal transactions from a block range, we explored the range where we knew the account existed in the state trie. Hashing all of the `to` and `from` of the transactions [we found the transaction](https://etherscan.io/tx/0xf23b2c233410141cda0c6d24f21f0074c494565bfd54ce008c5ce1b30b23b0da) that deleted the account with a self destruct. Despite the account becoming a contract just during that transaction, we saw that 900 blocks before it was [created with a transfer](https://etherscan.io/tx/0xbc9f52ba45a6915878318be944cb20bd3bb1bbf36b2ce8ff5e6575ce1689f1b6). The result of the self destruct was the transfer of 0.044 ETH from one account to another.
+
+The specific transaction that created the contract: https://etherscan.io/tx/0xf23b2c233410141cda0c6d24f21f0074c494565bfd54ce008c5ce1b30b23b0da

docs/internal/l1/healing.md

@@ -0,0 +1,94 @@
+# Healing Algorithm Explanation and Documentation (Before Path Based)
+
+Healing is the last step of Snap Sync. Snap begins the downloading of the state and storage tries by downloading the leaves (account states and storage slots), and from those leaves we reconstruct the intermediate nodes (branches and extension). Afterwards we may be left with a malformed trie, as that step will resume the download of leaves with a new state root if the old one times out.
+
+The purpose of the healing algorithm is “heal” that trie so that it ends up in a consistent state.
+
+# Healing Conceptually
+
+The malformed trie is going to have large sections of the trie which are in a correct state, as we had all of the leaves in that sections and those accounts haven’t been modified in the blocks that happened concurrently to the snapsync algorithm.
+
+![Image of a trie, where the root node is in red, indicating that it’s in an incorrect state. It points to two branches, one is correct and one was computed from faulty data, and such doesn’t exist in the latest block](healing/Example_1_Step_0.svg)
+
+Example of a trie where 3 leaves where downloaded in block 1 and 1 was downloaded in block 2. The trie root is different from the state root of block 2, as one of the leaf nodes was modified in block 2.
+
+The algorithm attempts to rebuild the trie through downloading the missing nodes, starting from the top. If the node is present in the database that means that we have that and all of their child nodes present in the database. If not, we download the node and check if the children of the root are present, applying the algorithm recursively.
+
+![Iteration 1 of algorithm](healing/Example_1_Step_1.svg)
+
+Iteration 1 of algorithm
+
+![Iteration 2 of algorithm](healing/Example_1_Step_2.svg)
+
+Iteration 2 of algorithm
+
+![Iteration 3 of algorithm](healing/Example_1_Step_3.svg)
+
+Iteration 3 of algorithm
+
+![Final state of trie after healing](healing/Example_1_Step_4.svg)
+
+Final state of trie after healing
+
+# Implementation
+
+The algorithm is implemented in ethrex currently in `crates/networking/p2p/sync/state_healings.rs` and `crates/networking/p2p/sync/storage_healing.rs`. All of our code examples are from the account state trie.
+
+### API
+
+The API used is the ethereum capability snap/1, documented at https://github.com/ethereum/devp2p/blob/master/caps/snap.md and for healing the only method used is `GetTrieNodes`. This method allows us to ask our peers for nodes in a trie. We ask the nodes by **path** to the node, not by hash. 
+
+```rust
+pub struct GetTrieNodes {    
+    pub id: u64,    
+    pub root_hash: H256,    
+    // [[acc_path, slot_path_1, slot_path_2,...]...]    
+    // The paths can be either full paths (hash) or 
+    // only the partial path (compact-encoded nibbles)    
+    pub paths: Vec<Vec<Bytes>>,    
+    pub bytes: u64,
+}
+```
+
+### Staleness
+
+The spec allows the nodes to stop responding if the request is older than 128 blocks. In that case, the response to the `GetTrieNodes` will be empty. As such, our algorithm checks periodically if the block is stale, and stops executing. In that scenario, we must be sure that the we leave the storage in a consistent state at any given time and doesn’t break our invariants.
+
+```rust
+// Current Staleness logic code
+// We check with a clock if we are stale        
+if !is_stale && current_unix_time() > staleness_timestamp {
+    info!("state healing is stale");            
+    is_stale = true;       
+}
+// We make sure that we have stored everything that we need to the database
+if is_stale && nodes_to_heal.is_empty() && inflight_tasks == 0 {
+  info!("Finished inflight tasks");            
+  db_joinset.join_all().await;            
+  break;
+}
+```
+
+### Membatch
+
+Currently, our algorithm has an invariant, which is that if we have a node in storage we have its and all of its children are present. Therefore, when we download for a node if some of it’s children are missing we can’t immediately store it on disk. Our implementation currently stores the nodes in temporary structure called membatch, which stores the node and how many of it’s children are missing. When a child gets stored, we reduce the counter of missing children of the parent. If that numbers reaches 0, we write the parent to the database.
+
+In code, the membatch is current `HashMap<Nibbles, MembatchEntryValue>` with the value being the following struct 
+
+```rust
+pub struct MembatchEntryValue {
+    /// The node to be flushed into storage
+    node: Node,
+    /// How many of the nodes that are child of this are not in storage
+    children_not_in_storage_count: u64,
+    /// Which is the parent of this node
+    parent_path: Nibbles,
+}
+```
+
+## Known Optimization Issues
+
+- Membatch gets cleared between iterations, while it could be preserved and the hash checked.
+- When checking if a child is present in storage, we can also check if it’s in the membatch. If it is, we can skip that download and act like we have immediately downloaded that node.
+- Membatch is currently a `HashMap`, a `BTreeMap` or other structures may be faster in real use.
+- Storage healing receives as a parameter a list of accounts that need to be healed and it has get their state before it can run. Doing those reads could be more efficient.