Merge pull request 2140-dev#311 from nyonson/break-up-core

Author: rustaceanrob

README.md

@@ -135,18 +135,6 @@ just integrate
 
 If you do not have `bitcoind` installed, you may simply run `just integrate` and it will be installed for you in the `build` folder.
 
-## Project Layout
-
-`chain`: Contains all logic for syncing block headers, filter headers, filters, parsing blocks. Also contains preset checkpoints for Signet, Regtest, and Bitcoin networks. Notable files: `chain.rs`
-
-`core`: Organizes the primary user-facing components of the API. This includes both the `Node` and the `Client` that all developers will interact with, as well as the `NodeBuilder`. Importantly includes `peer_map.rs`, which is the primary file that handles peer threads by sending messages, persisting new peers, banning peers, and managing peer task handles. `node.rs` is the main application loop, responsible for driving the node actions. Notable files: `node.rs`, `peer_map.rs`, `builder.rs`, `client.rs`
-
-`db`: Defines how data must be persisted with `traits.rs`, and includes some opinionated defaults for database components.
-
-`filters`: Additional structures for managing compact filter headers and filters, used by `chain.rs`
-
-`network`: Opens and closes connections, handles encryption and decryption of messages, generates messages, parses messages, times message response times, performs DNS lookups. Notable files: `peer.rs`, `reader.rs`, `parsers.rs`, `outbound_messages.rs`
-
 ## Contributing
 
 Please read [CONTRIBUTING.md](./CONTRIBUTING.md) to get started.

doc/DETAILS.md

@@ -81,18 +81,23 @@ The wallet required 12 block downloads, and took 5 minutes.
 
 This section details what behavior to expect when using Kyoto, and why such decisions were made.
 
-### Peer Selection
+## Peer Selection
 
 Kyoto will first connect to all of the configured peers to maintain the connection requirement, and will use peers gleaned from the peer-to-peer gossip thereafter. If no peers are configured when building the node, and no peers are in the database, Kyoto will resort to DNS. Kyoto will not select a peer of the same netgroup (`/16`) as a previously connected peer. When selecting a new peer from the database, a random preference will be selected between a "new" peer and a peer that has been "tried." Rational is derived from [this research](https://www.ethanheilman.com/p/eclipse/index.html)
 
-### Block Headers and Storage
+## Block Headers and Storage
 
 Kyoto expects users to adopt some form of persistence between sessions when it comes to block header data. Reason being, Kyoto emits block headers that have been reorganized back to the client in such an event. To do so, in a rare but potential circumstance where the client has shut down on a stale tip, one that is reorganized in the future, Kyoto may use the header persistence to load the older chain into memory. Further, this allows the memory footprint of storing headers in a chain structure to remain small. Kyoto has a soft limit of 20,000 headers in memory at any given time, and if the chain representation exceeds that, Kyoto has a reliable backend to move the excess of block headers. To compensate for this, Kyoto only expects some generic datastore, and does not care about how persistence is implemented.
 
-### Filters
+## Filters
 
 Block filters for a full block may be 300-400 bytes, and may be needless overhead if scripts are revealed "into the future" for the underlying wallet. Full filters are checked for matches as they are downloaded, but are discarded thereafter. As a result, if the user adds scripts that are believe to be included in blocks _in the past_, Kyoto will have to redownload the filters. But if the wallet has up to date information, a revealing a new script is guaranteed to have not been used. This memory tradeoff was deemed worthwhile, as it is expected rescanning will only occur for recovery scenarios.
 
-### Structure
+## Structure
+
+* `chain`: Contains all logic for syncing block headers, filter headers, filters, parsing blocks. Also contains preset checkpoints for Signet, Regtest, and Bitcoin networks. Notable files: `chain.rs`
+* `db`: Defines how data must be persisted with `traits.rs`, and includes some opinionated defaults for database components.
+* `filters`: Additional structures for managing compact filter headers and filters, used by `chain.rs`
+* `network`: Opens and closes connections, handles encryption and decryption of messages, generates messages, parses messages, times message response times, performs DNS lookups. Notable files: `peer.rs`, `reader.rs`, `parsers.rs`, `outbound_messages.rs`
 
 ![Layout](https://github.com/user-attachments/assets/21280bb4-aa88-4e11-9223-aed35a885e99)

example/managed.rs

@@ -2,8 +2,8 @@
 //! many possible scripts to check. Enable the `filter-control` feature to check filters
 //! manually in your program.
 
-use kyoto::core::messages::Event;
-use kyoto::{chain::checkpoints::HeaderCheckpoint, core::builder::NodeBuilder, Client};
+use kyoto::messages::Event;
+use kyoto::{builder::NodeBuilder, chain::checkpoints::HeaderCheckpoint, Client};
 use kyoto::{AddrV2, Address, BlockHash, LogLevel, Network, ServiceFlags, TrustedPeer};
 use std::collections::HashSet;
 use std::{net::Ipv4Addr, str::FromStr};

example/rescan.rs

@@ -2,7 +2,7 @@
 //! the filters for inclusions of these scripts and download the relevant
 //! blocks.
 
-use kyoto::{chain::checkpoints::HeaderCheckpoint, core::builder::NodeBuilder};
+use kyoto::{builder::NodeBuilder, chain::checkpoints::HeaderCheckpoint};
 use kyoto::{Address, Client, Event, Network};
 use std::collections::HashSet;
 use std::str::FromStr;

example/signet.rs

@@ -1,6 +1,6 @@
 //! Usual sync on Signet.
 
-use kyoto::{chain::checkpoints::HeaderCheckpoint, core::builder::NodeBuilder};
+use kyoto::{builder::NodeBuilder, chain::checkpoints::HeaderCheckpoint};
 use kyoto::{AddrV2, Address, Client, Event, Log, Network, ServiceFlags, TrustedPeer};
 use std::collections::HashSet;
 use std::{

example/testnet4.rs

@@ -1,6 +1,6 @@
 //! Usual sync on Testnet.
 
-use kyoto::{chain::checkpoints::HeaderCheckpoint, core::builder::NodeBuilder};
+use kyoto::{builder::NodeBuilder, chain::checkpoints::HeaderCheckpoint};
 use kyoto::{Address, Client, Event, Log, Network, PeerStoreSizeConfig, TrustedPeer};
 use std::collections::HashSet;
 use std::{net::Ipv4Addr, str::FromStr};

src/core/broadcaster.rs src/broadcaster.rssrc/core/broadcaster.rs renamed to src/broadcaster.rs

@@ -5,7 +5,7 @@ use bitcoin::Network;
 #[cfg(not(feature = "filter-control"))]
 use bitcoin::ScriptBuf;
 
-use super::{client::Client, config::NodeConfig, node::Node, FilterSyncPolicy};
+use super::{client::Client, config::NodeConfig, node::Node};
 #[cfg(feature = "database")]
 use crate::db::error::SqlInitializationError;
 #[cfg(feature = "database")]
@@ -15,7 +15,7 @@ use crate::{
     chain::checkpoints::HeaderCheckpoint,
     db::traits::{HeaderStore, PeerStore},
 };
-use crate::{ConnectionType, LogLevel, PeerStoreSizeConfig, TrustedPeer};
+use crate::{ConnectionType, FilterSyncPolicy, LogLevel, PeerStoreSizeConfig, TrustedPeer};
 
 #[cfg(feature = "database")]
 /// The default node returned from the [`NodeBuilder`](crate::core).

src/core/builder.rs src/builder.rssrc/core/builder.rs renamed to src/builder.rs

@@ -4,8 +4,8 @@ use bitcoin::BlockHash;
 use tokio::time::Instant;
 
 #[cfg(feature = "filter-control")]
-use crate::core::messages::BlockRequest;
-use crate::core::messages::BlockSender;
+use crate::messages::BlockRequest;
+use crate::messages::BlockSender;
 
 const SPAM_LIMIT: u64 = 5;
 

src/chain/block_queue.rs

@@ -20,18 +20,13 @@ use super::{
     HeightMonitor,
 };
 #[cfg(feature = "filter-control")]
-use crate::core::error::FetchBlockError;
+use crate::error::FetchBlockError;
 #[cfg(feature = "filter-control")]
-use crate::core::messages::BlockRequest;
+use crate::messages::BlockRequest;
 #[cfg(feature = "filter-control")]
 use crate::IndexedFilter;
 use crate::{
     chain::header_batch::HeadersBatch,
-    core::{
-        dialog::Dialog,
-        error::HeaderPersistenceError,
-        messages::{Event, Warning},
-    },
     db::traits::HeaderStore,
     filters::{
         cfheader_batch::CFHeaderBatch,
@@ -41,6 +36,11 @@ use crate::{
         Filter, CF_HEADER_BATCH_SIZE, FILTER_BATCH_SIZE,
     },
     IndexedBlock,
+    {
+        dialog::Dialog,
+        error::HeaderPersistenceError,
+        messages::{Event, Warning},
+    },
 };
 
 const MAX_REORG_DEPTH: u32 = 5_000;
@@ -902,11 +902,11 @@ mod tests {
             checkpoints::{HeaderCheckpoint, HeaderCheckpoints},
             error::HeaderSyncError,
         },
-        core::{
+        filters::cfheader_chain::AppendAttempt,
+        {
             dialog::Dialog,
             messages::{Event, Log, Warning},
         },
-        filters::cfheader_chain::AppendAttempt,
     };
 
     use super::{Chain, HeightMonitor};