Skip to content

Commit 24034ab

Browse files
prestwichclaude
andauthored
refactor(storage): introduce connector-based architecture (#35)
* feat(storage): add configuration and builder for flexible storage instantiation Add configuration layer to signet-storage crate enabling three deployment modes: - Hot-only (MDBX only, no cold storage) - Hot + Cold MDBX (both hot and cold use MDBX) - Hot MDBX + Cold SQL (hot uses MDBX, cold uses PostgreSQL/SQLite) Changes: - Add config module with StorageMode enum and environment variable parsing - Add builder module with StorageBuilder and StorageInstance types - Add StorageInstance enum to safely distinguish hot-only vs unified storage - Update error types to support configuration and backend errors - Add factory methods and into_hot() to UnifiedStorage - Add sql and sqlite features for PostgreSQL and SQLite support - Add comprehensive unit tests for configuration and builder The implementation is fully backward compatible with existing code. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * chore: bump version to 0.6.4 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor(storage): introduce connector-based architecture Replace mode-based builder with trait-based connectors to decouple backend instantiation. Adds HotConnect and ColdConnect traits, unified MdbxConnector implementing both, and SqlConnector for auto-detecting PostgreSQL/SQLite. Removes StorageMode enum and StorageInstance enum, simplifying the API to always return UnifiedStorage. Users now pass connector objects to the builder's fluent API (.hot().cold().build()), enabling flexible backend composition without tight coupling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(storage): simplify EitherCold with dispatch_async! macro Reduces EitherCold boilerplate from 286 lines to 212 lines by introducing a dispatch_async! macro that generates the repetitive match-and-forward pattern for all 16 ColdStorage trait methods. Method signatures remain explicit for documentation while the async match blocks are generated by the macro, maintaining zero runtime overhead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(storage): use connector-specific errors, remove Config variants Creates dedicated error types for connector initialization: - MdbxConnectorError in cold-mdbx crate - SqlConnectorError in cold-sql crate Removes Config(String) variants from MdbxError and SqlColdError, which were only used for from_env() methods. The new connector error types are more specific and properly handle missing environment variables. ConfigError now uses From implementations to convert from connector errors, simplifying error handling in the builder. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(storage): desugar async fn in ColdConnect trait to specify Send bound Addresses async-fn-in-trait clippy lint by explicitly desugaring async fn to return impl Future + Send. This makes the Send bound explicit and allows callers to know the future can be sent across threads. Changes: - Desugar ColdConnect::connect to return impl Future + Send - Add #[allow(clippy::manual_async_fn)] to MDBX and EitherCold impls - Fix redundant closures in error mapping - Add Debug derive to StorageBuilder - Add serial_test to workspace for test isolation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent 1a15300 commit 24034ab

File tree

16 files changed

+815
-8
lines changed

16 files changed

+815
-8
lines changed

Cargo.toml

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ members = ["crates/*"]
33
resolver = "2"
44

55
[workspace.package]
6-
version = "0.6.3"
6+
version = "0.6.4"
77
edition = "2024"
88
rust-version = "1.92"
99
authors = ["init4"]
@@ -35,13 +35,13 @@ incremental = false
3535

3636
[workspace.dependencies]
3737
# internal
38-
signet-hot = { version = "0.6.3", path = "./crates/hot" }
39-
signet-hot-mdbx = { version = "0.6.3", path = "./crates/hot-mdbx" }
40-
signet-cold = { version = "0.6.3", path = "./crates/cold" }
41-
signet-cold-mdbx = { version = "0.6.3", path = "./crates/cold-mdbx" }
42-
signet-cold-sql = { version = "0.6.3", path = "./crates/cold-sql" }
43-
signet-storage = { version = "0.6.3", path = "./crates/storage" }
44-
signet-storage-types = { version = "0.6.3", path = "./crates/types" }
38+
signet-hot = { version = "0.6.4", path = "./crates/hot" }
39+
signet-hot-mdbx = { version = "0.6.4", path = "./crates/hot-mdbx" }
40+
signet-cold = { version = "0.6.4", path = "./crates/cold" }
41+
signet-cold-mdbx = { version = "0.6.4", path = "./crates/cold-mdbx" }
42+
signet-cold-sql = { version = "0.6.4", path = "./crates/cold-sql" }
43+
signet-storage = { version = "0.6.4", path = "./crates/storage" }
44+
signet-storage-types = { version = "0.6.4", path = "./crates/types" }
4545

4646
# External, in-house
4747
signet-libmdbx = { version = "0.8.0" }
@@ -66,6 +66,7 @@ parking_lot = "0.12.5"
6666
rand = "0.9.2"
6767
rayon = "1.10"
6868
serde = { version = "1.0.217", features = ["derive"] }
69+
serial_test = "3.3"
6970
tempfile = "3.20.0"
7071
thiserror = "2.0.18"
7172
tokio = { version = "1.45.0", features = ["full"] }

crates/cold-mdbx/src/connector.rs

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
//! MDBX storage connector.
2+
//!
3+
//! Unified connector that can open both hot and cold MDBX databases.
4+
5+
use crate::{MdbxColdBackend, MdbxColdError};
6+
use signet_cold::ColdConnect;
7+
use signet_hot::HotConnect;
8+
use signet_hot_mdbx::{DatabaseArguments, DatabaseEnv, MdbxError};
9+
use std::path::PathBuf;
10+
11+
/// Errors that can occur when initializing MDBX connectors.
12+
#[derive(Debug, thiserror::Error)]
13+
pub enum MdbxConnectorError {
14+
/// Missing environment variable.
15+
#[error("missing environment variable: {0}")]
16+
MissingEnvVar(&'static str),
17+
18+
/// Hot storage initialization failed.
19+
#[error("hot storage initialization failed: {0}")]
20+
HotInit(#[from] MdbxError),
21+
22+
/// Cold storage initialization failed.
23+
#[error("cold storage initialization failed: {0}")]
24+
ColdInit(#[from] MdbxColdError),
25+
}
26+
27+
/// Connector for MDBX storage (both hot and cold).
28+
///
29+
/// This unified connector can open MDBX databases for both hot and cold storage.
30+
/// It holds the path and database arguments, which can include custom geometry,
31+
/// sync mode, max readers, and other MDBX-specific configuration.
32+
///
33+
/// # Example
34+
///
35+
/// ```ignore
36+
/// use signet_hot_mdbx::{MdbxConnector, DatabaseArguments};
37+
///
38+
/// // Hot storage with custom args
39+
/// let hot = MdbxConnector::new("/tmp/hot")
40+
/// .with_db_args(DatabaseArguments::new().with_max_readers(1000));
41+
///
42+
/// // Cold storage with default args
43+
/// let cold = MdbxConnector::new("/tmp/cold");
44+
/// ```
45+
#[derive(Debug, Clone)]
46+
pub struct MdbxConnector {
47+
path: PathBuf,
48+
db_args: DatabaseArguments,
49+
}
50+
51+
impl MdbxConnector {
52+
/// Create a new MDBX connector with default database arguments.
53+
pub fn new(path: impl Into<PathBuf>) -> Self {
54+
Self { path: path.into(), db_args: DatabaseArguments::new() }
55+
}
56+
57+
/// Set custom database arguments.
58+
///
59+
/// This allows configuring MDBX-specific settings like geometry, sync mode,
60+
/// max readers, and exclusive mode.
61+
#[must_use]
62+
pub const fn with_db_args(mut self, db_args: DatabaseArguments) -> Self {
63+
self.db_args = db_args;
64+
self
65+
}
66+
67+
/// Get a reference to the path.
68+
pub fn path(&self) -> &std::path::Path {
69+
&self.path
70+
}
71+
72+
/// Get a reference to the database arguments.
73+
pub const fn db_args(&self) -> &DatabaseArguments {
74+
&self.db_args
75+
}
76+
77+
/// Create a connector from environment variables.
78+
///
79+
/// Reads the path from the specified environment variable.
80+
///
81+
/// # Example
82+
///
83+
/// ```ignore
84+
/// use signet_cold_mdbx::MdbxConnector;
85+
///
86+
/// let hot = MdbxConnector::from_env("SIGNET_HOT_PATH")?;
87+
/// let cold = MdbxConnector::from_env("SIGNET_COLD_PATH")?;
88+
/// ```
89+
pub fn from_env(env_var: &'static str) -> Result<Self, MdbxConnectorError> {
90+
let path: PathBuf =
91+
std::env::var(env_var).map_err(|_| MdbxConnectorError::MissingEnvVar(env_var))?.into();
92+
Ok(Self::new(path))
93+
}
94+
}
95+
96+
impl HotConnect for MdbxConnector {
97+
type Hot = DatabaseEnv;
98+
type Error = MdbxError;
99+
100+
fn connect(&self) -> Result<Self::Hot, Self::Error> {
101+
self.db_args.clone().open_rw(&self.path)
102+
}
103+
}
104+
105+
impl ColdConnect for MdbxConnector {
106+
type Cold = MdbxColdBackend;
107+
type Error = MdbxColdError;
108+
109+
#[allow(clippy::manual_async_fn)]
110+
fn connect(&self) -> impl std::future::Future<Output = Result<Self::Cold, Self::Error>> + Send {
111+
// MDBX open is sync, but wrapped in async for trait consistency
112+
// Opens read-write and creates tables
113+
let path = self.path.clone();
114+
async move { MdbxColdBackend::open_rw(&path) }
115+
}
116+
}

crates/cold-mdbx/src/lib.rs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,4 +46,7 @@ pub use tables::{
4646
mod backend;
4747
pub use backend::MdbxColdBackend;
4848

49+
mod connector;
50+
pub use connector::{MdbxConnector, MdbxConnectorError};
51+
4952
pub use signet_hot_mdbx::{DatabaseArguments, DatabaseEnvKind};

crates/cold-sql/src/connector.rs

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
//! SQL cold storage connector.
2+
3+
use crate::{SqlColdBackend, SqlColdError};
4+
use signet_cold::ColdConnect;
5+
6+
/// Errors that can occur when initializing SQL connectors.
7+
#[derive(Debug, thiserror::Error)]
8+
pub enum SqlConnectorError {
9+
/// Missing environment variable.
10+
#[error("missing environment variable: {0}")]
11+
MissingEnvVar(&'static str),
12+
13+
/// Cold storage initialization failed.
14+
#[error("cold storage initialization failed: {0}")]
15+
ColdInit(#[from] SqlColdError),
16+
}
17+
18+
/// Connector for SQL cold storage (PostgreSQL or SQLite).
19+
///
20+
/// Automatically detects the database type from the URL:
21+
/// - URLs starting with `postgres://` or `postgresql://` use PostgreSQL
22+
/// - URLs starting with `sqlite:` use SQLite
23+
///
24+
/// # Example
25+
///
26+
/// ```ignore
27+
/// use signet_cold_sql::SqlConnector;
28+
///
29+
/// // PostgreSQL
30+
/// let pg = SqlConnector::new("postgres://localhost/signet");
31+
/// let backend = pg.connect().await?;
32+
///
33+
/// // SQLite
34+
/// let sqlite = SqlConnector::new("sqlite::memory:");
35+
/// let backend = sqlite.connect().await?;
36+
/// ```
37+
#[cfg(any(feature = "sqlite", feature = "postgres"))]
38+
#[derive(Debug, Clone)]
39+
pub struct SqlConnector {
40+
url: String,
41+
}
42+
43+
#[cfg(any(feature = "sqlite", feature = "postgres"))]
44+
impl SqlConnector {
45+
/// Create a new SQL connector.
46+
///
47+
/// The database type is detected from the URL prefix.
48+
pub fn new(url: impl Into<String>) -> Self {
49+
Self { url: url.into() }
50+
}
51+
52+
/// Get a reference to the connection URL.
53+
pub fn url(&self) -> &str {
54+
&self.url
55+
}
56+
57+
/// Create a connector from environment variables.
58+
///
59+
/// Reads the SQL URL from the specified environment variable.
60+
///
61+
/// # Example
62+
///
63+
/// ```ignore
64+
/// use signet_cold_sql::SqlConnector;
65+
///
66+
/// let cold = SqlConnector::from_env("SIGNET_COLD_SQL_URL")?;
67+
/// ```
68+
pub fn from_env(env_var: &'static str) -> Result<Self, SqlConnectorError> {
69+
let url = std::env::var(env_var).map_err(|_| SqlConnectorError::MissingEnvVar(env_var))?;
70+
Ok(Self::new(url))
71+
}
72+
}
73+
74+
#[cfg(any(feature = "sqlite", feature = "postgres"))]
75+
impl ColdConnect for SqlConnector {
76+
type Cold = SqlColdBackend;
77+
type Error = SqlColdError;
78+
79+
fn connect(&self) -> impl std::future::Future<Output = Result<Self::Cold, Self::Error>> + Send {
80+
let url = self.url.clone();
81+
async move { SqlColdBackend::connect(&url).await }
82+
}
83+
}

crates/cold-sql/src/lib.rs

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,11 @@ mod backend;
4747
#[cfg(any(feature = "sqlite", feature = "postgres"))]
4848
pub use backend::SqlColdBackend;
4949

50+
#[cfg(any(feature = "sqlite", feature = "postgres"))]
51+
mod connector;
52+
#[cfg(any(feature = "sqlite", feature = "postgres"))]
53+
pub use connector::{SqlConnector, SqlConnectorError};
54+
5055
/// Backward-compatible alias for [`SqlColdBackend`] when using SQLite.
5156
#[cfg(feature = "sqlite")]
5257
pub type SqliteColdBackend = SqlColdBackend;

crates/cold/src/connect.rs

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
//! Connection traits for cold storage backends.
2+
3+
use crate::ColdStorage;
4+
5+
/// Connector trait for cold storage backends.
6+
///
7+
/// Abstracts the connection/opening process for cold storage, allowing
8+
/// different backends to implement their own initialization logic.
9+
pub trait ColdConnect {
10+
/// The cold storage type produced by this connector.
11+
type Cold: ColdStorage;
12+
13+
/// The error type returned by connection attempts.
14+
type Error: std::error::Error + Send + Sync + 'static;
15+
16+
/// Connect to the cold storage backend asynchronously.
17+
///
18+
/// Async to support backends that require async initialization
19+
/// (like SQL connection pools).
20+
fn connect(&self) -> impl std::future::Future<Output = Result<Self::Cold, Self::Error>> + Send;
21+
}

crates/cold/src/lib.rs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,9 @@ pub use stream::{StreamParams, produce_log_stream_default};
161161
mod traits;
162162
pub use traits::{BlockData, ColdStorage, LogStream};
163163

164+
pub mod connect;
165+
pub use connect::ColdConnect;
166+
164167
/// Task module containing the storage task runner and handles.
165168
pub mod task;
166169
pub use task::{ColdStorageHandle, ColdStorageReadHandle, ColdStorageTask};

crates/hot/src/connect.rs

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
//! Connection traits for hot storage backends.
2+
3+
use crate::model::HotKv;
4+
5+
/// Connector trait for hot storage backends.
6+
///
7+
/// Abstracts the connection/opening process for hot storage, allowing
8+
/// different backends to implement their own initialization logic.
9+
pub trait HotConnect {
10+
/// The hot storage type produced by this connector.
11+
type Hot: HotKv;
12+
13+
/// The error type returned by connection attempts.
14+
type Error: std::error::Error + Send + Sync + 'static;
15+
16+
/// Connect to the hot storage backend.
17+
///
18+
/// Synchronous since most hot storage backends use sync initialization.
19+
fn connect(&self) -> Result<Self::Hot, Self::Error>;
20+
}

crates/hot/src/lib.rs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,9 @@
105105
#[cfg(any(test, feature = "test-utils"))]
106106
pub mod conformance;
107107

108+
pub mod connect;
109+
pub use connect::HotConnect;
110+
108111
pub mod db;
109112
pub use db::{HistoryError, HistoryRead, HistoryWrite};
110113

crates/storage/Cargo.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,18 @@ rustdoc-args = ["--cfg", "docsrs"]
1717

1818
[dependencies]
1919
signet-cold.workspace = true
20+
signet-cold-mdbx.workspace = true
21+
signet-cold-sql = { workspace = true, optional = true }
2022
signet-hot.workspace = true
23+
signet-hot-mdbx.workspace = true
2124
signet-storage-types.workspace = true
2225

2326
alloy.workspace = true
2427
thiserror.workspace = true
2528
tokio-util.workspace = true
2629

2730
[dev-dependencies]
31+
serial_test.workspace = true
2832
signet-storage = { path = ".", features = ["test-utils"] }
2933
tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
3034
tokio-util.workspace = true
@@ -33,3 +37,5 @@ trevm.workspace = true
3337
[features]
3438
default = []
3539
test-utils = ["signet-hot/test-utils", "signet-cold/test-utils"]
40+
postgres = ["signet-cold-sql/postgres"]
41+
sqlite = ["signet-cold-sql/sqlite"]

0 commit comments

Comments
 (0)