RCP-240814A: Generalized Wallet Persistence in bp-wallet

### Background

Bitlight is proposing a flexible and customizable storage solution for its wallet implementation, which can be integrated with various storage backends, including but not limited to FS. Previously, the project relied on two key functions in `bp-wallet`: `BpWallet::detach` (to extract the internal wallet structure) and `BpWallet::restore` (to restore the wallet from a stored state).

### Problem

With the removal of the `BpWallet::detach` and `BpWallet::restore` APIs in the Beta7 release of `bp-wallet`, there is a need for a more generalized and versatile approach to wallet storage and recovery. This would provide downstream development projects with a seamless transition and an effective alternative for implementing their storage solutions.

### Proposal Goal

This RFC proposes a new design pattern for `bp-wallet` that provides a generalized and extensible persistence interface. This design will offer a standard interface for downstream projects to implement custom storage backends, while also ensuring high performance and compatibility with existing projects. The proposed design will maintain synchronous APIs to avoid breaking changes, while still allowing for asynchronous storage operations internally.

### Additional Notes

Since there is currently no RFC submission process available under the bp-wg organization, this proposal is being submitted under the rgb-wg organization. If needed, I am fully prepared to adjust the proposal's placement and design according to your guidance.

### Next Steps

Once we have reached an agreement on the approach, the Bitlight team is prepared to assist in developing and implementing this proposal within `bp-wallet`.

### Wallet Persistence Design Document

### Initial Consideration

The initial plan was to implement all APIs asynchronously. However, making the persistence operations asynchronous would require significant changes to the existing codebase, as all public APIs in `bp-wallet` are currently synchronous. Converting these APIs to asynchronous would introduce a substantial breaking change, especially since persistence operations are typically triggered during the execution of public APIs or when an object is dropped.

For downstream developers, it's crucial that `bp-wallet` can automatically persist wallet data during runtime using a user-defined storage mechanism, without requiring explicit calls to `detach` for saving internal fields.

To avoid extensive refactoring and maintain the synchronous nature of the current APIs, this proposal outlines two synchronous persistence design approaches for the `Wallet` structure.

**Tip:** If asynchronous storage is required (e.g., for S3), user can use `block_on` or similar methods within the `StoreProvider` synchronous API to execute asynchronous operations, thereby achieving asynchronous persistence without altering the public APIs.

---

### Approach 1: Unified `StoreProvider` Synchronous API Design

### Design Overview

In the first approach, a unified `StoreProvider` trait is used to handle the persistence of all parts of the `Wallet`. This design centralizes all storage operations within a single trait, where `bp-wallet` serializes the data as strings or bytes before passing it to the `StoreProvider`. Users only need to implement the storage logic.

### `StoreProvider` Trait

```rust
pub trait StoreProvider: Send + Debug {
    type Error;

    fn load_descr(&self) -> Result<String, Self::Error>;
    fn save_descr(&self, data: &str) -> Result<(), Self::Error>;

    fn load_data(&self) -> Result<String, Self::Error>;
    fn save_data(&self, data: &str) -> Result<(), Self::Error>;

    fn load_cache(&self) -> Result<String, Self::Error>;
    fn save_cache(&self, data: &str) -> Result<(), Self::Error>;

    fn load_layer2(&self) -> Result<Vec<u8>, Self::Error>;
    fn save_layer2(&self, data: &[u8]) -> Result<(), Self::Error>;
}

```

### `Wallet` Structure and Methods

- **Loading the `Wallet`**: The `StoreProvider` interface is used to load descriptions, data, cache, and Layer 2 data, which are then deserialized.
- **Saving the `Wallet`**: The `save` method serializes each component of the `Wallet` and stores them using the `StoreProvider` if the `dirty` flag is set to `true`.
- **Auto-Save on Drop**: The `Wallet` automatically saves itself when dropped if the `dirty` flag is `true`.

### Advantages

- **Unified Interface**: All storage operations are managed through a single `StoreProvider` interface, simplifying the API design.
- **Reduced Implementation Complexity**: Users only need to implement one `StoreProvider` trait.

---

### Approach 2: Multiple `StoreProvider` Synchronous API Design

### Design Overview

The second approach is more flexible, allowing each component of the `Wallet` (such as descriptions, data, cache, and Layer 2 data) to use separate `StoreProvider` implementations. This enables different storage strategies for each component. **This design was inspired a lot by your PR, thank you so much for exploring.**

### `StoreProvider` Trait

```rust
pub trait StoreProvider<T>: Send + Debug {
    type Error;

    fn load(&self) -> Result<T, Self::Error>;
    fn store(&self, object: &T) -> Result<(), Self::Error>;
}

```

### `Wallet` Structure and Methods

- **Loading the `Wallet`**: Each part of the `Wallet` is loaded using its corresponding `StoreProvider`.
- **Saving the `Wallet`**: The `save` method saves each part of the `Wallet` using its respective `StoreProvider` if the `dirty` flag is set to `true`.
- **Auto-Save on Drop**: The `Wallet` automatically saves itself when dropped if the `dirty` flag is `true`.

### Advantages

- **Flexible Storage Strategies**: Different storage strategies can be used for different parts of the `Wallet`.
- **Fine-Grained Control**: Developers can implement different `StoreProvider` traits for different components as needed.

---

### Comparison and Selection

- **Unified Interface vs. Flexibility**:
    - The first design provides a unified interface, simplifying storage implementation.
    - The second design offers greater flexibility by allowing different storage strategies for each component.
- **Complexity**:
    - The first design is simpler and better suited for scenarios where storage needs are consistent.
    - The second design is more complex but ideal for cases requiring varied storage strategies.

### Conclusion

Both designs maintain the synchronous API nature, avoiding large-scale refactoring of the existing codebase. The choice of which design to adopt depends on the specific requirements of the use case. If the application needs to handle different storage strategies for various parts of the `Wallet`, the second design is more suitable. Otherwise, the first design offers sufficient flexibility with simpler implementation.

**Tip:** For scenarios requiring asynchronous storage (such as S3), `block_on` or similar techniques can be employed within the synchronous `StoreProvider` API to perform asynchronous operations, thus achieving asynchronous persistence without modifying the public APIs.

---

### Mock Code Examples (Draft Version)

### Approach 1

```rust
use std::fmt::Debug;

pub trait StoreProvider: Send + Debug {
    type Error;

    fn load_descr(&self) -> Result<String, Self::Error>;
    fn save_descr(&self, data: &str) -> Result<(), Self::Error>;

    fn load_data(&self) -> Result<String, Self::Error>;
    fn save_data(&self, data: &str) -> Result<(), Self::Error>;

    fn load_cache(&self) -> Result<String, Self::Error>;
    fn save_cache(&self, data: &str) -> Result<(), Self::Error>;

    fn load_layer2(&self) -> Result<Vec<u8>, Self::Error>;
    fn save_layer2(&self, data: &[u8]) -> Result<(), Self::Error>;
}

pub struct Wallet<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider>
where
    Self: Save,
{
    descr: WalletDescr<K, D, L2::Descr>,
    data: WalletData<L2::Data>,
    cache: WalletCache<L2::Cache>,
    layer2: L2,
    storage: S,
    dirty: bool,
}

impl<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider> Wallet<K, D, L2, S>
where
    for<'de> WalletDescr<K, D>: serde::Serialize + serde::Deserialize<'de>,
    for<'de> D: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Descr: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Data: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Cache: serde::Serialize + serde::Deserialize<'de>,
{
    pub fn load(storage: S) -> Result<(Self, Vec<Warning>), LoadError<L2::LoadError, S::Error>> {
        let mut warnings = Vec::new();

        let descr = storage.load_descr().map_err(LoadError::Storage)?;
        let descr: WalletDescr<K, D> = toml::from_str(&descr).map_err(LoadError::Deserialize)?;

        let data = storage.load_data().map_err(LoadError::Storage)?;
        let data: WalletData<L2::Data> = toml::from_str(&data).map_err(LoadError::Deserialize)?;

        let cache = storage
            .load_cache()
            .map_err(|_| Warning::CacheAbsent)
            .and_then(|cache| {
                serde_yaml::from_str(&cache).map_err(|err| Warning::CacheDamaged(err))
            })
            .unwrap_or_else(|warn| {
                warnings.push(warn);
                WalletCache::default()
            });

        let layer2_data = storage.load_layer2().map_err(LoadError::Storage)?;
        let layer2 = L2::load_from_bytes(&layer2_data).map_err(LoadError

::Layer2)?;

        let wallet = Wallet {
            descr,
            data,
            cache,
            layer2,
            storage,
            dirty: false,
        };
        Ok((wallet, warnings))
    }
}

impl<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider> Save for Wallet<K, D, L2, S>
where
    for<'de> WalletDescr<K, D>: serde::Serialize + serde::Deserialize<'de>,
    for<'de> D: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Descr: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Data: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Cache: serde::Serialize + serde::Deserialize<'de>,
{
    type SaveErr = StoreError<L2::StoreError, S::Error>;

    fn save(&self) -> Result<bool, Self::SaveErr> {
        if self.dirty {
            let descr = toml::to_string_pretty(&self.descr).map_err(StoreError::Serialize)?;
            self.storage.save_descr(&descr).map_err(StoreError::Storage)?;

            let data = toml::to_string_pretty(&self.data).map_err(StoreError::Serialize)?;
            self.storage.save_data(&data).map_err(StoreError::Storage)?;

            let cache = serde_yaml::to_string(&self.cache).map_err(StoreError::Serialize)?;
            self.storage.save_cache(&cache).map_err(StoreError::Storage)?;

            let layer2_data = self.layer2.store_to_bytes().map_err(StoreError::Layer2)?;
            self.storage.save_layer2(&layer2_data).map_err(StoreError::Storage)?;

            Ok(true)
        } else {
            Ok(false)
        }
    }
}

impl<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider> Drop for Wallet<K, D, L2, S>
where
    Self: Save,
{
    fn drop(&mut self) {
        if self.dirty {
            let _ = self.save();
        }
    }
}

```

### Approach 2 

```rust
use std::fmt::Debug;

pub trait StoreProvider<T>: Send + Debug {
    type Error;

    fn load(&self) -> Result<T, Self::Error>;
    fn store(&self, object: &T) -> Result<(), Self::Error>;
}

// remove fs_config
#[derive(Clone, Debug)]
pub struct Wallet<K, D: Descriptor<K>, L2: Layer2,
    DescrStore: StoreProvider<WalletDescr<K, D, L2::Descr>>,
    DataStore: StoreProvider<WalletData<L2::Data>>,
    CacheStore: StoreProvider<WalletCache<L2::Cache>>,
    Layer2Store: StoreProvider<L2>>
{
    descr: WalletDescr<K, D, L2::Descr>,
    data: WalletData<L2::Data>,
    cache: WalletCache<L2::Cache>,
    layer2: L2,
    dirty: bool,
}

// append store_provider
#[derive(Getters, Debug)]
pub struct WalletDescr<K, D, L2 = NoLayer2>
where
    D: Descriptor<K>,
    L2: Layer2Descriptor,
{
    generator: D,
    #[getter(as_copy)]
    network: Network,
    layer2: L2,
    #[cfg_attr(feature = "serde", serde(skip))]
    store_provider: Option<Box<dyn StoreProvider<Self>>>,
    #[cfg_attr(feature = "serde", serde(skip))]
    _phantom: PhantomData<K>,
}

// append store_provider
pub struct WalletData<L2: Layer2Data> {
    pub name: String,
    pub tx_annotations: BTreeMap<Txid, String>,
    pub txout_annotations: BTreeMap<Outpoint, String>,
    pub txin_annotations: BTreeMap<Outpoint, String>,
    pub addr_annotations: BTreeMap<Address, String>,
    pub layer2_annotations: L2,
    pub last_used: BTreeMap<Keychain, NormalIndex>,
    #[cfg_attr(feature = "serde", serde(skip))]
    store_provider: Option<Box<dyn StoreProvider<Self>>>,
}

// append store_provider
#[derive(Debug)]
pub struct WalletCache<L2: Layer2Cache> {
    pub last_block: MiningInfo,
    pub last_change: NormalIndex,
    pub headers: BTreeSet<BlockInfo>,
    pub tx: BTreeMap<Txid, WalletTx>,
    pub utxo: BTreeSet<Outpoint>,
    pub addr: BTreeMap<Keychain, BTreeSet<WalletAddr>>,
    pub layer2: L2,
    #[cfg_attr(feature = "serde", serde(skip))]
    store_provider: Option<Box<dyn StoreProvider<Self>>>,
}

impl<K, D: Descriptor<K>, L2: Layer2,
    DescrStore: StoreProvider<WalletDescr<K, D, L2::Descr>>,
    DataStore: StoreProvider<WalletData<L2::Data>>,
    CacheStore: StoreProvider<WalletCache<L2::Cache>>,
    Layer2Store: StoreProvider<L2>> Wallet<K, D, L2, DescrStore, DataStore, CacheStore, Layer2Store>
{

    pub fn make_descr_store_provider(
        &mut self,
        provider: impl StoreProvider<WalletCache<L2::Cache>> + 'static,
    ) {
        self.descr.store_provider = Some(Box::new(provider));
    }

    pub fn make_data_store_provider(
        &mut self,
        provider: impl StoreProvider<WalletCache<L2::Cache>> + 'static,
    ) {
        self.data.store_provider = Some(Box::new(provider));
    }

    pub fn make_cache_store_provider(
        &mut self,
        provider: impl StoreProvider<WalletCache<L2::Cache>> + 'static,
    ) {
        self.cache.store_provider = Some(Box::new(provider));
    }

    pub fn with(
        descr: WalletDescr<K, D, L2::Descr>,
        data: WalletData<L2::Data>,
        cache: WalletCache<L2::Cache>,
        layer2: L2,
    ) -> Self {
        Wallet {
            descr,
            data,
            cache,
            layer2,
            dirty: false,
            #[cfg(feature = "fs")]
            fs: None,
        }
    }

    pub fn save(&self) -> Result<(), StoreError> {
            if self.dirty {
                self.descr.store_provider.as_ref().map(|provider| provider.store(&self.descr));
                self.data.store_provider.as_ref().map(|provider| provider.store(&self.data));
                self.cache.store_provider.as_ref().map(|provider| provider.store(&self.cache));
            }
        Ok(())
    }
}

impl<K, D: Descriptor<K>, L2: Layer2,
    DescrStore: StoreProvider<WalletDescr<K, D, L2::Descr>>,
    DataStore: StoreProvider<WalletData<L2::Data>>,
    CacheStore: StoreProvider<WalletCache<L2::Cache>>,
    Layer2Store: StoreProvider<L2>> Drop for Wallet<K, D, L2, DescrStore, DataStore, CacheStore, Layer2Store>
{
    fn drop(&mut self) {
        if self.dirty {
            let _ = self.save();
        }
    }
}

// OpendalContainer is a S3 Operator
#[derive(Debug)]
pub struct OpendalContainer {
    operator: &'static OpendalOperator,
    user_id: String,
}

impl OpendalContainer {
    pub async fn make_container(
        user_id: &str,
        config: OpendalConfig,
    ) -> io::Result<OpendalContainer> {
        let operator = config.opendal_operator().await;
        Ok(OpendalContainer {
            operator,
            user_id: user_id.to_string(),
        })
    }
}

impl<K, D: Descriptor<K>, L2D: Layer2Descriptor> StoreProvider<WalletDescr<K, D, L2D>>
    for OpendalContainer
where for<'a> WalletDescr<K, D, L2D>: Serialize + Deserialize<'a>
{
    fn load(&self) -> Result<WalletDescr<K, D, L2D>, LoadError> {
        let data = block_in_place(|| {
            Handle::current().block_on(async {
                let path = format!("{}/descr.toml", self.user_id);
                let buffer = self.operator.inner.read(&path).await?;
                let string = String::from_utf8(buffer.to_bytes().to_vec())
                    .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
                Ok::<_, io::Error>(string)
            })
        })?;
        Ok(toml::from_str(&data)?)
    }

    fn store(&self, object: &WalletDescr<K, D, L2D>) -> Result<(), StoreError> {
        let data = toml::to_string_pretty(object).expect("");

        block_in_place(move || {
            Handle::current().block_on(async move {
                let path = format!("{}/descr.toml", self.user_id);
                self.operator.inner.write(&path, data.into_bytes()).await?;
                Ok::<_, io::Error>(())
            })
        })?;
        Ok(())
    }
}

impl<L2> StoreProvider<WalletData<L2>> for OpendalContainer
where
    for<'a> WalletData<L2>: Serialize + Deserialize<'a>,
    L2: Layer2Data,
{
    fn load(&self) -> Result<WalletData<L2>, LoadError> {
        let data = block_in_place(|| {
            Handle::current().block_on(async {
                let path = format!("{}/data.toml", self.user_id);
                let buffer = self.operator.inner.read(&path).await?;
                let string = String::from_utf8(buffer.to_bytes().to_vec())
                    .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
                Ok::<_, io::Error>(string)
            })
        })?;
        Ok(toml::from_str(&data)?)
    }

    fn store(&self, object: &WalletData<L2>) -> Result<(), StoreError> {
        let data = toml::to_string_pretty(object).expect("");

        block_in_place(move || {
            Handle::current().block_on(async move {
                let path = format!("{}/data.toml", self.user_id);
                self.operator.inner.write(&path, data.into_bytes()).await?;
                Ok::<_, io::Error>(())
            })
        })?;
        Ok(())
    }
}

impl<L2: Layer2Cache> StoreProvider<WalletCache<L2>> for OpendalContainer
where for<'a> WalletCache<L2>: Serialize + Deserialize<'a>
{
    fn load(&self) -> Result<WalletCache<L2>, LoadError> {
        let data = block_in_place(|| {
            Handle::current().block_on(async {
                let path = format!("{}/cache.yaml", self.user_id);
                let buffer = self.operator.inner.read(&path).await.expect("");
                let string = String::from_utf8(buffer.to_bytes().to_vec())
                    .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
                Ok::<_, io::Error>(string)
            })
        })?;
        Ok(serde_yaml::from_str(&data).expect(""))
    }

    fn store(&self, object: &WalletCache<L2>) -> Result<(), StoreError> {
        let data = serde_yaml::to_string(object).expect("");

        block_in_place(move || {
            Handle::current().block_on(async move {
                let path = format!("{}/cache.yaml", self.user_id);
                self.operator.inner.write(&path, data.into_bytes()).await?;
                Ok::<_, io::Error>(())
            })
        })?;
        Ok(())
    }
}

```

This RFC outlines the problem, proposes two different approaches for the solution, and provides example code to illustrate each design. The goal is to allow `bp-wallet` to offer a generalized, customizable, and efficient persistence solution that supports various backend storage options without requiring significant changes to existing APIs.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RCP-240814A: Generalized Wallet Persistence in bp-wallet #11

Background

Problem

Proposal Goal

Additional Notes

Next Steps

Wallet Persistence Design Document

Initial Consideration

Approach 1: Unified `StoreProvider` Synchronous API Design

Design Overview

`StoreProvider` Trait

`Wallet` Structure and Methods

Advantages

Approach 2: Multiple `StoreProvider` Synchronous API Design

Design Overview

`StoreProvider` Trait

`Wallet` Structure and Methods

Advantages

Comparison and Selection

Conclusion

Mock Code Examples (Draft Version)

Approach 1

Approach 2

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RCP-240814A: Generalized Wallet Persistence in bp-wallet #11

Description

Background

Problem

Proposal Goal

Additional Notes

Next Steps

Wallet Persistence Design Document

Initial Consideration

Approach 1: Unified StoreProvider Synchronous API Design

Design Overview

StoreProvider Trait

Wallet Structure and Methods

Advantages

Approach 2: Multiple StoreProvider Synchronous API Design

Design Overview

StoreProvider Trait

Wallet Structure and Methods

Advantages

Comparison and Selection

Conclusion

Mock Code Examples (Draft Version)

Approach 1

Approach 2

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Approach 1: Unified `StoreProvider` Synchronous API Design

`StoreProvider` Trait

`Wallet` Structure and Methods

Approach 2: Multiple `StoreProvider` Synchronous API Design

`StoreProvider` Trait

`Wallet` Structure and Methods