RCP-240814A: Generalized Wallet Persistence in bp-wallet

[will-bitlight]

Background

Bitlight is proposing a flexible and customizable storage solution for its wallet implementation, which can be integrated with various storage backends, including but not limited to FS. Previously, the project relied on two key functions in bp-wallet: BpWallet::detach (to extract the internal wallet structure) and BpWallet::restore (to restore the wallet from a stored state).

Problem

With the removal of the BpWallet::detach and BpWallet::restore APIs in the Beta7 release of bp-wallet, there is a need for a more generalized and versatile approach to wallet storage and recovery. This would provide downstream development projects with a seamless transition and an effective alternative for implementing their storage solutions.

Proposal Goal

This RFC proposes a new design pattern for bp-wallet that provides a generalized and extensible persistence interface. This design will offer a standard interface for downstream projects to implement custom storage backends, while also ensuring high performance and compatibility with existing projects. The proposed design will maintain synchronous APIs to avoid breaking changes, while still allowing for asynchronous storage operations internally.

Additional Notes

Since there is currently no RFC submission process available under the bp-wg organization, this proposal is being submitted under the rgb-wg organization. If needed, I am fully prepared to adjust the proposal's placement and design according to your guidance.

Next Steps

Once we have reached an agreement on the approach, the Bitlight team is prepared to assist in developing and implementing this proposal within bp-wallet.

Wallet Persistence Design Document

Initial Consideration

The initial plan was to implement all APIs asynchronously. However, making the persistence operations asynchronous would require significant changes to the existing codebase, as all public APIs in bp-wallet are currently synchronous. Converting these APIs to asynchronous would introduce a substantial breaking change, especially since persistence operations are typically triggered during the execution of public APIs or when an object is dropped.

For downstream developers, it's crucial that bp-wallet can automatically persist wallet data during runtime using a user-defined storage mechanism, without requiring explicit calls to detach for saving internal fields.

To avoid extensive refactoring and maintain the synchronous nature of the current APIs, this proposal outlines two synchronous persistence design approaches for the Wallet structure.

Tip: If asynchronous storage is required (e.g., for S3), user can use block_on or similar methods within the StoreProvider synchronous API to execute asynchronous operations, thereby achieving asynchronous persistence without altering the public APIs.

---

Approach 1: Unified StoreProvider Synchronous API Design

Design Overview

In the first approach, a unified StoreProvider trait is used to handle the persistence of all parts of the Wallet. This design centralizes all storage operations within a single trait, where bp-wallet serializes the data as strings or bytes before passing it to the StoreProvider. Users only need to implement the storage logic.

StoreProvider Trait

pub trait StoreProvider: Send + Debug {
    type Error;

    fn load_descr(&self) -> Result<String, Self::Error>;
    fn save_descr(&self, data: &str) -> Result<(), Self::Error>;

    fn load_data(&self) -> Result<String, Self::Error>;
    fn save_data(&self, data: &str) -> Result<(), Self::Error>;

    fn load_cache(&self) -> Result<String, Self::Error>;
    fn save_cache(&self, data: &str) -> Result<(), Self::Error>;

    fn load_layer2(&self) -> Result<Vec<u8>, Self::Error>;
    fn save_layer2(&self, data: &[u8]) -> Result<(), Self::Error>;
}

Wallet Structure and Methods

• Loading the Wallet: The StoreProvider interface is used to load descriptions, data, cache, and Layer 2 data, which are then deserialized.
• Saving the Wallet: The save method serializes each component of the Wallet and stores them using the StoreProvider if the dirty flag is set to true.
• Auto-Save on Drop: The Wallet automatically saves itself when dropped if the dirty flag is true.

Advantages

• Unified Interface: All storage operations are managed through a single StoreProvider interface, simplifying the API design.
• Reduced Implementation Complexity: Users only need to implement one StoreProvider trait.

---

Approach 2: Multiple StoreProvider Synchronous API Design

Design Overview

The second approach is more flexible, allowing each component of the Wallet (such as descriptions, data, cache, and Layer 2 data) to use separate StoreProvider implementations. This enables different storage strategies for each component. This design was inspired a lot by your PR, thank you so much for exploring.

StoreProvider Trait

pub trait StoreProvider<T>: Send + Debug {
    type Error;

    fn load(&self) -> Result<T, Self::Error>;
    fn store(&self, object: &T) -> Result<(), Self::Error>;
}

Wallet Structure and Methods

• Loading the Wallet: Each part of the Wallet is loaded using its corresponding StoreProvider.
• Saving the Wallet: The save method saves each part of the Wallet using its respective StoreProvider if the dirty flag is set to true.
• Auto-Save on Drop: The Wallet automatically saves itself when dropped if the dirty flag is true.

Advantages

• Flexible Storage Strategies: Different storage strategies can be used for different parts of the Wallet.
• Fine-Grained Control: Developers can implement different StoreProvider traits for different components as needed.

---

Comparison and Selection

• Unified Interface vs. Flexibility:
  • The first design provides a unified interface, simplifying storage implementation.
  • The second design offers greater flexibility by allowing different storage strategies for each component.
• Complexity:
  • The first design is simpler and better suited for scenarios where storage needs are consistent.
  • The second design is more complex but ideal for cases requiring varied storage strategies.

Conclusion

Both designs maintain the synchronous API nature, avoiding large-scale refactoring of the existing codebase. The choice of which design to adopt depends on the specific requirements of the use case. If the application needs to handle different storage strategies for various parts of the Wallet, the second design is more suitable. Otherwise, the first design offers sufficient flexibility with simpler implementation.

Tip: For scenarios requiring asynchronous storage (such as S3), block_on or similar techniques can be employed within the synchronous StoreProvider API to perform asynchronous operations, thus achieving asynchronous persistence without modifying the public APIs.

---

Mock Code Examples (Draft Version)

Approach 1

use std::fmt::Debug;

pub trait StoreProvider: Send + Debug {
    type Error;

    fn load_descr(&self) -> Result<String, Self::Error>;
    fn save_descr(&self, data: &str) -> Result<(), Self::Error>;

    fn load_data(&self) -> Result<String, Self::Error>;
    fn save_data(&self, data: &str) -> Result<(), Self::Error>;

    fn load_cache(&self) -> Result<String, Self::Error>;
    fn save_cache(&self, data: &str) -> Result<(), Self::Error>;

    fn load_layer2(&self) -> Result<Vec<u8>, Self::Error>;
    fn save_layer2(&self, data: &[u8]) -> Result<(), Self::Error>;
}

pub struct Wallet<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider>
where
    Self: Save,
{
    descr: WalletDescr<K, D, L2::Descr>,
    data: WalletData<L2::Data>,
    cache: WalletCache<L2::Cache>,
    layer2: L2,
    storage: S,
    dirty: bool,
}

impl<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider> Wallet<K, D, L2, S>
where
    for<'de> WalletDescr<K, D>: serde::Serialize + serde::Deserialize<'de>,
    for<'de> D: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Descr: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Data: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Cache: serde::Serialize + serde::Deserialize<'de>,
{
    pub fn load(storage: S) -> Result<(Self, Vec<Warning>), LoadError<L2::LoadError, S::Error>> {
        let mut warnings = Vec::new();

        let descr = storage.load_descr().map_err(LoadError::Storage)?;
        let descr: WalletDescr<K, D> = toml::from_str(&descr).map_err(LoadError::Deserialize)?;

        let data = storage.load_data().map_err(LoadError::Storage)?;
        let data: WalletData<L2::Data> = toml::from_str(&data).map_err(LoadError::Deserialize)?;

        let cache = storage
            .load_cache()
            .map_err(|_| Warning::CacheAbsent)
            .and_then(|cache| {
                serde_yaml::from_str(&cache).map_err(|err| Warning::CacheDamaged(err))
            })
            .unwrap_or_else(|warn| {
                warnings.push(warn);
                WalletCache::default()
            });

        let layer2_data = storage.load_layer2().map_err(LoadError::Storage)?;
        let layer2 = L2::load_from_bytes(&layer2_data).map_err(LoadError

::Layer2)?;

        let wallet = Wallet {
            descr,
            data,
            cache,
            layer2,
            storage,
            dirty: false,
        };
        Ok((wallet, warnings))
    }
}

impl<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider> Save for Wallet<K, D, L2, S>
where
    for<'de> WalletDescr<K, D>: serde::Serialize + serde::Deserialize<'de>,
    for<'de> D: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Descr: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Data: serde::Serialize + serde::Deserialize<'de>,
    for<'de> L2::Cache: serde::Serialize + serde::Deserialize<'de>,
{
    type SaveErr = StoreError<L2::StoreError, S::Error>;

    fn save(&self) -> Result<bool, Self::SaveErr> {
        if self.dirty {
            let descr = toml::to_string_pretty(&self.descr).map_err(StoreError::Serialize)?;
            self.storage.save_descr(&descr).map_err(StoreError::Storage)?;

            let data = toml::to_string_pretty(&self.data).map_err(StoreError::Serialize)?;
            self.storage.save_data(&data).map_err(StoreError::Storage)?;

            let cache = serde_yaml::to_string(&self.cache).map_err(StoreError::Serialize)?;
            self.storage.save_cache(&cache).map_err(StoreError::Storage)?;

            let layer2_data = self.layer2.store_to_bytes().map_err(StoreError::Layer2)?;
            self.storage.save_layer2(&layer2_data).map_err(StoreError::Storage)?;

            Ok(true)
        } else {
            Ok(false)
        }
    }
}

impl<K, D: Descriptor<K>, L2: Layer2, S: StoreProvider> Drop for Wallet<K, D, L2, S>
where
    Self: Save,
{
    fn drop(&mut self) {
        if self.dirty {
            let _ = self.save();
        }
    }
}

Approach 2

use std::fmt::Debug;

pub trait StoreProvider<T>: Send + Debug {
    type Error;

    fn load(&self) -> Result<T, Self::Error>;
    fn store(&self, object: &T) -> Result<(), Self::Error>;
}

// remove fs_config
#[derive(Clone, Debug)]
pub struct Wallet<K, D: Descriptor<K>, L2: Layer2,
    DescrStore: StoreProvider<WalletDescr<K, D, L2::Descr>>,
    DataStore: StoreProvider<WalletData<L2::Data>>,
    CacheStore: StoreProvider<WalletCache<L2::Cache>>,
    Layer2Store: StoreProvider<L2>>
{
    descr: WalletDescr<K, D, L2::Descr>,
    data: WalletData<L2::Data>,
    cache: WalletCache<L2::Cache>,
    layer2: L2,
    dirty: bool,
}

// append store_provider
#[derive(Getters, Debug)]
pub struct WalletDescr<K, D, L2 = NoLayer2>
where
    D: Descriptor<K>,
    L2: Layer2Descriptor,
{
    generator: D,
    #[getter(as_copy)]
    network: Network,
    layer2: L2,
    #[cfg_attr(feature = "serde", serde(skip))]
    store_provider: Option<Box<dyn StoreProvider<Self>>>,
    #[cfg_attr(feature = "serde", serde(skip))]
    _phantom: PhantomData<K>,
}

// append store_provider
pub struct WalletData<L2: Layer2Data> {
    pub name: String,
    pub tx_annotations: BTreeMap<Txid, String>,
    pub txout_annotations: BTreeMap<Outpoint, String>,
    pub txin_annotations: BTreeMap<Outpoint, String>,
    pub addr_annotations: BTreeMap<Address, String>,
    pub layer2_annotations: L2,
    pub last_used: BTreeMap<Keychain, NormalIndex>,
    #[cfg_attr(feature = "serde", serde(skip))]
    store_provider: Option<Box<dyn StoreProvider<Self>>>,
}

// append store_provider
#[derive(Debug)]
pub struct WalletCache<L2: Layer2Cache> {
    pub last_block: MiningInfo,
    pub last_change: NormalIndex,
    pub headers: BTreeSet<BlockInfo>,
    pub tx: BTreeMap<Txid, WalletTx>,
    pub utxo: BTreeSet<Outpoint>,
    pub addr: BTreeMap<Keychain, BTreeSet<WalletAddr>>,
    pub layer2: L2,
    #[cfg_attr(feature = "serde", serde(skip))]
    store_provider: Option<Box<dyn StoreProvider<Self>>>,
}

impl<K, D: Descriptor<K>, L2: Layer2,
    DescrStore: StoreProvider<WalletDescr<K, D, L2::Descr>>,
    DataStore: StoreProvider<WalletData<L2::Data>>,
    CacheStore: StoreProvider<WalletCache<L2::Cache>>,
    Layer2Store: StoreProvider<L2>> Wallet<K, D, L2, DescrStore, DataStore, CacheStore, Layer2Store>
{

    pub fn make_descr_store_provider(
        &mut self,
        provider: impl StoreProvider<WalletCache<L2::Cache>> + 'static,
    ) {
        self.descr.store_provider = Some(Box::new(provider));
    }

    pub fn make_data_store_provider(
        &mut self,
        provider: impl StoreProvider<WalletCache<L2::Cache>> + 'static,
    ) {
        self.data.store_provider = Some(Box::new(provider));
    }

    pub fn make_cache_store_provider(
        &mut self,
        provider: impl StoreProvider<WalletCache<L2::Cache>> + 'static,
    ) {
        self.cache.store_provider = Some(Box::new(provider));
    }

    pub fn with(
        descr: WalletDescr<K, D, L2::Descr>,
        data: WalletData<L2::Data>,
        cache: WalletCache<L2::Cache>,
        layer2: L2,
    ) -> Self {
        Wallet {
            descr,
            data,
            cache,
            layer2,
            dirty: false,
            #[cfg(feature = "fs")]
            fs: None,
        }
    }

    pub fn save(&self) -> Result<(), StoreError> {
            if self.dirty {
                self.descr.store_provider.as_ref().map(|provider| provider.store(&self.descr));
                self.data.store_provider.as_ref().map(|provider| provider.store(&self.data));
                self.cache.store_provider.as_ref().map(|provider| provider.store(&self.cache));
            }
        Ok(())
    }
}

impl<K, D: Descriptor<K>, L2: Layer2,
    DescrStore: StoreProvider<WalletDescr<K, D, L2::Descr>>,
    DataStore: StoreProvider<WalletData<L2::Data>>,
    CacheStore: StoreProvider<WalletCache<L2::Cache>>,
    Layer2Store: StoreProvider<L2>> Drop for Wallet<K, D, L2, DescrStore, DataStore, CacheStore, Layer2Store>
{
    fn drop(&mut self) {
        if self.dirty {
            let _ = self.save();
        }
    }
}

// OpendalContainer is a S3 Operator
#[derive(Debug)]
pub struct OpendalContainer {
    operator: &'static OpendalOperator,
    user_id: String,
}

impl OpendalContainer {
    pub async fn make_container(
        user_id: &str,
        config: OpendalConfig,
    ) -> io::Result<OpendalContainer> {
        let operator = config.opendal_operator().await;
        Ok(OpendalContainer {
            operator,
            user_id: user_id.to_string(),
        })
    }
}

impl<K, D: Descriptor<K>, L2D: Layer2Descriptor> StoreProvider<WalletDescr<K, D, L2D>>
    for OpendalContainer
where for<'a> WalletDescr<K, D, L2D>: Serialize + Deserialize<'a>
{
    fn load(&self) -> Result<WalletDescr<K, D, L2D>, LoadError> {
        let data = block_in_place(|| {
            Handle::current().block_on(async {
                let path = format!("{}/descr.toml", self.user_id);
                let buffer = self.operator.inner.read(&path).await?;
                let string = String::from_utf8(buffer.to_bytes().to_vec())
                    .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
                Ok::<_, io::Error>(string)
            })
        })?;
        Ok(toml::from_str(&data)?)
    }

    fn store(&self, object: &WalletDescr<K, D, L2D>) -> Result<(), StoreError> {
        let data = toml::to_string_pretty(object).expect("");

        block_in_place(move || {
            Handle::current().block_on(async move {
                let path = format!("{}/descr.toml", self.user_id);
                self.operator.inner.write(&path, data.into_bytes()).await?;
                Ok::<_, io::Error>(())
            })
        })?;
        Ok(())
    }
}

impl<L2> StoreProvider<WalletData<L2>> for OpendalContainer
where
    for<'a> WalletData<L2>: Serialize + Deserialize<'a>,
    L2: Layer2Data,
{
    fn load(&self) -> Result<WalletData<L2>, LoadError> {
        let data = block_in_place(|| {
            Handle::current().block_on(async {
                let path = format!("{}/data.toml", self.user_id);
                let buffer = self.operator.inner.read(&path).await?;
                let string = String::from_utf8(buffer.to_bytes().to_vec())
                    .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
                Ok::<_, io::Error>(string)
            })
        })?;
        Ok(toml::from_str(&data)?)
    }

    fn store(&self, object: &WalletData<L2>) -> Result<(), StoreError> {
        let data = toml::to_string_pretty(object).expect("");

        block_in_place(move || {
            Handle::current().block_on(async move {
                let path = format!("{}/data.toml", self.user_id);
                self.operator.inner.write(&path, data.into_bytes()).await?;
                Ok::<_, io::Error>(())
            })
        })?;
        Ok(())
    }
}

impl<L2: Layer2Cache> StoreProvider<WalletCache<L2>> for OpendalContainer
where for<'a> WalletCache<L2>: Serialize + Deserialize<'a>
{
    fn load(&self) -> Result<WalletCache<L2>, LoadError> {
        let data = block_in_place(|| {
            Handle::current().block_on(async {
                let path = format!("{}/cache.yaml", self.user_id);
                let buffer = self.operator.inner.read(&path).await.expect("");
                let string = String::from_utf8(buffer.to_bytes().to_vec())
                    .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
                Ok::<_, io::Error>(string)
            })
        })?;
        Ok(serde_yaml::from_str(&data).expect(""))
    }

    fn store(&self, object: &WalletCache<L2>) -> Result<(), StoreError> {
        let data = serde_yaml::to_string(object).expect("");

        block_in_place(move || {
            Handle::current().block_on(async move {
                let path = format!("{}/cache.yaml", self.user_id);
                self.operator.inner.write(&path, data.into_bytes()).await?;
                Ok::<_, io::Error>(())
            })
        })?;
        Ok(())
    }
}

This RFC outlines the problem, proposes two different approaches for the solution, and provides example code to illustrate each design. The goal is to allow bp-wallet to offer a generalized, customizable, and efficient persistence solution that supports various backend storage options without requiring significant changes to existing APIs.

[crisdut]

Hi @will-bitlight

We have to be careful when using futures::executor (or even tokio::task): both don't work well when we compile in wasm (to use in web extensions, browser APIs, etc...), because the browser doesn't allow block threads.

I hope I helped.