From e0ff6df2107e16fb588ace129b5d6ad25fd03772 Mon Sep 17 00:00:00 2001 From: James-Mart Date: Wed, 2 Oct 2024 16:35:39 +0000 Subject: [PATCH] rework plugin data persistence spec --- .../app-architecture/plugins.md | 68 ++++++------------- 1 file changed, 19 insertions(+), 49 deletions(-) diff --git a/doc/psidk/src/specifications/app-architecture/plugins.md b/doc/psidk/src/specifications/app-architecture/plugins.md index 8d8070c31..fdb3911ed 100644 --- a/doc/psidk/src/specifications/app-architecture/plugins.md +++ b/doc/psidk/src/specifications/app-architecture/plugins.md @@ -1,9 +1,9 @@ # Plugins -Plugins manage client-side user interactions and enable external integrations with psibase apps. Typically, they are stored in and served from a service and they run within a users's browser. +Plugins manage client-side user interactions and enable external integrations with psibase apps. Typically, they are stored in and served from a service and they run within a users's browser. ```mermaid -sequenceDiagram +sequenceDiagram box rgb(85,85,85) client-side actor Alice @@ -35,7 +35,7 @@ Furthermore, the fact that they are WebAssembly components means that plugin dev ## Plugin mechanics -The user interface of one app can use a front-end library to send a message to the [supervisor](./supervisor.md), attempting to call an action on a plugin. The plugin is then downloaded from the corresponding service and instantiated in the browser. +The user interface of one app can use a front-end library to send a message to the [supervisor](./supervisor.md), attempting to call an action on a plugin. The plugin is then downloaded from the corresponding service and instantiated in the browser. Plugins have many capabilities at their disposal, such as client-side storage management, making synchronous calls to other plugins, and adding actions into transactions. All of this functionality is available in libraries that are dynamically generated by bindings-generators in whatever language is being used to author the plugin. @@ -50,7 +50,7 @@ Transactions contain the data and authentication payload necessary to execute an Actions are added to transactions in a FIFO queue. For example, the following sequence diagram will finally submit a transaction containing actions in the order: C, A, B. ```mermaid -sequenceDiagram +sequenceDiagram box rgb(85,85,85) client-side actor Alice @@ -82,21 +82,21 @@ Supervisor->>psinode: Submit transaction In other Web3 application frameworks, front-ends are untrusted by the authenticator ("wallet"). Therefore, the authenticator requires that any transaction proposed by an application front-end must be explicitly confirmed by the user. However, the authenticator does not understand the payload, as it contains packed data that is specific to the callee context. -With psibase apps, authentication works differently. There is no external authenticator. In general, when a front-end of an application is calling into its own plugin to execute a transaction on its own service, there is no explicit user-confirmation needed, because the identity of the actor responsible for proposing the transaction is already known to have been served directly from the service that is the target of the transaction. In other words, the UI itself *is* the authenticator for any transactions sent to the server. +With psibase apps, authentication works differently. There is no external authenticator. In general, when a front-end of an application is calling into its own plugin to execute a transaction on its own service, there is no explicit user-confirmation needed, because the identity of the actor responsible for proposing the transaction is already known to have been served directly from the service that is the target of the transaction. In other words, the UI itself _is_ the authenticator for any transactions sent to the server. -Plugins are themselves permitted to define how to acquire explicit user confirmation in cases where external applications are accessing their functionality. Typically, this should look similar to a standard OAuth flow, where the user given a context-aware authorization window in which to evaluate whether they consent to the proposed transaction. This "context-aware" property means that each app is responsible for defining its own user experience as it relates to transaction authorization. This is important not only for user experience, but also for security, so that it is clear to what a user is consenting. Contrast this with a traditional Web3 application experience wherein the authentication prompt is opaque and context-unaware, and users rarely know exactly what they are signing and instead rely on the reputation of the user-interface that generated the proposed transaction. +Plugins are themselves permitted to define how to acquire explicit user confirmation in cases where external applications are accessing their functionality. Typically, this should look similar to a standard OAuth flow, where the user given a context-aware authorization window in which to evaluate whether they consent to the proposed transaction. This "context-aware" property means that each app is responsible for defining its own user experience as it relates to transaction authorization. This is important not only for user experience, but also for security, so that it is clear to what a user is consenting. Contrast this with a traditional Web3 application experience wherein the authentication prompt is opaque and context-unaware, and users rarely know exactly what they are signing and instead rely on the reputation of the user-interface that generated the proposed transaction. Therefore, it is anticipated that the number of authentication prompts will be signficantly reduced for psibase apps, and furthermore, those prompts that remain will be context-aware rather than context-unaware, allowing for a much more secure and user-friendly authentication experience. ### Dynamic updates -Plugins are dynamically linked at runtime to facilitate potentially complex interactions between a user and many applications. This dynamic linkage means that an application can update its plugin with changes and bugfixes without breaking downstream applications by maintaining a consistent API. +Plugins are dynamically linked at runtime to facilitate potentially complex interactions between a user and many applications. This dynamic linkage means that an application can update its plugin with changes and bugfixes without breaking downstream applications by maintaining a consistent API. For example, in a traditional Web3 context, a smart contract API update could break all downstream user-interfaces that interacted with an older version of the smart contract. In psibase, a service action interface could change and all foreign integrations that depend on the updated application will continue to function without requiring any changes since all integration happens through its plugin. ### Composability -In general, this plugin architecture allows for unprecedented collaboration between Web3 apps. With psibase apps, a website or application can access resources or execute code hosted by other apps on behalf of a user. +In general, this plugin architecture allows for unprecedented collaboration between Web3 apps. With psibase apps, a website or application can access resources or execute code hosted by other apps on behalf of a user. For example, if a user has provided her address or credit card information in one application, other applications can simply make use this data without requiring the user to re-enter the same information. And all of this information can remain client-side, protecting user privacy. @@ -112,58 +112,29 @@ Blockchains typically only have public data. If private client-side data is coll ## Local data storage -### Access to database operations +### Access to database operations -To enable database actions within plugins, a plugin must import database read/write functionality. +The psibase plugin host provides a `wasi:keyvalue` interface for plugins to use. However, this interface is not very ergonomic for plugin development. Furthermore, the wasi-keyvalue interface [proposal](https://github.com/WebAssembly/wasi-keyvalue) is still in stage 2 of standardization, and therefore the implementation provided by the host is subject to change. -Read/write functionality is provided by a system plugin in the `clientdata` namespace. That plugin provides a high level key/value API for plugins to use that is specific to psibase plugins. +To enable database actions within plugins, a plugin should import the ergonomic wrapper around the host's `wasi:keyvalue` interface. This wrapper is provided by a system plugin in the `clientdata` namespace. -As an MVP implementation, the API provides a get/set implementation: -``` - set : func(key : string, value : string) -> result<_, error>; - get : func(key : string) -> result; -``` - -
- Phase 2 API - -> Phase 2: The API should mirror the [idb-keyval](https://github.com/jakearchibald/idb-keyval) API, except for the `update` function, which would require passing functions: +This `clientdata` plugin provides a high level key/value API for plugins. ``` - set : func(key : string, value : string) -> result<_, error>; - get : func(key : string) -> result; - set-many : func(items: list>) -> result<_, error>; - get-many : func(keys: list) -> result, error>; - del : func(key: string) -> result<_, error>; - del-many : func(keys: list) -> result<_, error>; - clear : func() -> result<_, error>; - entries : func() -> result>, error>; - keys : func() -> result, error>; - values : func() -> result, error>; + get: func(key: string) -> option>; + set: func(key: string, value: list) -> result<_, error>; + delete: func(key: string); ``` -
- -#### Design implications - -There is actually no standardized wasi-keyvalue interface (The [proposal](https://github.com/WebAssembly/wasi-keyvalue) is still in stage 2 at the time of writing), so any interface we enable today will be subject to future change. This design specifies that plugins should import a custom psibase-plugin-specific API that itself leverages the `wasi-keyvalue` standard. -### Host provided functionality - -The `clientdata` plugin must itself import functionality that allows it to interact with some persistent storage layer. - -This host functionality is a browser shim implementation of the `wasi-keyvalue` API. Technically, any plugin can import this API and use it directly. But using the standard clientdata plugin simplifies the integration (uses standard error types, exposes a more convenient interface). +### Data backing The `wasi-keyvalue` implementation currently uses LocalStorage as its data backing, because it is a synchronous interface which is easiest to integrate with wasm. -> Phase 2: Wasi-keyvalue shim should be written to make use of the [idb-keyval](https://github.com/jakearchibald/idb-keyval) library (or simply `idb`), which is built on top of the browser's IndexedDB storage API. This would be easiest if it could be postponed until [JSPI](https://github.com/WebAssembly/js-promise-integration) is farther along in its standardization/integration process. This will allow much more storage, and also allow subsequent phases of the persistence design (see below). +Note: This imposes a major restriction on the amount of memory available for psibase plugins: Currently there is a hard limit of 5MB for the total amount of data stored in LocalStorage across all psibase apps. This restriction will be lifted in a future implementation of plugin storage that uses a different data backing layer such as IndexedDB. ### Persistence -When private user data is written, it is immediately persisted. - -> Phase 2: User data shall not be immediately persisted. It shall only be persisted after all plugin calls have popped off the stack with successful return values. - -> Phase 3+: The client-side data should work similarly to the server-side data. If a transaction was scheduled for execution from the plugin execution context, then any written data is not persisted until after an irreversible block is created that contains the successfully executed transaction. We can revisit this design later, but perhaps any client data writes during the window of uncertainty (before irreversible) happen on both the head branch and on the pending branch. If the pending branch is confirmed irreversible, the writes to the head branch are thrown out, and if the transaction is forked out then the writes to the pending branch are thrown out. +When a plugin writes private user data, it is immediately persisted. ### Concurrency @@ -171,5 +142,4 @@ All reads/writes are currently synchronous, with no concurrency. ### Synchronization across devices -Plugins all run within the domain of the supervisor, which ensures that all plugin data is written-to/read-from a storage backing tied to the supervisor domain. Cloning storage in the supervisor domain between TLDs is a way to synchronize plugin data across devices. - +Plugins all run within the domain of the supervisor, which ensures that all plugin data is written-to/read-from a storage backing tied to the supervisor domain. Cloning storage in the supervisor domain between top-level domains is a way to synchronize plugin data across devices.