From 1a309a49ae1a782f69244d81495c8d2f2377a441 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 12:34:04 +0100 Subject: [PATCH 01/15] Added initial dev guide --- docs/dev-guide.md | 100 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 docs/dev-guide.md diff --git a/docs/dev-guide.md b/docs/dev-guide.md new file mode 100644 index 00000000..fa30b474 --- /dev/null +++ b/docs/dev-guide.md @@ -0,0 +1,100 @@ +# Developer Guide + +This guide introduces all the concepts needed to build applications that use Walrus as a storage +or availability layer. The [overview](./overview..md) provides more background and explains in +more detail how Walrus operates internally. + +**Disclaimer about the Walrus developer preview: This release of Walrus \& Walrus Sites is a +developer preview, to showcase the technology and solicit feedback from builders. All storage nodes +and aggregators are operated by MystenLabs, all transactions are executed on the Sui testnet, +and use testnet SUI which has no value. The state of the store can be, and will be wiped, at any +point and possibly with no warning. Do not rely on this the developer preview for any production +purposes, it comes with no availability or persistence guarantees.** + +## Components + +From a developer perspective Walrus has some components that are objects and smart contracts on +Sui, and some components that are an independent set of services. As a rule Sui is used to manage +blob and storage node metadata, while off-sui components are used to actually store and read blob +data, which can be large. + +Walrus defines a number of objects and smart contracts on Sui: + +- A shared *system object*, records and manages the current committee of storage nodes. +- *Storage resources*, represent empty storage space that may be used to store blobs. +- *Blob resources*, represent blobs being registered and certified as stored. + +The system object ID for Walrus can be found in the Walrus `client_config.yaml` file. You may use +any Sui explorer to look at its content, as well as explore the content of blob objects. + +Walrus is also composed of a number of services and binaries: + +- A client (binary) can be executed locally and provides a + [Command Line Interface (CLI)](client-cli.html), a [JSON API](json-api.md) + and an [HTTP API](web-api.md) to perform Walrus operations. +- Aggregators are services that allow download of blobs via HTTP requests. +- Publishers are services used to upload blobs to Walrus. +- A set of storage nodes store encoded stored blobs. + +Aggregators, Publishers and other services use the client APIs to interact with Walrus. End-users +of services using walrus interact with the store via custom services, aggregators or publishers that +expose HTTP APIs to avoid the need to run locally a binary client. + +## Operations + +### Store + +Walrus may be used to **store a blob**, via the native client APIs or a publisher. Under the hood a +number of operations happen both on Sui as well as on storage nodes: + +- The client or publisher encodes the blob and derives a *blob ID* that identifies the blob. This + us a `u256` number often encoded as a URL safe base64 string. +- A transaction is executed on Sui to purchase some storage from the system object, and then to + *register the blob ID* with this storage. Client APIs return the *Sui blob object ID*. The + transactions use SUI to purchase storage and pay for gas. +- Encoded slivers of the blob are distributed to all storage nodes. They each sign a receipt. +- The signed receipts are aggregated and submitted to the Sui blob object to *certify the blob*. + Certifying a blob emits a Sui event with the blob ID and the period of availability. + +A blob can be considered as available on Walrus once the corresponding Sui blob object has been +certified in the final step. The steps involved in a store can be executed by the binary client, +or a publisher that accepts and publishes blobs via HTTP. + +Walrus currently allows the storage of up to XXXX bytes using the client, and YYY bytes using the +aggregator. You may store larger blobs by splitting them into smaller chunks. TODO sizes. + +### Read + +Walrus can then be used to **read a blob** by providing its blob ID. A read is executed by +performing the following steps: + +- The system object on Sui is read to determine the Walrus storage node committee. +- A number of storage nodes are queried for the slivers they store. +- The blob is reconstructed and checked against the blob ID from the recovered slivers. + +The steps involved in the read operation are performed by the binary client, or the aggregator +service that exposes an HTTP interface to read blobs. + +### Certify Availability + +One may *certify the availability of a blob* using Sui. This may be done in 3 different ways: + +- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be + used to authenticate the certified blob event emitted when the blob ID was certified on Sui. +- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be + used to authenticate the blob Sui object corresponding to the blob ID is certified. +- A Sui smart contract can be provided with the blob object on Sui (or a reference to it) to check + is is certified. + +The underlying protocol of the Sui light client returns digitally signed evidence for emitted events +or objects, and can be used by off-line or non-interactive applications as a proof of availability +for the blob ID for a certain number of epochs. + +Once a blob is certified, the Walrus store will ensure that sufficient slivers will always be +available on storage nodes to be able to recover it within the specified epochs. + +## Walrus Sites + +Walrus Sites is a service that uses Walrus to store websites and serve them to web clients. + +TODO \ No newline at end of file From 485d8721bdce30e7f87d4927988978ff8cfcb40a Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 14:08:11 +0100 Subject: [PATCH 02/15] Added reference for Sui object --- docs/dev-guide.md | 107 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 106 insertions(+), 1 deletion(-) diff --git a/docs/dev-guide.md b/docs/dev-guide.md index fa30b474..8521866e 100644 --- a/docs/dev-guide.md +++ b/docs/dev-guide.md @@ -1,7 +1,7 @@ # Developer Guide This guide introduces all the concepts needed to build applications that use Walrus as a storage -or availability layer. The [overview](./overview..md) provides more background and explains in +or availability layer. The [overview](./overview.md) provides more background and explains in more detail how Walrus operates internally. **Disclaimer about the Walrus developer preview: This release of Walrus \& Walrus Sites is a @@ -93,6 +93,111 @@ for the blob ID for a certain number of epochs. Once a blob is certified, the Walrus store will ensure that sufficient slivers will always be available on storage nodes to be able to recover it within the specified epochs. +## Sui Structures Reference + +Walrus blobs are represented as Sui `Blob` types. A blob may be registered, indicating that the +storage nodes should expect slivers from a Blob ID to be stored. Then a blob can be certified +indicating that a sufficient number of slivers have been stored to guarantee the blob's +availability. When a blob is certified its `certified` filed contains the epoch in which it was +certified. + +A `Storage` object is always associated with a Blob, reserving enough space for +a long enough period for its storage. A certified blob is available for the period the +underlying storage resource guarantees storage. + +```move + /// The blob structure represents a blob that has been registered to with some + /// storage, and then may eventually be certified as being available in the + /// system. + public struct Blob has key, store { + id: UID, + stored_epoch: u64, + blob_id: u256, + size: u64, + erasure_code_type: u8, + certified: option::Option, // The epoch first certified, + // or None if not certified. + storage: Storage, + } + + /// Reservation for storage for a given period, which is inclusive start, + /// exclusive end. + public struct Storage has key, store { + id: UID, + start_epoch: u64, + end_epoch: u64, + storage_size: u64, + } +``` + +When a blob is first registered a `BlobRegistered` event is emitted that informs storage nodes +that they should expect slivers associated with its Blob ID. Eventually when the blob is +certified a `BlobCertified` is emitted containing information about the blob ID and the epoch +after which the blob will be deleted. Before that epoch the blob is guaranteed to be available. + + +```move + /// Signals a blob with meta-data is registered. + public struct BlobRegistered has copy, drop { + epoch: u64, + blob_id: u256, + size: u64, + erasure_code_type: u8, + end_epoch: u64, + } + + /// Signals a blob is certified. + public struct BlobCertified has copy, drop { + epoch: u64, + blob_id: u256, + end_epoch: u64, + } + + /// Signals that a BlobID is invalid. + public struct InvalidBlobID has copy, drop { + epoch: u64, // The epoch in which the blob ID is first registered as invalid + blob_id: u256, + } +``` + +The Walrus system object contains meta-data about the available and used storage, as well as the +price of storage per 100 Kib of storage in MIST. The committee +structure within the system object can be used to read the current epoch number, as well as +information about the committee. + +```move + public struct System has key, store { + + id: UID, + + /// The current committee, with the current epoch. + /// The option is always Some, but need it for swap. + current_committee: Option, + + /// When we first enter the current epoch we SYNC, + /// and then we are DONE after a cert from a quorum. + epoch_status: u8, + + // Some accounting + total_capacity_size : u64, + used_capacity_size : u64, + + /// The price per unit size of storage. + price_per_unit_size: u64, + + /// Tables about the future and the past. + past_committees: Table, + future_accounting: FutureAccountingRingBuffer, + } + + public struct Committee has store { + epoch: u64, + bls_committee : BlsCommittee, + } + + +``` + ## Walrus Sites Walrus Sites is a service that uses Walrus to store websites and serve them to web clients. From e03be61c474d3f67f7d0878f7b836052bd01f6ef Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 14:43:58 +0100 Subject: [PATCH 03/15] Structured the dev guide in sections --- docs/SUMMARY.md | 4 + docs/components.md | 29 ++++++ docs/dev-guide.md | 197 ++--------------------------------------- docs/dev-operations.md | 52 +++++++++++ docs/sui-struct.md | 147 ++++++++++++++++++++++++++++++ 5 files changed, 238 insertions(+), 191 deletions(-) create mode 100644 docs/components.md create mode 100644 docs/dev-operations.md create mode 100644 docs/sui-struct.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 9df42782..41b6460e 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -19,6 +19,10 @@ # Usage +- [Developer Guide](dev-guide.md) + - [Components](components.md) + - [Operations](dev-operations.md) + - [Sui structures](sui-struct.md) - [Setup](./usage/setup.md) - [Prerequisites](./usage/prerequisites.md) - [Installation](./usage/installation.md) diff --git a/docs/components.md b/docs/components.md new file mode 100644 index 00000000..4af925d3 --- /dev/null +++ b/docs/components.md @@ -0,0 +1,29 @@ +# Components + +From a developer perspective Walrus has some components that are objects and smart contracts on +Sui, and some components that are an independent set of services. As a rule Sui is used to manage +blob and storage node metadata, while off-sui components are used to actually store and read blob +data, which can be large. + +Walrus defines a number of objects and smart contracts on Sui: + +- A shared *system object*, records and manages the current committee of storage nodes. +- *Storage resources*, represent empty storage space that may be used to store blobs. +- *Blob resources*, represent blobs being registered and certified as stored. + +The system object ID for Walrus can be found in the Walrus `client_config.yaml` file. You may use +any Sui explorer to look at its content, as well as explore the content of blob objects. You can +find more information in the [quick reference to the Walrus Sui structures](sui-struct.md). + +Walrus is also composed of a number of services and binaries: + +- A client (binary) can be executed locally and provides a + [Command Line Interface (CLI)](client-cli.html), a [JSON API](json-api.md) + and an [HTTP API](web-api.md) to perform Walrus operations. +- Aggregators are services that allow download of blobs via HTTP requests. +- Publishers are services used to upload blobs to Walrus. +- A set of storage nodes store encoded stored blobs. + +Aggregators, Publishers and other services use the client APIs to interact with Walrus. End-users +of services using walrus interact with the store via custom services, aggregators or publishers that +expose HTTP APIs to avoid the need to run locally a binary client. diff --git a/docs/dev-guide.md b/docs/dev-guide.md index 8521866e..b91f4183 100644 --- a/docs/dev-guide.md +++ b/docs/dev-guide.md @@ -11,195 +11,10 @@ and use testnet SUI which has no value. The state of the store can be, and will point and possibly with no warning. Do not rely on this the developer preview for any production purposes, it comes with no availability or persistence guarantees.** -## Components +This developer guide describes: -From a developer perspective Walrus has some components that are objects and smart contracts on -Sui, and some components that are an independent set of services. As a rule Sui is used to manage -blob and storage node metadata, while off-sui components are used to actually store and read blob -data, which can be large. - -Walrus defines a number of objects and smart contracts on Sui: - -- A shared *system object*, records and manages the current committee of storage nodes. -- *Storage resources*, represent empty storage space that may be used to store blobs. -- *Blob resources*, represent blobs being registered and certified as stored. - -The system object ID for Walrus can be found in the Walrus `client_config.yaml` file. You may use -any Sui explorer to look at its content, as well as explore the content of blob objects. - -Walrus is also composed of a number of services and binaries: - -- A client (binary) can be executed locally and provides a - [Command Line Interface (CLI)](client-cli.html), a [JSON API](json-api.md) - and an [HTTP API](web-api.md) to perform Walrus operations. -- Aggregators are services that allow download of blobs via HTTP requests. -- Publishers are services used to upload blobs to Walrus. -- A set of storage nodes store encoded stored blobs. - -Aggregators, Publishers and other services use the client APIs to interact with Walrus. End-users -of services using walrus interact with the store via custom services, aggregators or publishers that -expose HTTP APIs to avoid the need to run locally a binary client. - -## Operations - -### Store - -Walrus may be used to **store a blob**, via the native client APIs or a publisher. Under the hood a -number of operations happen both on Sui as well as on storage nodes: - -- The client or publisher encodes the blob and derives a *blob ID* that identifies the blob. This - us a `u256` number often encoded as a URL safe base64 string. -- A transaction is executed on Sui to purchase some storage from the system object, and then to - *register the blob ID* with this storage. Client APIs return the *Sui blob object ID*. The - transactions use SUI to purchase storage and pay for gas. -- Encoded slivers of the blob are distributed to all storage nodes. They each sign a receipt. -- The signed receipts are aggregated and submitted to the Sui blob object to *certify the blob*. - Certifying a blob emits a Sui event with the blob ID and the period of availability. - -A blob can be considered as available on Walrus once the corresponding Sui blob object has been -certified in the final step. The steps involved in a store can be executed by the binary client, -or a publisher that accepts and publishes blobs via HTTP. - -Walrus currently allows the storage of up to XXXX bytes using the client, and YYY bytes using the -aggregator. You may store larger blobs by splitting them into smaller chunks. TODO sizes. - -### Read - -Walrus can then be used to **read a blob** by providing its blob ID. A read is executed by -performing the following steps: - -- The system object on Sui is read to determine the Walrus storage node committee. -- A number of storage nodes are queried for the slivers they store. -- The blob is reconstructed and checked against the blob ID from the recovered slivers. - -The steps involved in the read operation are performed by the binary client, or the aggregator -service that exposes an HTTP interface to read blobs. - -### Certify Availability - -One may *certify the availability of a blob* using Sui. This may be done in 3 different ways: - -- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be - used to authenticate the certified blob event emitted when the blob ID was certified on Sui. -- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be - used to authenticate the blob Sui object corresponding to the blob ID is certified. -- A Sui smart contract can be provided with the blob object on Sui (or a reference to it) to check - is is certified. - -The underlying protocol of the Sui light client returns digitally signed evidence for emitted events -or objects, and can be used by off-line or non-interactive applications as a proof of availability -for the blob ID for a certain number of epochs. - -Once a blob is certified, the Walrus store will ensure that sufficient slivers will always be -available on storage nodes to be able to recover it within the specified epochs. - -## Sui Structures Reference - -Walrus blobs are represented as Sui `Blob` types. A blob may be registered, indicating that the -storage nodes should expect slivers from a Blob ID to be stored. Then a blob can be certified -indicating that a sufficient number of slivers have been stored to guarantee the blob's -availability. When a blob is certified its `certified` filed contains the epoch in which it was -certified. - -A `Storage` object is always associated with a Blob, reserving enough space for -a long enough period for its storage. A certified blob is available for the period the -underlying storage resource guarantees storage. - -```move - /// The blob structure represents a blob that has been registered to with some - /// storage, and then may eventually be certified as being available in the - /// system. - public struct Blob has key, store { - id: UID, - stored_epoch: u64, - blob_id: u256, - size: u64, - erasure_code_type: u8, - certified: option::Option, // The epoch first certified, - // or None if not certified. - storage: Storage, - } - - /// Reservation for storage for a given period, which is inclusive start, - /// exclusive end. - public struct Storage has key, store { - id: UID, - start_epoch: u64, - end_epoch: u64, - storage_size: u64, - } -``` - -When a blob is first registered a `BlobRegistered` event is emitted that informs storage nodes -that they should expect slivers associated with its Blob ID. Eventually when the blob is -certified a `BlobCertified` is emitted containing information about the blob ID and the epoch -after which the blob will be deleted. Before that epoch the blob is guaranteed to be available. - - -```move - /// Signals a blob with meta-data is registered. - public struct BlobRegistered has copy, drop { - epoch: u64, - blob_id: u256, - size: u64, - erasure_code_type: u8, - end_epoch: u64, - } - - /// Signals a blob is certified. - public struct BlobCertified has copy, drop { - epoch: u64, - blob_id: u256, - end_epoch: u64, - } - - /// Signals that a BlobID is invalid. - public struct InvalidBlobID has copy, drop { - epoch: u64, // The epoch in which the blob ID is first registered as invalid - blob_id: u256, - } -``` - -The Walrus system object contains meta-data about the available and used storage, as well as the -price of storage per 100 Kib of storage in MIST. The committee -structure within the system object can be used to read the current epoch number, as well as -information about the committee. - -```move - public struct System has key, store { - - id: UID, - - /// The current committee, with the current epoch. - /// The option is always Some, but need it for swap. - current_committee: Option, - - /// When we first enter the current epoch we SYNC, - /// and then we are DONE after a cert from a quorum. - epoch_status: u8, - - // Some accounting - total_capacity_size : u64, - used_capacity_size : u64, - - /// The price per unit size of storage. - price_per_unit_size: u64, - - /// Tables about the future and the past. - past_committees: Table, - future_accounting: FutureAccountingRingBuffer, - } - - public struct Committee has store { - epoch: u64, - bls_committee : BlsCommittee, - } - - -``` - -## Walrus Sites - -Walrus Sites is a service that uses Walrus to store websites and serve them to web clients. - -TODO \ No newline at end of file +- [Components](components.md) of Walrus of interest to developers that wish to use it for + storage or availability. +- [Operations](dev-operations.md) supported through client binaries, APIs, or Sui operations. +- [The Sui structures](sui-struct.md) Walrus uses to store metadata, and how they can be read + from Sui smart contracts, or through the Sui SDK. \ No newline at end of file diff --git a/docs/dev-operations.md b/docs/dev-operations.md new file mode 100644 index 00000000..77280c8c --- /dev/null +++ b/docs/dev-operations.md @@ -0,0 +1,52 @@ +# Operations + +### Store + +Walrus may be used to **store a blob**, via the native client APIs or a publisher. Under the hood a +number of operations happen both on Sui as well as on storage nodes: + +- The client or publisher encodes the blob and derives a *blob ID* that identifies the blob. This + us a `u256` number often encoded as a URL safe base64 string. +- A transaction is executed on Sui to purchase some storage from the system object, and then to + *register the blob ID* with this storage. Client APIs return the *Sui blob object ID*. The + transactions use SUI to purchase storage and pay for gas. +- Encoded slivers of the blob are distributed to all storage nodes. They each sign a receipt. +- The signed receipts are aggregated and submitted to the Sui blob object to *certify the blob*. + Certifying a blob emits a Sui event with the blob ID and the period of availability. + +A blob can be considered as available on Walrus once the corresponding Sui blob object has been +certified in the final step. The steps involved in a store can be executed by the binary client, +or a publisher that accepts and publishes blobs via HTTP. + +Walrus currently allows the storage of up to XXXX bytes using the client, and YYY bytes using the +aggregator. You may store larger blobs by splitting them into smaller chunks. TODO sizes. + +### Read + +Walrus can then be used to **read a blob** by providing its blob ID. A read is executed by +performing the following steps: + +- The system object on Sui is read to determine the Walrus storage node committee. +- A number of storage nodes are queried for the slivers they store. +- The blob is reconstructed and checked against the blob ID from the recovered slivers. + +The steps involved in the read operation are performed by the binary client, or the aggregator +service that exposes an HTTP interface to read blobs. + +### Certify Availability + +One may *certify the availability of a blob* using Sui. This may be done in 3 different ways: + +- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be + used to authenticate the certified blob event emitted when the blob ID was certified on Sui. +- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be + used to authenticate the blob Sui object corresponding to the blob ID is certified. +- A Sui smart contract can be provided with the blob object on Sui (or a reference to it) to check + is is certified. + +The underlying protocol of the Sui light client returns digitally signed evidence for emitted events +or objects, and can be used by off-line or non-interactive applications as a proof of availability +for the blob ID for a certain number of epochs. + +Once a blob is certified, the Walrus store will ensure that sufficient slivers will always be +available on storage nodes to be able to recover it within the specified epochs. diff --git a/docs/sui-struct.md b/docs/sui-struct.md new file mode 100644 index 00000000..440989ad --- /dev/null +++ b/docs/sui-struct.md @@ -0,0 +1,147 @@ + +# Walrus Sui Reference + +## Blob and Storage Objects + +Walrus blobs are represented as Sui `Blob` types. A blob may be registered, indicating that the +storage nodes should expect slivers from a Blob ID to be stored. Then a blob can be certified +indicating that a sufficient number of slivers have been stored to guarantee the blob's +availability. When a blob is certified its `certified` filed contains the epoch in which it was +certified. + +A `Storage` object is always associated with a Blob, reserving enough space for +a long enough period for its storage. A certified blob is available for the period the +underlying storage resource guarantees storage. + +```move +/// The blob structure represents a blob that has been registered to with some +/// storage, and then may eventually be certified as being available in the +/// system. +public struct Blob has key, store { + id: UID, + stored_epoch: u64, + blob_id: u256, + size: u64, + erasure_code_type: u8, + certified: option::Option, // The epoch first certified, + // or None if not certified. + storage: Storage, +} + +/// Reservation for storage for a given period, which is inclusive start, +/// exclusive end. +public struct Storage has key, store { + id: UID, + start_epoch: u64, + end_epoch: u64, + storage_size: u64, +} +``` + +All fields of `Blob` and `Storage` objects can be read using the expected functions: +```move +// Blob functions +public fun stored_epoch(b: &Blob) : u64; +public fun blob_id(b: &Blob) : u256; +public fun size(b: &Blob) : u64; +public fun erasure_code_type(b: &Blob) : u8; +public fun certified(b: &Blob) : &Option; +public fun storage(b: &Blob) : &Storage; + +// Storage functions +public fun start_epoch(self: &Storage) : u64; +public fun end_epoch(self: &Storage) : u64; +public fun storage_size(self: &Storage) : u64; + +``` + + +## Events + +When a blob is first registered a `BlobRegistered` event is emitted that informs storage nodes +that they should expect slivers associated with its Blob ID. Eventually when the blob is +certified a `BlobCertified` is emitted containing information about the blob ID and the epoch +after which the blob will be deleted. Before that epoch the blob is guaranteed to be available. + + +```move +/// Signals a blob with meta-data is registered. +public struct BlobRegistered has copy, drop { + epoch: u64, + blob_id: u256, + size: u64, + erasure_code_type: u8, + end_epoch: u64, +} + +/// Signals a blob is certified. +public struct BlobCertified has copy, drop { + epoch: u64, + blob_id: u256, + end_epoch: u64, +} + +/// Signals that a BlobID is invalid. +public struct InvalidBlobID has copy, drop { + epoch: u64, // The epoch in which the blob ID is first registered as invalid + blob_id: u256, +} +``` + +The `InvalidBlobID` event is emitted when storage nodes detect and incorrectly encoded blob. +Such a blob is guaranteed to be also detected as invalid when a read is attempted. + +## Walrus System information + +The Walrus system object contains meta-data about the available and used storage, as well as the +price of storage per 100 Kib of storage in MIST. The committee +structure within the system object can be used to read the current epoch number, as well as +information about the committee. + +```move +public struct System has key, store { + + id: UID, + + /// The current committee, with the current epoch. + /// The option is always Some, but need it for swap. + current_committee: Option, + + /// When we first enter the current epoch we SYNC, + /// and then we are DONE after a cert from a quorum. + epoch_status: u8, + + // Some accounting + total_capacity_size : u64, + used_capacity_size : u64, + + /// The price per unit size of storage. + price_per_unit_size: u64, + + /// Tables about the future and the past. + past_committees: Table, + future_accounting: FutureAccountingRingBuffer, +} + +public struct Committee has store { + epoch: u64, + bls_committee : BlsCommittee, +} +``` + +A few public functions of the committee allow contracts to read metadata: + +```move +/// Get epoch. Uses the committee to get the epoch. +public fun epoch(self: &System) : u64; + +/// Accessor for total capacity size. +public fun total_capacity_size(self: &System) : u64; + +/// Accessor for used capacity size. +public fun used_capacity_size(self: &System) : u64; + +// The number of shards +public fun n_shards(self: &System) : u16; + +``` \ No newline at end of file From 2c60d6f73bb6673ac7912bcf520bfab81a43358e Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 14:52:56 +0100 Subject: [PATCH 04/15] Added explanation to Sui ref --- docs/sui-struct.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/sui-struct.md b/docs/sui-struct.md index 440989ad..e31d87eb 100644 --- a/docs/sui-struct.md +++ b/docs/sui-struct.md @@ -1,6 +1,15 @@ # Walrus Sui Reference +This section is optional and enables advanced use cases. + +You can interact with Walrus purely +through the client CLI, and JSON or HTTP APIs provided, without reading or writing to the Sui chain +directly. However, Sui is used to manage the metadata of Walrus and developers can use information +about the Walrus system, as well as stored blobs using Sui smart contracts. + +This section provides and overview of how you may use Walrus objects in your Sui smart contracts. + ## Blob and Storage Objects Walrus blobs are represented as Sui `Blob` types. A blob may be registered, indicating that the From 5cedff2ee7dc7da27d5bf6e7dc981804694c0b79 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 16:36:10 +0100 Subject: [PATCH 05/15] Copy edited and proof read docs --- docs/SUMMARY.md | 4 +++- docs/components.md | 30 +++++++++++++++++------------- docs/configuration.md | 1 + docs/dev-guide.md | 19 +++++++++++-------- docs/dev-operations.md | 28 ++++++++++++++++------------ docs/examples.md | 1 + docs/python.md | 13 +++++++++++++ docs/sui-struct.md | 34 ++++++++++++++++++++-------------- 8 files changed, 82 insertions(+), 48 deletions(-) create mode 100644 docs/configuration.md create mode 100644 docs/examples.md create mode 100644 docs/python.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 41b6460e..b97c4fb5 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -23,6 +23,7 @@ - [Components](components.md) - [Operations](dev-operations.md) - [Sui structures](sui-struct.md) + - [Setup](./usage/setup.md) - [Prerequisites](./usage/prerequisites.md) - [Installation](./usage/installation.md) @@ -31,7 +32,8 @@ - [Using the client CLI](./usage/client-cli.md) - [Using the client JSON API](./usage/json-api.md) - [Using the client HTTP API](./usage/web-api.md) -- [Examples]() +- [Examples](examples.md) + - [Python](python.md) # Walrus sites diff --git a/docs/components.md b/docs/components.md index 4af925d3..8b4efd2c 100644 --- a/docs/components.md +++ b/docs/components.md @@ -1,29 +1,33 @@ # Components -From a developer perspective Walrus has some components that are objects and smart contracts on -Sui, and some components that are an independent set of services. As a rule Sui is used to manage -blob and storage node metadata, while off-sui components are used to actually store and read blob -data, which can be large. +From a developer perspective, some Walrus components are objects and smart contracts on +Sui, and some components are Walrus-specific binaries and services. As a rule Sui is used to +manage blob and storage node meta-data, while Walrus-specific services are used to store and +read blob contents, which can be very large. Walrus defines a number of objects and smart contracts on Sui: - A shared *system object*, records and manages the current committee of storage nodes. - *Storage resources*, represent empty storage space that may be used to store blobs. - *Blob resources*, represent blobs being registered and certified as stored. +- Changes to these objects emit *Walrus-related events*. -The system object ID for Walrus can be found in the Walrus `client_config.yaml` file. You may use -any Sui explorer to look at its content, as well as explore the content of blob objects. You can -find more information in the [quick reference to the Walrus Sui structures](sui-struct.md). +The Walrus system object ID can be found in the Walrus `client_config.yaml` file +(see [Configuration](configuration.md)). You may use +any Sui explorer to look at its content, as well as explore the content of blob objects. +There is more information about these in the +[quick reference to the Walrus Sui structures](sui-struct.md). -Walrus is also composed of a number of services and binaries: +Walrus is also composed of a number of Walrus-specific services and binaries: - A client (binary) can be executed locally and provides a [Command Line Interface (CLI)](client-cli.html), a [JSON API](json-api.md) and an [HTTP API](web-api.md) to perform Walrus operations. -- Aggregators are services that allow download of blobs via HTTP requests. -- Publishers are services used to upload blobs to Walrus. -- A set of storage nodes store encoded stored blobs. +- Aggregators services allow reading blobs via HTTP requests. +- Publishers services are used store blobs to Walrus. +- A set of storage nodes store encoded stored blobs. These nodes form the decentralized + storage infrastructure of Walrus. -Aggregators, Publishers and other services use the client APIs to interact with Walrus. End-users -of services using walrus interact with the store via custom services, aggregators or publishers that +Aggregators, publishers and other services use the client APIs to interact with Walrus. End-users +of services using Walrus interact with the store via custom services, aggregators or publishers that expose HTTP APIs to avoid the need to run locally a binary client. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 00000000..a025a48b --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1 @@ +# Configuration diff --git a/docs/dev-guide.md b/docs/dev-guide.md index b91f4183..045e8c90 100644 --- a/docs/dev-guide.md +++ b/docs/dev-guide.md @@ -4,17 +4,20 @@ This guide introduces all the concepts needed to build applications that use Wal or availability layer. The [overview](./overview.md) provides more background and explains in more detail how Walrus operates internally. -**Disclaimer about the Walrus developer preview: This release of Walrus \& Walrus Sites is a -developer preview, to showcase the technology and solicit feedback from builders. All storage nodes -and aggregators are operated by MystenLabs, all transactions are executed on the Sui testnet, -and use testnet SUI which has no value. The state of the store can be, and will be wiped, at any -point and possibly with no warning. Do not rely on this the developer preview for any production -purposes, it comes with no availability or persistence guarantees.** - This developer guide describes: - [Components](components.md) of Walrus of interest to developers that wish to use it for storage or availability. - [Operations](dev-operations.md) supported through client binaries, APIs, or Sui operations. - [The Sui structures](sui-struct.md) Walrus uses to store metadata, and how they can be read - from Sui smart contracts, or through the Sui SDK. \ No newline at end of file + from Sui smart contracts, or through the Sui SDK. + + +## Disclaimer about the Walrus developer preview + +**This release of Walrus \& Walrus Sites is a +developer preview, to showcase the technology and solicit feedback from builders. All storage nodes +and aggregators are operated by MystenLabs, all transactions are executed on the Sui testnet, +and use testnet SUI which has no value. The state of the store can be, and will be wiped, at any +point and possibly with no warning. Do not rely on this the developer preview for any production +purposes, it comes with no availability or persistence guarantees.** diff --git a/docs/dev-operations.md b/docs/dev-operations.md index 77280c8c..8f2923fa 100644 --- a/docs/dev-operations.md +++ b/docs/dev-operations.md @@ -6,15 +6,15 @@ Walrus may be used to **store a blob**, via the native client APIs or a publishe number of operations happen both on Sui as well as on storage nodes: - The client or publisher encodes the blob and derives a *blob ID* that identifies the blob. This - us a `u256` number often encoded as a URL safe base64 string. + is a `u256` often encoded as a URL safe base64 string. - A transaction is executed on Sui to purchase some storage from the system object, and then to *register the blob ID* with this storage. Client APIs return the *Sui blob object ID*. The transactions use SUI to purchase storage and pay for gas. - Encoded slivers of the blob are distributed to all storage nodes. They each sign a receipt. -- The signed receipts are aggregated and submitted to the Sui blob object to *certify the blob*. +- Signed receipts are aggregated and submitted to the Sui blob object to *certify the blob*. Certifying a blob emits a Sui event with the blob ID and the period of availability. -A blob can be considered as available on Walrus once the corresponding Sui blob object has been +A blob is considered available on Walrus once the corresponding Sui blob object has been certified in the final step. The steps involved in a store can be executed by the binary client, or a publisher that accepts and publishes blobs via HTTP. @@ -31,22 +31,26 @@ performing the following steps: - The blob is reconstructed and checked against the blob ID from the recovered slivers. The steps involved in the read operation are performed by the binary client, or the aggregator -service that exposes an HTTP interface to read blobs. +service that exposes an HTTP interface to read blobs. Reads are extremely resilient and will +succeed in recovering the blob stored even if up to one-third of storage nodes are +unavailable. ### Certify Availability -One may *certify the availability of a blob* using Sui. This may be done in 3 different ways: +Walrus can be used to **certify the availability of a blob** using Sui. This may be done in 3 +different ways: -- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be +- A Sui SDK read can be used to authenticate the certified blob event emitted when the blob ID was certified on Sui. -- A Sui [light-client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) may be - used to authenticate the blob Sui object corresponding to the blob ID is certified. -- A Sui smart contract can be provided with the blob object on Sui (or a reference to it) to check +- A Sui SDK read may be + used to authenticate the Sui blob object corresponding to the blob ID, and check it is certified. +- A Sui smart contract can read the blob object on Sui (or a reference to it) to check is is certified. -The underlying protocol of the Sui light client returns digitally signed evidence for emitted events +The underlying protocol of the [Sui light client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) +client returns digitally signed evidence for emitted events or objects, and can be used by off-line or non-interactive applications as a proof of availability for the blob ID for a certain number of epochs. -Once a blob is certified, the Walrus store will ensure that sufficient slivers will always be -available on storage nodes to be able to recover it within the specified epochs. +Once a blob is certified, Walrus will ensure that sufficient slivers will always be +available on storage nodes to recover it within the specified epochs. diff --git a/docs/examples.md b/docs/examples.md new file mode 100644 index 00000000..df635b4e --- /dev/null +++ b/docs/examples.md @@ -0,0 +1 @@ +# Examples diff --git a/docs/python.md b/docs/python.md new file mode 100644 index 00000000..5afc5e9d --- /dev/null +++ b/docs/python.md @@ -0,0 +1,13 @@ +# Python + +The [python examples](https://github.com/MystenLabs/walrus-docs/tree/main/examples/python) folder +contains a number of examples: + +- How to [use the HTTP API](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_webapi.py) + to store and read a blob. +- How to [use the JSON API](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_jsonapi.py) + to store, read, and check the availability of a blob. Checking the certification of a blob + illustrates reading the Blob Sui object that certifies + (see the [Walrus Sui reference](sui-struct.md)). +- How to [read information from the Walrus system object](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_sui_system.py). +- How to [track Walrus related Events](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/track_walrus_events.py). diff --git a/docs/sui-struct.md b/docs/sui-struct.md index e31d87eb..d155e448 100644 --- a/docs/sui-struct.md +++ b/docs/sui-struct.md @@ -1,27 +1,30 @@ -# Walrus Sui Reference +# Sui Structures This section is optional and enables advanced use cases. You can interact with Walrus purely -through the client CLI, and JSON or HTTP APIs provided, without reading or writing to the Sui chain -directly. However, Sui is used to manage the metadata of Walrus and developers can use information -about the Walrus system, as well as stored blobs using Sui smart contracts. +through the client CLI, and JSON or HTTP APIs provided, without querying or executing transactions +on Sui directly. However, Walrus uses Sui to manage its meta-data and smart contract developers can +read information about the Walrus system, as well as stored blobs, on Sui. This section provides and overview of how you may use Walrus objects in your Sui smart contracts. ## Blob and Storage Objects -Walrus blobs are represented as Sui `Blob` types. A blob may be registered, indicating that the -storage nodes should expect slivers from a Blob ID to be stored. Then a blob can be certified +Walrus blobs are represented as Sui objects of type `Blob`. A blob is first registered, indicating +that the storage nodes should expect slivers from a Blob ID to be stored. Then a blob is certified indicating that a sufficient number of slivers have been stored to guarantee the blob's -availability. When a blob is certified its `certified` filed contains the epoch in which it was +availability. When a blob is certified its `certified` field contains the epoch in which it was certified. -A `Storage` object is always associated with a Blob, reserving enough space for -a long enough period for its storage. A certified blob is available for the period the +A `Storage` object is always associated with a `Blob` object, reserving enough space for +a long enough period for the blob's storage. A certified blob is available for the period the underlying storage resource guarantees storage. +Concretely, `Blob` and `Storage` objects have the following fields, that can be read through the +Sui SDKs: + ```move /// The blob structure represents a blob that has been registered to with some /// storage, and then may eventually be certified as being available in the @@ -89,7 +92,13 @@ public struct BlobCertified has copy, drop { blob_id: u256, end_epoch: u64, } +``` + +The `InvalidBlobID` event is emitted when storage nodes detect and incorrectly encoded blob. +Such a blob is guaranteed to be also detected as invalid when a read is attempted. + +```move /// Signals that a BlobID is invalid. public struct InvalidBlobID has copy, drop { epoch: u64, // The epoch in which the blob ID is first registered as invalid @@ -97,10 +106,7 @@ public struct InvalidBlobID has copy, drop { } ``` -The `InvalidBlobID` event is emitted when storage nodes detect and incorrectly encoded blob. -Such a blob is guaranteed to be also detected as invalid when a read is attempted. - -## Walrus System information +## System information The Walrus system object contains meta-data about the available and used storage, as well as the price of storage per 100 Kib of storage in MIST. The committee @@ -138,7 +144,7 @@ public struct Committee has store { } ``` -A few public functions of the committee allow contracts to read metadata: +A few public functions of the committee allow contracts to read Walrus meta-data: ```move /// Get epoch. Uses the committee to get the epoch. From 342fced5c290278d77efc7f4eb20dca57ee37ec9 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 16:37:48 +0100 Subject: [PATCH 06/15] Appease linter --- docs/dev-operations.md | 6 +++--- docs/sui-struct.md | 3 +-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/dev-operations.md b/docs/dev-operations.md index 8f2923fa..ea579e0a 100644 --- a/docs/dev-operations.md +++ b/docs/dev-operations.md @@ -1,6 +1,6 @@ # Operations -### Store +## Store Walrus may be used to **store a blob**, via the native client APIs or a publisher. Under the hood a number of operations happen both on Sui as well as on storage nodes: @@ -21,7 +21,7 @@ or a publisher that accepts and publishes blobs via HTTP. Walrus currently allows the storage of up to XXXX bytes using the client, and YYY bytes using the aggregator. You may store larger blobs by splitting them into smaller chunks. TODO sizes. -### Read +## Read Walrus can then be used to **read a blob** by providing its blob ID. A read is executed by performing the following steps: @@ -35,7 +35,7 @@ service that exposes an HTTP interface to read blobs. Reads are extremely resili succeed in recovering the blob stored even if up to one-third of storage nodes are unavailable. -### Certify Availability +## Certify Availability Walrus can be used to **certify the availability of a blob** using Sui. This may be done in 3 different ways: diff --git a/docs/sui-struct.md b/docs/sui-struct.md index d155e448..ef6642ce 100644 --- a/docs/sui-struct.md +++ b/docs/sui-struct.md @@ -67,7 +67,6 @@ public fun storage_size(self: &Storage) : u64; ``` - ## Events When a blob is first registered a `BlobRegistered` event is emitted that informs storage nodes @@ -159,4 +158,4 @@ public fun used_capacity_size(self: &System) : u64; // The number of shards public fun n_shards(self: &System) : u16; -``` \ No newline at end of file +``` From 889dd841a07a52d6ed99fbdc2dcc59060b375086 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 16:42:08 +0100 Subject: [PATCH 07/15] Lint more --- docs/dev-guide.md | 1 - docs/python.md | 2 ++ docs/sui-struct.md | 4 +--- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/dev-guide.md b/docs/dev-guide.md index 045e8c90..dd8321d3 100644 --- a/docs/dev-guide.md +++ b/docs/dev-guide.md @@ -12,7 +12,6 @@ This developer guide describes: - [The Sui structures](sui-struct.md) Walrus uses to store metadata, and how they can be read from Sui smart contracts, or through the Sui SDK. - ## Disclaimer about the Walrus developer preview **This release of Walrus \& Walrus Sites is a diff --git a/docs/python.md b/docs/python.md index 5afc5e9d..b4d414c7 100644 --- a/docs/python.md +++ b/docs/python.md @@ -3,6 +3,7 @@ The [python examples](https://github.com/MystenLabs/walrus-docs/tree/main/examples/python) folder contains a number of examples: + - How to [use the HTTP API](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_webapi.py) to store and read a blob. - How to [use the JSON API](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_jsonapi.py) @@ -11,3 +12,4 @@ contains a number of examples: (see the [Walrus Sui reference](sui-struct.md)). - How to [read information from the Walrus system object](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_sui_system.py). - How to [track Walrus related Events](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/track_walrus_events.py). + diff --git a/docs/sui-struct.md b/docs/sui-struct.md index ef6642ce..0ca4d2ac 100644 --- a/docs/sui-struct.md +++ b/docs/sui-struct.md @@ -51,6 +51,7 @@ public struct Storage has key, store { ``` All fields of `Blob` and `Storage` objects can be read using the expected functions: + ```move // Blob functions public fun stored_epoch(b: &Blob) : u64; @@ -64,7 +65,6 @@ public fun storage(b: &Blob) : &Storage; public fun start_epoch(self: &Storage) : u64; public fun end_epoch(self: &Storage) : u64; public fun storage_size(self: &Storage) : u64; - ``` ## Events @@ -74,7 +74,6 @@ that they should expect slivers associated with its Blob ID. Eventually when the certified a `BlobCertified` is emitted containing information about the blob ID and the epoch after which the blob will be deleted. Before that epoch the blob is guaranteed to be available. - ```move /// Signals a blob with meta-data is registered. public struct BlobRegistered has copy, drop { @@ -96,7 +95,6 @@ public struct BlobCertified has copy, drop { The `InvalidBlobID` event is emitted when storage nodes detect and incorrectly encoded blob. Such a blob is guaranteed to be also detected as invalid when a read is attempted. - ```move /// Signals that a BlobID is invalid. public struct InvalidBlobID has copy, drop { From 161cd1092ae306b1cb9188421cd979903e65ac99 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Tue, 11 Jun 2024 16:43:02 +0100 Subject: [PATCH 08/15] More lint --- docs/dev-operations.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/dev-operations.md b/docs/dev-operations.md index ea579e0a..80fb72da 100644 --- a/docs/dev-operations.md +++ b/docs/dev-operations.md @@ -47,7 +47,8 @@ different ways: - A Sui smart contract can read the blob object on Sui (or a reference to it) to check is is certified. -The underlying protocol of the [Sui light client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) +The underlying protocol of the +[Sui light client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) client returns digitally signed evidence for emitted events or objects, and can be used by off-line or non-interactive applications as a proof of availability for the blob ID for a certain number of epochs. From 49f48c6d42ac1a95d86b9a08affb237741a864e4 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Wed, 12 Jun 2024 12:22:42 +0100 Subject: [PATCH 09/15] Added changes post review --- docs/SUMMARY.md | 1 - docs/components.md | 2 +- docs/dev-guide.md | 4 ++-- docs/dev-operations.md | 14 ++++++++------ docs/examples.md | 6 ++++++ docs/sui-struct.md | 17 +++++++++-------- 6 files changed, 26 insertions(+), 18 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index b97c4fb5..df039056 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -23,7 +23,6 @@ - [Components](components.md) - [Operations](dev-operations.md) - [Sui structures](sui-struct.md) - - [Setup](./usage/setup.md) - [Prerequisites](./usage/prerequisites.md) - [Installation](./usage/installation.md) diff --git a/docs/components.md b/docs/components.md index 8b4efd2c..ba58911e 100644 --- a/docs/components.md +++ b/docs/components.md @@ -2,7 +2,7 @@ From a developer perspective, some Walrus components are objects and smart contracts on Sui, and some components are Walrus-specific binaries and services. As a rule Sui is used to -manage blob and storage node meta-data, while Walrus-specific services are used to store and +manage blob and storage node metadata, while Walrus-specific services are used to store and read blob contents, which can be very large. Walrus defines a number of objects and smart contracts on Sui: diff --git a/docs/dev-guide.md b/docs/dev-guide.md index dd8321d3..311ba400 100644 --- a/docs/dev-guide.md +++ b/docs/dev-guide.md @@ -16,7 +16,7 @@ This developer guide describes: **This release of Walrus \& Walrus Sites is a developer preview, to showcase the technology and solicit feedback from builders. All storage nodes -and aggregators are operated by MystenLabs, all transactions are executed on the Sui testnet, +and aggregators are operated by Mysten Labs, all transactions are executed on the Sui testnet, and use testnet SUI which has no value. The state of the store can be, and will be wiped, at any -point and possibly with no warning. Do not rely on this the developer preview for any production +point and possibly with no warning. Do not rely on this developer preview for any production purposes, it comes with no availability or persistence guarantees.** diff --git a/docs/dev-operations.md b/docs/dev-operations.md index 80fb72da..b32a25f8 100644 --- a/docs/dev-operations.md +++ b/docs/dev-operations.md @@ -18,8 +18,9 @@ A blob is considered available on Walrus once the corresponding Sui blob object certified in the final step. The steps involved in a store can be executed by the binary client, or a publisher that accepts and publishes blobs via HTTP. -Walrus currently allows the storage of up to XXXX bytes using the client, and YYY bytes using the -aggregator. You may store larger blobs by splitting them into smaller chunks. TODO sizes. +Walrus currently allows the storage of blob up to a maximum size that may be determined +through the `walrus info` command. You may store larger blobs by splitting them into smaller +chunks. ## Read @@ -27,12 +28,12 @@ Walrus can then be used to **read a blob** by providing its blob ID. A read is e performing the following steps: - The system object on Sui is read to determine the Walrus storage node committee. -- A number of storage nodes are queried for the slivers they store. +- A number of storage nodes are queried for blob metadata and the slivers they store. - The blob is reconstructed and checked against the blob ID from the recovered slivers. The steps involved in the read operation are performed by the binary client, or the aggregator service that exposes an HTTP interface to read blobs. Reads are extremely resilient and will -succeed in recovering the blob stored even if up to one-third of storage nodes are +succeed in recovering the blob stored even if up to two-thirds of storage nodes are unavailable. ## Certify Availability @@ -41,7 +42,8 @@ Walrus can be used to **certify the availability of a blob** using Sui. This may different ways: - A Sui SDK read can be - used to authenticate the certified blob event emitted when the blob ID was certified on Sui. + used to authenticate the certified blob event emitted when the blob ID was certified on Sui. The + client `walrus blob-status` command may be used to identify the event ID that needs to be checked. - A Sui SDK read may be used to authenticate the Sui blob object corresponding to the blob ID, and check it is certified. - A Sui smart contract can read the blob object on Sui (or a reference to it) to check @@ -49,7 +51,7 @@ different ways: The underlying protocol of the [Sui light client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client) -client returns digitally signed evidence for emitted events +returns digitally signed evidence for emitted events or objects, and can be used by off-line or non-interactive applications as a proof of availability for the blob ID for a certain number of epochs. diff --git a/docs/examples.md b/docs/examples.md index df635b4e..86c6f4f7 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1 +1,7 @@ # Examples + +As inspiration, we provide several simple examples in different programming languages to interact +with Walrus through the various interfaces. They are located at +. + +See the sub-chapters for further details. \ No newline at end of file diff --git a/docs/sui-struct.md b/docs/sui-struct.md index 0ca4d2ac..9d6475ac 100644 --- a/docs/sui-struct.md +++ b/docs/sui-struct.md @@ -5,10 +5,10 @@ This section is optional and enables advanced use cases. You can interact with Walrus purely through the client CLI, and JSON or HTTP APIs provided, without querying or executing transactions -on Sui directly. However, Walrus uses Sui to manage its meta-data and smart contract developers can +on Sui directly. However, Walrus uses Sui to manage its metadata and smart contract developers can read information about the Walrus system, as well as stored blobs, on Sui. -This section provides and overview of how you may use Walrus objects in your Sui smart contracts. +This section provides an overview of how you may use Walrus objects in your Sui smart contracts. ## Blob and Storage Objects @@ -75,7 +75,7 @@ certified a `BlobCertified` is emitted containing information about the blob ID after which the blob will be deleted. Before that epoch the blob is guaranteed to be available. ```move -/// Signals a blob with meta-data is registered. +/// Signals a blob with metadata is registered. public struct BlobRegistered has copy, drop { epoch: u64, blob_id: u256, @@ -92,7 +92,7 @@ public struct BlobCertified has copy, drop { } ``` -The `InvalidBlobID` event is emitted when storage nodes detect and incorrectly encoded blob. +The `InvalidBlobID` event is emitted when storage nodes detect an incorrectly encoded blob. Such a blob is guaranteed to be also detected as invalid when a read is attempted. ```move @@ -105,12 +105,14 @@ public struct InvalidBlobID has copy, drop { ## System information -The Walrus system object contains meta-data about the available and used storage, as well as the -price of storage per 100 Kib of storage in MIST. The committee +The Walrus system object contains metadata about the available and used storage, as well as the +price of storage per Kib of storage in MIST. The committee structure within the system object can be used to read the current epoch number, as well as information about the committee. ```move +const BYTES_PER_UNIT_SIZE : u64 = 1_024; + public struct System has key, store { id: UID, @@ -141,7 +143,7 @@ public struct Committee has store { } ``` -A few public functions of the committee allow contracts to read Walrus meta-data: +A few public functions of the committee allow contracts to read Walrus metadata: ```move /// Get epoch. Uses the committee to get the epoch. @@ -155,5 +157,4 @@ public fun used_capacity_size(self: &System) : u64; // The number of shards public fun n_shards(self: &System) : u16; - ``` From ec68cf15f37388997e49c4498aa0147e83df1b42 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Wed, 12 Jun 2024 12:29:16 +0100 Subject: [PATCH 10/15] Lint --- docs/examples.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/examples.md b/docs/examples.md index 86c6f4f7..0676b9b3 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -4,4 +4,4 @@ As inspiration, we provide several simple examples in different programming lang with Walrus through the various interfaces. They are located at . -See the sub-chapters for further details. \ No newline at end of file +See the sub-chapters for further details. From 6998b666ca3d6b5d38135c6d16684157b22ebd26 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Wed, 12 Jun 2024 12:42:08 +0100 Subject: [PATCH 11/15] two-thi --- docs/dev-operations.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/dev-operations.md b/docs/dev-operations.md index b32a25f8..61f4c040 100644 --- a/docs/dev-operations.md +++ b/docs/dev-operations.md @@ -33,8 +33,9 @@ performing the following steps: The steps involved in the read operation are performed by the binary client, or the aggregator service that exposes an HTTP interface to read blobs. Reads are extremely resilient and will -succeed in recovering the blob stored even if up to two-thirds of storage nodes are -unavailable. +succeed in recovering the blob stored even if up to one-third of storage nodes are +unavailable in all cases. Eventually, after synchronization is complete, event if two-thirds +of storage nodes are down reads will succeed. ## Certify Availability From 50042d5a8dabd0c51fecfcb061d848428c3d1eac Mon Sep 17 00:00:00 2001 From: George Danezis Date: Wed, 12 Jun 2024 12:56:26 +0100 Subject: [PATCH 12/15] Rebase --- docs/SUMMARY.md | 12 ++++++------ docs/configuration.md | 1 - docs/{ => dev-guide}/components.md | 0 docs/{ => dev-guide}/dev-guide.md | 0 docs/{ => dev-guide}/dev-operations.md | 0 docs/{ => dev-guide}/sui-struct.md | 0 docs/{ => examples}/examples.md | 0 docs/{ => examples}/python.md | 0 docs/exampoles/examples.md | 1 + 9 files changed, 7 insertions(+), 7 deletions(-) delete mode 100644 docs/configuration.md rename docs/{ => dev-guide}/components.md (100%) rename docs/{ => dev-guide}/dev-guide.md (100%) rename docs/{ => dev-guide}/dev-operations.md (100%) rename docs/{ => dev-guide}/sui-struct.md (100%) rename docs/{ => examples}/examples.md (100%) rename docs/{ => examples}/python.md (100%) create mode 100644 docs/exampoles/examples.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index df039056..73714a40 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -19,10 +19,10 @@ # Usage -- [Developer Guide](dev-guide.md) - - [Components](components.md) - - [Operations](dev-operations.md) - - [Sui structures](sui-struct.md) +- [Developer Guide](./dev-guide/dev-guide.md) + - [Components](./dev-guide/components.md) + - [Operations](./dev-guide/dev-operations.md) + - [Sui structures](./dev-guide/sui-struct.md) - [Setup](./usage/setup.md) - [Prerequisites](./usage/prerequisites.md) - [Installation](./usage/installation.md) @@ -31,8 +31,8 @@ - [Using the client CLI](./usage/client-cli.md) - [Using the client JSON API](./usage/json-api.md) - [Using the client HTTP API](./usage/web-api.md) -- [Examples](examples.md) - - [Python](python.md) +- [Examples](./examples/examples.md) + - [Python](./examples/python.md) # Walrus sites diff --git a/docs/configuration.md b/docs/configuration.md deleted file mode 100644 index a025a48b..00000000 --- a/docs/configuration.md +++ /dev/null @@ -1 +0,0 @@ -# Configuration diff --git a/docs/components.md b/docs/dev-guide/components.md similarity index 100% rename from docs/components.md rename to docs/dev-guide/components.md diff --git a/docs/dev-guide.md b/docs/dev-guide/dev-guide.md similarity index 100% rename from docs/dev-guide.md rename to docs/dev-guide/dev-guide.md diff --git a/docs/dev-operations.md b/docs/dev-guide/dev-operations.md similarity index 100% rename from docs/dev-operations.md rename to docs/dev-guide/dev-operations.md diff --git a/docs/sui-struct.md b/docs/dev-guide/sui-struct.md similarity index 100% rename from docs/sui-struct.md rename to docs/dev-guide/sui-struct.md diff --git a/docs/examples.md b/docs/examples/examples.md similarity index 100% rename from docs/examples.md rename to docs/examples/examples.md diff --git a/docs/python.md b/docs/examples/python.md similarity index 100% rename from docs/python.md rename to docs/examples/python.md diff --git a/docs/exampoles/examples.md b/docs/exampoles/examples.md new file mode 100644 index 00000000..df635b4e --- /dev/null +++ b/docs/exampoles/examples.md @@ -0,0 +1 @@ +# Examples From 4eac9aa3ffef85995bcbffe2ec4f5fc03f00491e Mon Sep 17 00:00:00 2001 From: George Danezis Date: Wed, 12 Jun 2024 13:00:44 +0100 Subject: [PATCH 13/15] Fix links --- docs/dev-guide/components.md | 6 +++--- docs/dev-guide/dev-guide.md | 2 +- docs/examples/python.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/dev-guide/components.md b/docs/dev-guide/components.md index ba58911e..d96fb4d7 100644 --- a/docs/dev-guide/components.md +++ b/docs/dev-guide/components.md @@ -13,7 +13,7 @@ Walrus defines a number of objects and smart contracts on Sui: - Changes to these objects emit *Walrus-related events*. The Walrus system object ID can be found in the Walrus `client_config.yaml` file -(see [Configuration](configuration.md)). You may use +(see [Configuration](../usage/configuration.md)). You may use any Sui explorer to look at its content, as well as explore the content of blob objects. There is more information about these in the [quick reference to the Walrus Sui structures](sui-struct.md). @@ -21,8 +21,8 @@ There is more information about these in the Walrus is also composed of a number of Walrus-specific services and binaries: - A client (binary) can be executed locally and provides a - [Command Line Interface (CLI)](client-cli.html), a [JSON API](json-api.md) - and an [HTTP API](web-api.md) to perform Walrus operations. + [Command Line Interface (CLI)](../usage/client-cli.md), a [JSON API](../usage/json-api.md) + and an [HTTP API](../usage/web-api.md) to perform Walrus operations. - Aggregators services allow reading blobs via HTTP requests. - Publishers services are used store blobs to Walrus. - A set of storage nodes store encoded stored blobs. These nodes form the decentralized diff --git a/docs/dev-guide/dev-guide.md b/docs/dev-guide/dev-guide.md index 311ba400..48020b3d 100644 --- a/docs/dev-guide/dev-guide.md +++ b/docs/dev-guide/dev-guide.md @@ -1,7 +1,7 @@ # Developer Guide This guide introduces all the concepts needed to build applications that use Walrus as a storage -or availability layer. The [overview](./overview.md) provides more background and explains in +or availability layer. The [overview](../overview/overview.md) provides more background and explains in more detail how Walrus operates internally. This developer guide describes: diff --git a/docs/examples/python.md b/docs/examples/python.md index b4d414c7..4d9fe8aa 100644 --- a/docs/examples/python.md +++ b/docs/examples/python.md @@ -9,7 +9,7 @@ contains a number of examples: - How to [use the JSON API](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_jsonapi.py) to store, read, and check the availability of a blob. Checking the certification of a blob illustrates reading the Blob Sui object that certifies - (see the [Walrus Sui reference](sui-struct.md)). + (see the [Walrus Sui reference](../dev-guide/sui-struct.md)). - How to [read information from the Walrus system object](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_sui_system.py). - How to [track Walrus related Events](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/track_walrus_events.py). From 59827d7623c23c213988ce836a91f7b51bfa8746 Mon Sep 17 00:00:00 2001 From: George Danezis Date: Wed, 12 Jun 2024 13:03:22 +0100 Subject: [PATCH 14/15] Lint --- docs/dev-guide/dev-guide.md | 4 ++-- docs/exampoles/examples.md | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) delete mode 100644 docs/exampoles/examples.md diff --git a/docs/dev-guide/dev-guide.md b/docs/dev-guide/dev-guide.md index 48020b3d..36faba94 100644 --- a/docs/dev-guide/dev-guide.md +++ b/docs/dev-guide/dev-guide.md @@ -1,8 +1,8 @@ # Developer Guide This guide introduces all the concepts needed to build applications that use Walrus as a storage -or availability layer. The [overview](../overview/overview.md) provides more background and explains in -more detail how Walrus operates internally. +or availability layer. The [overview](../overview/overview.md) provides more background and explains +in more detail how Walrus operates internally. This developer guide describes: diff --git a/docs/exampoles/examples.md b/docs/exampoles/examples.md deleted file mode 100644 index df635b4e..00000000 --- a/docs/exampoles/examples.md +++ /dev/null @@ -1 +0,0 @@ -# Examples From c26e65c32e8ef0a28cc5ab99c95f6e1e0ecbce9e Mon Sep 17 00:00:00 2001 From: George Danezis Date: Wed, 12 Jun 2024 13:10:01 +0100 Subject: [PATCH 15/15] Last typo :) --- docs/dev-guide/dev-operations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev-guide/dev-operations.md b/docs/dev-guide/dev-operations.md index 61f4c040..f5dcf7de 100644 --- a/docs/dev-guide/dev-operations.md +++ b/docs/dev-guide/dev-operations.md @@ -34,7 +34,7 @@ performing the following steps: The steps involved in the read operation are performed by the binary client, or the aggregator service that exposes an HTTP interface to read blobs. Reads are extremely resilient and will succeed in recovering the blob stored even if up to one-third of storage nodes are -unavailable in all cases. Eventually, after synchronization is complete, event if two-thirds +unavailable in all cases. Eventually, after synchronization is complete, even if two-thirds of storage nodes are down reads will succeed. ## Certify Availability