docs: add initial dev guide (#18)

Added a dev guide, with components, operations and Sui reference. As
well as an outline of the python examples.

Contributes to #5

docs/SUMMARY.md

@@ -19,6 +19,10 @@
 
 # Usage
 
+- [Developer Guide](./dev-guide/dev-guide.md)
+  - [Components](./dev-guide/components.md)
+  - [Operations](./dev-guide/dev-operations.md)
+  - [Sui structures](./dev-guide/sui-struct.md)
 - [Setup](./usage/setup.md)
   - [Prerequisites](./usage/prerequisites.md)
   - [Installation](./usage/installation.md)
@@ -27,7 +31,8 @@
   - [Using the client CLI](./usage/client-cli.md)
   - [Using the client JSON API](./usage/json-api.md)
   - [Using the client HTTP API](./usage/web-api.md)
-- [Examples]()
+- [Examples](./examples/examples.md)
+  - [Python](./examples/python.md)
 
 # Walrus sites
 

docs/dev-guide/components.md

@@ -0,0 +1,33 @@
+# Components
+
+From a developer perspective, some Walrus components are objects and smart contracts on
+Sui, and some components are Walrus-specific binaries and services. As a rule Sui is used to
+manage blob and storage node metadata, while Walrus-specific services are used to store and
+read blob contents, which can be very large.
+
+Walrus defines a number of objects and smart contracts on Sui:
+
+- A shared *system object*, records and manages the current committee of storage nodes.
+- *Storage resources*, represent empty storage space that may be used to store blobs.
+- *Blob resources*, represent blobs being registered and certified as stored.
+- Changes to these objects emit *Walrus-related events*.
+
+The Walrus system object ID can be found in the Walrus `client_config.yaml` file
+(see [Configuration](../usage/configuration.md)). You may use
+any Sui explorer to look at its content, as well as explore the content of blob objects.
+There is more information about these in the
+[quick reference to the Walrus Sui structures](sui-struct.md).
+
+Walrus is also composed of a number of Walrus-specific services and binaries:
+
+- A client (binary) can be executed locally and provides a
+  [Command Line Interface (CLI)](../usage/client-cli.md), a [JSON API](../usage/json-api.md)
+  and an [HTTP API](../usage/web-api.md) to perform Walrus operations.
+- Aggregators services allow reading blobs via HTTP requests.
+- Publishers services are used store blobs to Walrus.
+- A set of storage nodes store encoded stored blobs. These nodes form the decentralized
+  storage infrastructure of Walrus.
+
+Aggregators, publishers and other services use the client APIs to interact with Walrus. End-users
+of services using Walrus interact with the store via custom services, aggregators or publishers that
+expose HTTP APIs to avoid the need to run locally a binary client.

docs/dev-guide/dev-guide.md

@@ -0,0 +1,22 @@
+# Developer Guide
+
+This guide introduces all the concepts needed to build applications that use Walrus as a storage
+or availability layer. The [overview](../overview/overview.md) provides more background and explains
+in more detail how Walrus operates internally.
+
+This developer guide describes:
+
+- [Components](components.md) of Walrus of interest to developers that wish to use it for
+  storage or availability.
+- [Operations](dev-operations.md) supported through client binaries, APIs, or Sui operations.
+- [The Sui structures](sui-struct.md) Walrus uses to store metadata, and how they can be read
+  from Sui smart contracts, or through the Sui SDK.
+
+## Disclaimer about the Walrus developer preview
+
+**This release of Walrus \& Walrus Sites is a
+developer preview, to showcase the technology and solicit feedback from builders. All storage nodes
+and aggregators are operated by Mysten Labs, all transactions are executed on the Sui testnet,
+and use testnet SUI which has no value. The state of the store can be, and will be wiped, at any
+point and possibly with no warning. Do not rely on this developer preview for any production
+purposes, it comes with no availability or persistence guarantees.**

docs/dev-guide/dev-operations.md

@@ -0,0 +1,60 @@
+# Operations
+
+## Store
+
+Walrus may be used to **store a blob**, via the native client APIs or a publisher. Under the hood a
+number of operations happen both on Sui as well as on storage nodes:
+
+- The client or publisher encodes the blob and derives a *blob ID* that identifies the blob. This
+  is a `u256` often encoded as a URL safe base64 string.
+- A transaction is executed on Sui to purchase some storage from the system object, and then to
+  *register the blob ID* with this storage. Client APIs return the *Sui blob object ID*. The
+  transactions use SUI to purchase storage and pay for gas.
+- Encoded slivers of the blob are distributed to all storage nodes. They each sign a receipt.
+- Signed receipts are aggregated and submitted to the Sui blob object to *certify the blob*.
+  Certifying a blob emits a Sui event with the blob ID and the period of availability.
+
+A blob is considered available on Walrus once the corresponding Sui blob object has been
+certified in the final step. The steps involved in a store can be executed by the binary client,
+or a publisher that accepts and publishes blobs via HTTP.
+
+Walrus currently allows the storage of blob up to a maximum size that may be determined
+through the `walrus info` command. You may store larger blobs by splitting them into smaller
+chunks.
+
+## Read
+
+Walrus can then be used to **read a blob** by providing its blob ID. A read is executed by
+performing the following steps:
+
+- The system object on Sui is read to determine the Walrus storage node committee.
+- A number of storage nodes are queried for blob metadata and the slivers they store.
+- The blob is reconstructed and checked against the blob ID from the recovered slivers.
+
+The steps involved in the read operation are performed by the binary client, or the aggregator
+service that exposes an HTTP interface to read blobs. Reads are extremely resilient and will
+succeed in recovering the blob stored even if up to one-third of storage nodes are
+unavailable in all cases. Eventually, after synchronization is complete, even if two-thirds
+of storage nodes are down reads will succeed.
+
+## Certify Availability
+
+Walrus can be used to **certify the availability of a blob** using Sui. This may be done in 3
+different ways:
+
+- A Sui SDK read can be
+  used to authenticate the certified blob event emitted when the blob ID was certified on Sui. The
+  client `walrus blob-status` command may be used to identify the event ID that needs to be checked.
+- A Sui SDK read may be
+  used to authenticate the Sui blob object corresponding to the blob ID, and check it is certified.
+- A Sui smart contract can read the blob object on Sui (or a reference to it) to check
+  is is certified.
+
+The underlying protocol of the
+[Sui light client](https://github.com/MystenLabs/sui/tree/main/crates/sui-light-client)
+returns digitally signed evidence for emitted events
+or objects, and can be used by off-line or non-interactive applications as a proof of availability
+for the blob ID for a certain number of epochs.
+
+Once a blob is certified, Walrus will ensure that sufficient slivers will always be
+available on storage nodes to recover it within the specified epochs.

docs/dev-guide/sui-struct.md

@@ -0,0 +1,160 @@
+
+# Sui Structures
+
+This section is optional and enables advanced use cases.
+
+You can interact with Walrus purely
+through the client CLI, and JSON or HTTP APIs provided, without querying or executing transactions
+on Sui directly. However, Walrus uses Sui to manage its metadata and smart contract developers can
+read information about the Walrus system, as well as stored blobs, on Sui.
+
+This section provides an overview of how you may use Walrus objects in your Sui smart contracts.
+
+## Blob and Storage Objects
+
+Walrus blobs are represented as Sui objects of type `Blob`. A blob is first registered, indicating
+that the storage nodes should expect slivers from a Blob ID to be stored. Then a blob is certified
+indicating that a sufficient number of slivers have been stored to guarantee the blob's
+availability. When a blob is certified its `certified` field contains the epoch in which it was
+certified.
+
+A `Storage` object is always associated with a `Blob` object, reserving enough space for
+a long enough period for the blob's storage. A certified blob is available for the period the
+underlying storage resource guarantees storage.
+
+Concretely, `Blob` and `Storage` objects have the following fields, that can be read through the
+Sui SDKs:
+
+```move
+/// The blob structure represents a blob that has been registered to with some
+/// storage, and then may eventually be certified as being available in the
+/// system.
+public struct Blob has key, store {
+    id: UID,
+    stored_epoch: u64,
+    blob_id: u256,
+    size: u64,
+    erasure_code_type: u8,
+    certified: option::Option<u64>, // The epoch first certified,
+                                    // or None if not certified.
+    storage: Storage,
+}
+
+/// Reservation for storage for a given period, which is inclusive start,
+/// exclusive end.
+public struct Storage has key, store {
+    id: UID,
+    start_epoch: u64,
+    end_epoch: u64,
+    storage_size: u64,
+}
+```
+
+All fields of `Blob` and `Storage` objects can be read using the expected functions:
+
+```move
+// Blob functions
+public fun stored_epoch(b: &Blob) : u64;
+public fun blob_id(b: &Blob) : u256;
+public fun size(b: &Blob) : u64;
+public fun erasure_code_type(b: &Blob) : u8;
+public fun certified(b: &Blob) : &Option<u64>;
+public fun storage(b: &Blob) : &Storage;
+
+// Storage functions
+public fun start_epoch(self: &Storage) : u64;
+public fun end_epoch(self: &Storage) : u64;
+public fun storage_size(self: &Storage) : u64;
+```
+
+## Events
+
+When a blob is first registered a `BlobRegistered` event is emitted that informs storage nodes
+that they should expect slivers associated with its Blob ID. Eventually when the blob is
+certified a `BlobCertified` is emitted containing information about the blob ID and the epoch
+after which the blob will be deleted. Before that epoch the blob is guaranteed to be available.
+
+```move
+/// Signals a blob with metadata is registered.
+public struct BlobRegistered has copy, drop {
+    epoch: u64,
+    blob_id: u256,
+    size: u64,
+    erasure_code_type: u8,
+    end_epoch: u64,
+}
+
+/// Signals a blob is certified.
+public struct BlobCertified has copy, drop {
+    epoch: u64,
+    blob_id: u256,
+    end_epoch: u64,
+}
+```
+
+The `InvalidBlobID` event is emitted when storage nodes detect an incorrectly encoded blob.
+Such a blob is guaranteed to be also detected as invalid when a read is attempted.
+
+```move
+/// Signals that a BlobID is invalid.
+public struct InvalidBlobID has copy, drop {
+    epoch: u64, // The epoch in which the blob ID is first registered as invalid
+    blob_id: u256,
+}
+```
+
+## System information
+
+The Walrus system object contains metadata about the available and used storage, as well as the
+price of storage per Kib of storage in MIST. The committee
+structure within the system object can be used to read the current epoch number, as well as
+information about the committee.
+
+```move
+const BYTES_PER_UNIT_SIZE : u64 = 1_024;
+
+public struct System<phantom WAL> has key, store {
+
+    id: UID,
+
+    /// The current committee, with the current epoch.
+    /// The option is always Some, but need it for swap.
+    current_committee: Option<Committee>,
+
+    /// When we first enter the current epoch we SYNC,
+    /// and then we are DONE after a cert from a quorum.
+    epoch_status: u8,
+
+    // Some accounting
+    total_capacity_size : u64,
+    used_capacity_size : u64,
+
+    /// The price per unit size of storage.
+    price_per_unit_size: u64,
+
+    /// Tables about the future and the past.
+    past_committees: Table<u64, Committee>,
+    future_accounting: FutureAccountingRingBuffer<WAL>,
+}
+
+public struct Committee has store {
+    epoch: u64,
+    bls_committee : BlsCommittee,
+}
+```
+
+A few public functions of the committee allow contracts to read Walrus metadata:
+
+```move
+/// Get epoch. Uses the committee to get the epoch.
+public fun epoch<WAL>(self: &System<WAL>) : u64;
+
+/// Accessor for total capacity size.
+public fun total_capacity_size<WAL>(self: &System<WAL>) : u64;
+
+/// Accessor for used capacity size.
+public fun used_capacity_size<WAL>(self: &System<WAL>) : u64;
+
+// The number of shards
+public fun n_shards<WAL>(self: &System<WAL>) : u16;
+```

docs/examples/examples.md

@@ -0,0 +1,7 @@
+# Examples
+
+As inspiration, we provide several simple examples in different programming languages to interact
+with Walrus through the various interfaces. They are located at
+<https://github.com/MystenLabs/walrus-docs/tree/main/examples>.
+
+See the sub-chapters for further details.

docs/examples/python.md

@@ -0,0 +1,15 @@
+# Python
+
+The [python examples](https://github.com/MystenLabs/walrus-docs/tree/main/examples/python) folder
+contains a number of examples:
+
+<!-- markdownlint-disable -->
+- How to [use the HTTP API](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_webapi.py)
+  to store and read a blob.
+- How to [use the JSON API](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_jsonapi.py)
+  to store, read, and check the availability of a blob. Checking the certification of a blob
+  illustrates reading the Blob Sui object that certifies
+  (see the [Walrus Sui reference](../dev-guide/sui-struct.md)).
+- How to [read information from the Walrus system object](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/hello_walrus_sui_system.py).
+- How to [track Walrus related Events](https://github.com/MystenLabs/walrus-docs/blob/main/examples/python/track_walrus_events.py).
+<!-- markdownlint-enable -->