Skip to content

Commit

Permalink
MockDa support for demo-rollup (Sovereign-Labs#1193)
Browse files Browse the repository at this point in the history 
* Add mock config

* Update mock-da readme

* Test mock-da in the CI

* Fix readme

* Rneame config

* Update readme

* Rename job

* Update readme

* Update table of contents

* cleanup

* Fox ci

* Code review feedback

* Fix readme

* Add celestia info

* Update examples/demo-rollup/README.md

Co-authored-by: Cem Özer <[email protected]>

* Update readme

---------

Co-authored-by: Cem Özer <[email protected]>
  • Loading branch information
@bkolad @cemozerr
bkolad and cemozerr authored Dec 1, 2023
1 parent 88b77d6 commit 48e6c83
Show file tree
Hide file tree
Showing 21 changed files with 522 additions and 146 deletions.
36 changes: 35 additions & 1 deletion .github/workflows/rust.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -354,14 +354,47 @@ jobs:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- run: npm install -g doctoc
- name: Update table of contents
- name: Update table of contents for mock-da
run: doctoc README.md --github --notitle
working-directory: ./examples/demo-rollup
- name: Update table of contents for celestia-da
run: doctoc README_CELESTIA.md --github --notitle
working-directory: ./examples/demo-rollup
- name: Check table of contents
# Exit status 0 means no changes were made, so the table of contents is
# up to date.
run: git diff --exit-code
check-demo-rollup-bash-commands:
runs-on: buildjet-4vcpu-ubuntu-2204
needs: nextest
timeout-minutes: 60
steps:
- uses: actions/checkout@v3
- uses: rui314/setup-mold@v1
- name: Install Protoc
uses: arduino/setup-protoc@v2
with:
version: "23.2"
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install cargo-risczero
uses: taiki-e/install-action@v2
with:
tool: [email protected]
- name: Install risc0-zkvm toolchain # Use the risc0 cargo extension to install the risc0 std library for the current toolchain
run: cargo risczero install
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Install Rust
run: rustup show
- uses: Swatinem/rust-cache@v2
with:
cache-provider: "buildjet"
save-if: ${{ github.ref == 'refs/heads/nightly' }}
- name: Compile README.md to Bash
run: cargo run --bin bashtestmd -- --input examples/demo-rollup/README_CELESTIA.md --output demo-rollup-readme.sh --tag test-ci
- run: cat demo-rollup-readme.sh
- run: chmod +x demo-rollup-readme.sh && ./demo-rollup-readme.sh
check-demo-rollup-bash-commands-mock-da:
runs-on: buildjet-4vcpu-ubuntu-2204
needs: nextest
timeout-minutes: 60
Expand Down Expand Up @@ -391,6 +424,7 @@ jobs:
run: cargo run --bin bashtestmd -- --input examples/demo-rollup/README.md --output demo-rollup-readme.sh --tag test-ci
- run: cat demo-rollup-readme.sh
- run: chmod +x demo-rollup-readme.sh && ./demo-rollup-readme.sh

validate-packages-to-publish-yml:
runs-on: ubuntu-latest
steps:
Expand Down
28 changes: 19 additions & 9 deletions examples/demo-rollup/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ stop: check-docker
@$(compose_down)

# clean up rollup data and stop the celestia network
clean: stop clean-rollup-db
clean: stop clean-celestia-rollup-db

# listen to celestia network logs
compose-logs: check-docker
Expand All @@ -64,13 +64,13 @@ wait-compose-ready:
# update the rollup configuration with local network params
update-config: wait-compose-ready
ifeq ($(shell uname -s),Darwin)
@sed -i '' 's/^\(celestia_rpc_auth_token = \)"[^"]*"/\1"$(get_auth)"/' rollup_config.toml
@sed -i '' 's#^\(celestia_rpc_address = \)"[^"]*"#\1"http://127.0.0.1:$(RPC_PORT)"#' rollup_config.toml
@sed -i '' 's#^\(start_height = \)[0-9]*#\1$(START_HEIGHT)#' rollup_config.toml
@sed -i '' 's/^\(celestia_rpc_auth_token = \)"[^"]*"/\1"$(get_auth)"/' celestia_rollup_config.toml
@sed -i '' 's#^\(celestia_rpc_address = \)"[^"]*"#\1"http://127.0.0.1:$(RPC_PORT)"#' celestia_rollup_config.toml
@sed -i '' 's#^\(start_height = \)[0-9]*#\1$(START_HEIGHT)#' celestia_rollup_config.toml
else
@sed -i 's/^\(celestia_rpc_auth_token = \)"[^"]*"/\1"$(get_auth)"/' rollup_config.toml
@sed -i 's#^\(celestia_rpc_address = \)"[^"]*"#\1"http://127.0.0.1:$(RPC_PORT)"#' rollup_config.toml
@sed -i 's#^\(start_height = \)[0-9]*#\1$(START_HEIGHT)#' rollup_config.toml
@sed -i 's/^\(celestia_rpc_auth_token = \)"[^"]*"/\1"$(get_auth)"/' celestia_rollup_config.toml
@sed -i 's#^\(celestia_rpc_address = \)"[^"]*"#\1"http://127.0.0.1:$(RPC_PORT)"#' celestia_rollup_config.toml
@sed -i 's#^\(start_height = \)[0-9]*#\1$(START_HEIGHT)#' celestia_rollup_config.toml
endif

build-sov-cli:
Expand All @@ -91,11 +91,21 @@ test-create-token: set-rpc-url test-generate-create-token-tx import-keys
remove-insecure-keys: build-sov-cli
$(SOV_CLI_REL_PATH) keys remove by-address sov1l6n2cku82yfqld30lanm2nfw43n2auc8clw7r5u5m6s7p8jrm4zqrr8r94

clean-rollup-db:
$(eval path := ./$(shell awk -F'=' '/^path/ {print $$2}' rollup_config.toml | tr -d '[:space:]"\n'))
clean-celestia-rollup-db:
$(eval path := ./$(shell awk -F'=' '/^path/ {print $$2}' celestia_rollup_config.toml | tr -d '[:space:]"\n'))
@if [ -z "${path}" ] || [ "${path}" = "./" ]; then \
echo "Path is empty or too short, not safe to remove"; \
exit 1; \
fi
@echo removing rollup database "${path}"
rm -rf "${path}"


clean-mock-rollup-db:
$(eval path := ./$(shell awk -F'=' '/^path/ {print $$2}' mock_rollup_config.toml | tr -d '[:space:]"\n'))
@if [ -z "${path}" ] || [ "${path}" = "./" ]; then \
echo "Path is empty or too short, not safe to remove"; \
exit 1; \
fi
@echo removing rollup database "${path}"
rm -rf "${path}"
154 changes: 22 additions & 132 deletions examples/demo-rollup/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# Demo Rollup ![Time - ~5 mins](https://img.shields.io/badge/Time-~5_mins-informational)

This is a demo full node running a simple Sovereign SDK rollup on [Celestia](https://celestia.org/).

<p align="center">
<img width="50%" src="../../assets/discord-banner.png">
<br>
Expand All @@ -23,15 +21,11 @@ This is a demo full node running a simple Sovereign SDK rollup on [Celestia](htt
- [Start the Rollup Full Node](#start-the-rollup-full-node)
- [Sanity Check: Creating a Token](#sanity-check-creating-a-token)
- [How to Submit Transactions](#how-to-submit-transactions)
- [How to Submit Transactions](#how-to-submit-transactions-1)
- [1. Build `sov-cli`](#1-build-sov-cli)
- [2. Generate the Transaction](#2-generate-the-transaction)
- [3. Submit the Transaction(s)](#3-submit-the-transactions)
- [4. Verify the Token Supply](#4-verify-the-token-supply)
- [Makefile](#makefile)
- [Remote setup](#remote-setup)
- [How to Customize This Example](#how-to-customize-this-example)
- [1. Initialize the DA Service](#1-initialize-the-da-service)
- [2. Run the Main Loop](#2-run-the-main-loop)
- [Submit the Transaction(s)](#submit-the-transactions)
- [Verify the Token Supply](#verify-the-token-supply)
- [Disclaimer](#disclaimer)
- [Interacting with your Node via RPC](#interacting-with-your-node-via-rpc)
- [Key Concepts](#key-concepts)
Expand All @@ -41,12 +35,12 @@ This is a demo full node running a simple Sovereign SDK rollup on [Celestia](htt
- [`ledger_getBatches`](#ledger_getbatches)
- [`ledger_getTransactions`](#ledger_gettransactions)
- [`ledger_getEvents`](#ledger_getevents)
- [Testing with specific DA layers](#testing-with-specific-da-layers)
- [License](#license)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## What is This?

This demo shows how to integrate a State Transition Function (STF) with a Data Availability (DA) layer and a zkVM to create a full
zk-rollup. The code in this repository corresponds to running a full-node of the rollup, which executes
every transaction.
Expand All @@ -60,87 +54,45 @@ understand how to build your own state transition function, check out at the doc
If you are looking for a simple rollup with minimal dependencies as a starting point, please have a look here:
[sov-rollup-starter](https://github.com/Sovereign-Labs/sov-rollup-starter/)

### Run a local DA layer instance

1. Install Docker: <https://www.docker.com>.

2. Follow [this guide](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic)
to authorize yourself in github's container registry. (we use original celestia images which they publish in ghcr)

```shell
# this has to be ran only once, unless your token expires
$ echo $MY_PERSONAL_GITHUB_TOKEN | docker login ghcr.io -u $MY_GITHUB_USERNAME --password-stdin
```
### Run a local DA layer instance
This setup works with an in-memory DA that is easy to set up for testing purposes.

3. Switch to the `examples/demo-rollup` directory (which is where this `README.md` is located!), and compile the application:
### Start the Rollup Full Node
1. Switch to the `examples/demo-rollup` and compile the application:

```shell,test-ci
$ cd examples/demo-rollup/
$ cargo build --bins
```

4. Spin up a local Celestia instance as your DA layer. We've built a small Makefile to simplify that process:

```sh,test-ci,bashtestmd:long-running,bashtestmd:wait-until=genesis.json
$ make clean
# Make sure to run `make stop` or `make clean` when you're done with this demo!
$ make start
```

If interested, you can check out what the Makefile does [here](#Makefile).
The above command will also modify some configuration files:

2. Clean up the existing database.
Makefile to simplify that process:
```sh,test-ci
$ git status
..
..
modified: rollup_config.toml
$ make clean-mock-rollup-db
```

### Start the Rollup Full Node

Now run the demo-rollup full node, as shown below. You will see it consuming blocks from the Celestia node running inside Docker:

3. Now run the demo-rollup full node, as shown below.
```sh,test-ci,bashtestmd:long-running
# Make sure you're still in the examples/demo-rollup directory.
$ cargo run
2023-06-07T10:03:25.473920Z INFO sov_celestia_adapter::da_service: Fetching header at height=1...
2023-06-07T10:03:25.496853Z INFO sov_demo_rollup: Received 0 blobs
2023-06-07T10:03:25.497700Z INFO sov_demo_rollup: Requesting data for height 2 and prev_state_root 0xa96745d3184e54d098982daf44923d84c358800bd22c1864734ccb978027a670
2023-06-07T10:03:25.497719Z INFO sov_celestia_adapter::da_service: Fetching header at height=2...
2023-06-07T10:03:25.505412Z INFO sov_demo_rollup: Received 0 blobs
2023-06-07T10:03:25.505992Z INFO sov_demo_rollup: Requesting data for height 3 and prev_state_root 0xa96745d3184e54d098982daf44923d84c358800bd22c1864734ccb978027a670
2023-06-07T10:03:25.506003Z INFO sov_celestia_adapter::da_service: Fetching header at height=3...
2023-06-07T10:03:25.511237Z INFO sov_demo_rollup: Received 0 blobs
2023-06-07T10:03:25.511815Z INFO sov_demo_rollup: Requesting data for height 4 and prev_state_root 0xa96745d3184e54d098982daf44923d84c358800bd22c1864734ccb978027a670
```

Leave it running while you proceed with the rest of the demo.

### Sanity Check: Creating a Token

### Sanity Check: Creating a Token
After switching to a new terminal tab, let's submit our first transaction by creating a token:

```sh,test-ci
$ make test-create-token
```

...wait a few seconds and you will see the transaction receipt in the output of the demo-rollup full node:

```sh
2023-07-12T15:04:52.291073Z INFO sov_celestia_adapter::da_service: Fetching header at height=31...
2023-07-12T15:05:02.304393Z INFO sov_demo_rollup: Received 1 blobs at height 31
2023-07-12T15:05:02.305257Z INFO sov_demo_rollup: blob #0 at height 31 with blob_hash 0x4876c2258b57104356efa4630d3d9f901ccfda5dde426ba8aef81d4a3e357c79 has been applied with #1 transactions, sequencer outcome Rewarded(0)
2023-07-12T15:05:02.305280Z INFO sov_demo_rollup: tx #0 hash: 0x1e1892f77cf42c0abd2ca2acdd87eabb9aa65ec7497efea4ff9f5f33575f881a result Successful
2023-07-12T15:05:02.310714Z INFO sov_demo_rollup: Requesting data for height 32 and prev_state_root 0xae87adb5291d3e645c09ff74dfe3580a25ef0b893b67f09eb58ae70c1bf135c2
```

### How to Submit Transactions
The `make test-create-token` command above was useful to test if everything is running correctly. Now let's get a better understanding of how to create and submit a transaction

### How to Submit Transactions
The `make test-create-token` command above was useful to test if everything is running correctly. Now let's get a better understanding of how to create and submit a transaction.

#### 1. Build `sov-cli`

You'll need the `sov-cli` binary in order to create transactions. Build it with these commands:

```bash,test-ci,bashtestmd:compare-output
Expand Down Expand Up @@ -269,6 +221,7 @@ Options:

Let's go ahead and import the transaction into the wallet


```bash,test-ci,bashtestmd:compare-output
$ cargo run --bin sov-cli -- transactions import from-file bank --path ../test-data/requests/transfer.json
Adding the following transaction to batch:
Expand All @@ -285,87 +238,20 @@ Adding the following transaction to batch:
}
```

This output indicates that the wallet has saved the transaction details for later signing.

#### 3. Submit the Transaction(s)

#### Submit the Transaction(s)
You now have a batch with a single transaction in your wallet. If you want to submit any more transactions as part of this
batch, you can import them now. Finally, let's submit your transaction to the rollup.

```bash,test-ci
$ cargo run --bin sov-cli rpc submit-batch by-address sov1l6n2cku82yfqld30lanm2nfw43n2auc8clw7r5u5m6s7p8jrm4zqrr8r94
```

This command will use your default private key.

#### 4. Verify the Token Supply

#### Verify the Token Supply
```bash,test-ci,bashtestmd:compare-output
$ curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"bank_supplyOf","params":["sov1zdwj8thgev2u3yyrrlekmvtsz4av4tp3m7dm5mx5peejnesga27svq9m72"],"id":1}' http://127.0.0.1:12345
{"jsonrpc":"2.0","result":{"amount":1000},"id":1}
```

### Makefile

`demo-rollup/Makefile` automates a number of things for convenience:

- Starts docker compose with a Celestia network for a local setup
- `make start`:
- Performs a number of checks to ensure services are not already running
- Starts the docker compose setup
- Exposes the RPC port `26658`
- Waits until the container is started
- Sets up the config
- `examples/demo-rollup/rollup_config.toml` is modified -
- `start_height` is set to `3`, which is the block in which sequencers are funded with credits
- `celestia_rpc_auth_token` is set to the auth token exposed by sequencer (in <repo_root>/docker/credentials directory)
- `celestia_rpc_address` is set to point to `127.0.0.1` and the `RPC_PORT`
- `make stop`:
- Shuts down the Celestia docker compose setup, if running.
- Deletes all contents of the demo-rollup database.
- `make clean`:
- Stops any running containers with the name `sov-celestia-local` and also removes them
- Removes `demo-data` (or the configured path of the rollup database from rollup_config.toml)

### Remote setup

> 🚧 This feature is under development! 🚧
The above setup runs Celestia node locally to avoid any external network dependencies and to speed up development. Soon, the Sovereign SDK will also support connecting to the Celestia testnet using a Celestia light node running on your machine.

## How to Customize This Example

Any time you change out the state transition function, zkVM, or DA layer of your rollup, you'll
need to tweak this full-node code. At the very least, you'll need to modify the dependencies. In most cases,
your full node will also need to be aware of the STF's initialization logic, and how it exposes RPC.

Given that constraint, we won't try to give you specific instructions for supporting every imaginable
combination of DA layers and State Transition Functions. Instead, we'll explain at a high level what
tasks a full-node needs to accomplish.

### 1. Initialize the DA Service

The first _mandatory_ step is to initialize a DA service, which allows the full node implementation to
communicate with the DA layer's RPC endpoints.

If you're using Celestia as your DA layer, you can follow the instructions at the end
of this document to set up a local full node, or connect to
a remote node. Whichever option you pick, simply place the URL and authentication token
in the `rollup_config.toml` file and it will be
automatically picked up by the node implementation. For this tutorial, the Makefile below (which also helps start a local Celestia instance) handles this step for you.

### 2. Run the Main Loop

The full node implements a simple loop for processing blocks. The workflow is:

1. Fetch slot data from the DA service
2. Run `stf.begin_slot()`
3. Iterate over the blobs, running `apply_batch`
4. Run `stf.end_slot()`

In this demo, we also keep a `ledger_db`, which stores information
related to the chain's history - batches, transactions, receipts, etc.

## Disclaimer

> ⚠️ Warning! ⚠️
Expand Down Expand Up @@ -485,6 +371,10 @@ $ curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method"

This response indicates that event `1` has not been emitted yet.

## Testing with specific DA layers
Check [here](./README_CELESTIA.md) if you want to run with dockerized local Celestia instance.


## License

Licensed under the [Apache License, Version 2.0](../../LICENSE).
Expand Down
Loading

0 comments on commit 48e6c83

Please sign in to comment.