Skip to content

Latest commit

 

History

History
237 lines (167 loc) · 10.8 KB

adr-007-minimal-changes-to-tendermint.md

File metadata and controls

237 lines (167 loc) · 10.8 KB
Raw

ADR 007: From Ukraine, with Love

Changelog

  • 2021-08-20: Initial Description
  • 2022-05-03: Update pointing to ADR 008

Context

Currently, our fork of tendermint includes changes to how to erasure block data, minor changes to the header to commit to that data, additions to serve data availability sampling, along with some miscellaneous modification to adhere to the spec. Instead of incorporating all of these changes into our fork of tendermint, we will only make the strictly necessary changes and the other services and their code to the new celestia-node repo. Notably, we will also refactor some of the remaining necessary changes to be more isolated from the rest of the tendermint codebase. Both of these strategies should significantly streamline pulling updates from upstream, and allow us to iterate faster since most changes will be isolated to celestia-node.

Update: many of the changes described below have since been minimized or removed. Please see ADR 008 for a summarized list of changes. Notably, we removed intermediate state roots, adopted two new methods from ABCI++ instead of PreprocessTxs, and are still signing over the PartSetHeader.

Decision

Treat tendermint more as a "black box".

Detailed Design

Overview

We keep the bare-minimum changes to tendermint in our fork, celestia-core. Where necessary and possible we augment the tendermint node in a separate process, via celestia-node, which communicates with the tendermint node via RPC. All data availability sampling logic, including all Celestia-specific networking logic not already provided by tendermint, is moved into celestia node:

core node relation

The detailed design of celestia-node will be defined in the repository itself.

Necessary changes to tendermint

Changing the repo import names to celestiaorg

Changes to the README.md other basic things

Adding the extra types of block data

  • Update core data types (celestiaorg#17)
    • Create the Message/Messages types
    • Proto and the tendermint version
    • Create the IntermediateStateRoots type
    • Proto and the tendermint version
  • Data availability for evidence (celestiaorg#19)
    • Add both types to types.Data
    • Modify proto
    • Add EvidenceData to types.Data

Add the HeaderHash to the Commit

Adding the consts package in types

Remove iavl as a dependency

Using the DataAvailabilityHeader to calculate the DataHash

The DataAvailabilityHeader struct will be used by celestia-core as well as by the celestia-node. It might make sense to (eventually) move the struct together with all the DA-related code into a separate repository and go-module. @Wondertan explored this as part of #427. This way all client implementations can depend on that module without running into circular dependencies. Hence, we only describe how to hash the block data here:

  • Update core types (celestiaorg#17)
    • Replace the Data.Hash() with DAH.Hash()
    • Use DAH to fill DataHash when filling the header
    • Fill the DAH when making a block to generate the data hash

Add availableDataOriginalSharesUsed to the header

Reap some number of transactions probably using the app or some other mech

  • Enforce a minimum square size (celestiaorg#282)
  • Use squares with a width that is a power of two(celestiaorg#331)
  • Adopt reamping from the mempool to max square size (celestiaorg#77)
  • Proposal: Decide on a mech to pick square size and communicate that to the app (celestiaorg#454)
  • Also see ABCI++ for a less hacky solution

Filling the DAH using share merging and splitting

  • Compute Shares (not merged) (celestiaorg#60)
    • part II (not merged) (celestiaorg#63)
    • while this was not merged, we will need some function to compute the shares that make up the block data
  • Share Splitting (celestiaorg#246)
    • Serialize each constituent of block data
    • Split into shares
      • Txs (contiguous)
      • Messages (not contiguous)
      • Evidence (contiguous)
      • IntermediateStateRoots (contiguous)
  • Combine shares into original square
  • ExtendBlockData
  • Generate nmt root of each row and col
  • Use those roots to generate the DataHash
  • Share Merging (celestiaorg#261)
    • Sort by namespace
    • Parse each reserved type
    • Parse remaining messages

Add the wrapper around nmt to erasure namespaces

Add PreprocessTxs to ABCI

  • Add PreprocessTxs method to ABCI (celestiaorg#110)
  • Add method to ABCI interface
  • Create sync and async versions
  • Add sync version the the CreateProposalBlock method of BlockExecutor

Fill the DAH while making the block

Only produce blocks on some interval

Stop signing over the PartSetHeader

  • Replace canonical blockID with just a hash in the CononicalVote
  • Replace the LastBlockID in the header with just a hash

Optionally remove some unused code

Nice to Haves

We should also take this opportunity to refactor as many additions to tendermint into their own package as possible. This will hopefully make updating to future versions of tendermint easier. For example, when we fill the data availability header, instead of using a method on Block, it could be handled by a function that takes types.Data as input and returns the DAH, the number of shares used in the square, along with the obligatory error.

func FillDataAvailabilityHeader(data types.Data) (types.DataAvailabilityHeader, numOrigDataShares, error)

We could perform a similar treatment to the splitIntoShares methods and their helper method ComputeShares. Instead of performing the share splitting logic in those methods, we could keep it in a different package and instead call the equivalent function to compute the shares.

Beyond refactoring and some minor additions, we will also have to remove and revert quite a few changes to get to the minimum desired changes specified above.

Changes that will need to be reverted

IPLD Plugin

Adding DAH to Stuff

Embedding DAS

BlockID and PartSetHeader

  • Decouple ParSetHeader from BlockID (celestiaorg#441)
  • Stop Signing over the PartSetHeader (celestiaorg#457)
  • We still don’t want to sign over the PartSetHeader, but we will not be able to use the same mechanism used in the linked PR, as that way requires decoupling of the PSH from the BlockID
  • Remove PSH from some consensus messages (celestiaorg#479)

Note: This ADR overrides ADR 005 Decouple BlockID and the PartSetHeader. The PartSetHeader and the BlockID will mostly remain the same. This will make pulling changes from upstream much easier

Status

Accepted

Consequences

Positive

  • Pulling changes from upstream is streamlined
  • Separation of functionality will help us iterate faster
  • Creates a great opportunity for reconsidering past design choices without fully starting from scratch
  • Prepare for future designs
  • Don’t have to have two p2p stacks in a single repo

Negative

  • Perform some computation multiple times
  • Running multiple nodes instead of a single node is less convenient for node operators (but only in the case the full celestia-node wants to participate in the consensus protocol)

References

Tracking Issue #491