verkle: verkle package (#2923)

* verkle: initial commit * verkle: scaffold verkle trie implementation and types * verkle: fix installation issues * verkle: move rust verkle wasm to verkle package * verkle: remove package json and readme from rust verkle wasm dir * util: add some bigint LE utils for verkle * verkle: refactor constants and cryptographic methods to the verkle package * verkle: remove unnecessary constants & methods * verkle: verkle node classes and types * verkle: get method and rawNode utils * verkle: handle case where array item is not found * verkle: wip findPath method * verkle: wip * verkle: fromRawNode method * verkle: update packages * verkle: minor adjustments * verkle: wip * verkle: add test case for put and get * verkle: wip insertstem and related methods for verkle nodes * verkle: internal node test * verkle: default values and minor fixes to internalNode * verkle: make internal verkle node options optional * verkle: leafNode basic test * verkle: remove unused import * verkle: setDepth method for leafNode and commented out code for create method * verkle: wip * verkle: add Point interface * verkle: general cleanup and improvements * verkle: minor test adjustments * verkle: readme and testing related updates * verkle: capital V for verkle workflow * Update packages/verkle/README.md Co-authored-by: Scorbajio <[email protected]> * Update packages/verkle/README.md Co-authored-by: Scorbajio <[email protected]> * Update packages/verkle/README.md Co-authored-by: Scorbajio <[email protected]> * Update packages/verkle/README.md Co-authored-by: Scorbajio <[email protected]> * verkle: use Lock from util package * client: undo removal of link * util: update byte<>int conversion helpers * verkle: use Uint8Array instead of hexstrings for db * verkle: add imports to example * verkle: update db to Uint8Arrays * verkle: revert to using strings as keys for cache * verkle: update crypto import * verkle: adjust Key and Value encoding in create method * verkle: adjust value encoding in db del method * verkle: add missing key and value encoding opts to batch options * verkle: remove extra line --------- Co-authored-by: acolytec3 <[email protected]> Co-authored-by: Scorbajio <[email protected]>
ethereumjs · Oct 26, 2023 · 2353bc7 · 2353bc7
1 parent 876e560
commit 2353bc7
Show file tree

Hide file tree

Showing 44 changed files with 2,623 additions and 0 deletions.
diff --git a/.github/ISSUE_TEMPLATE/package--ethereumjs-verkle.md b/.github/ISSUE_TEMPLATE/package--ethereumjs-verkle.md
@@ -0,0 +1,7 @@
+---
+name: 'Package: @ethereumjs/verkle'
+about: Create issue for @ethereumjs/verkle package
+title: ''
+labels: 'package: verkle'
+assignees: ''
+---
diff --git a/.github/workflows/verkle-build.yml b/.github/workflows/verkle-build.yml
@@ -0,0 +1,42 @@
+name: Verkle
+on:
+  push:
+    branches: [master, develop]
+    tags: ['*']
+  pull_request:
+    types: [opened, reopened, synchronize]
+  workflow_dispatch:
+
+env:
+  cwd: ${{github.workspace}}/packages/verkle
+
+defaults:
+  run:
+    working-directory: packages/verkle
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  test-verkle:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        node-version: [18]
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: recursive
+
+      - name: Use Node.js ${{ matrix.node-version }}
+        uses: actions/setup-node@v3
+        with:
+          node-version: ${{ matrix.node-version }}
+          cache: 'npm'
+
+      - run: npm ci --omit=peer
+        working-directory: ${{github.workspace}}
+
+      - run: npm run lint
+      - run: npm run test:node # Only run node tests for now until vitest browser test issues are sorted out
diff --git a/README.md b/README.md
@@ -35,6 +35,7 @@ Below you can find a list of the packages included in this repository.
 | [@ethereumjs/trie][trie-package]                 | [![NPM Package][trie-npm-badge]][trie-npm-link]                 | [![Trie Issues][trie-issues-badge]][trie-issues-link]                         | [![Actions Status][trie-actions-badge]][trie-actions-link]                 | [![Code Coverage][trie-coverage-badge]][trie-coverage-link]                 |
 | [@ethereumjs/tx][tx-package]                     | [![NPM Package][tx-npm-badge]][tx-npm-link]                     | [![Tx Issues][tx-issues-badge]][tx-issues-link]                               | [![Actions Status][tx-actions-badge]][tx-actions-link]                     | [![Code Coverage][tx-coverage-badge]][tx-coverage-link]                     |
 | [@ethereumjs/util][util-package]                 | [![NPM Package][util-npm-badge]][util-npm-link]                 | [![Util Issues][util-issues-badge]][util-issues-link]                         | [![Actions Status][util-actions-badge]][util-actions-link]                 | [![Code Coverage][util-coverage-badge]][util-coverage-link]                 |
+| [@ethereumjs/verkle][verkle-package]             | [![NPM Package][verkle-npm-badge]][verkle-npm-link]             | [![VM Issues][verkle-issues-badge]][verkle-issues-link]                       | [![Actions Status][verkle-actions-badge]][verkle-actions-link]             | [![Code Coverage][verkle-coverage-badge]][verkle-coverage-link]             |
 | [@ethereumjs/vm][vm-package]                     | [![NPM Package][vm-npm-badge]][vm-npm-link]                     | [![VM Issues][vm-issues-badge]][vm-issues-link]                               | [![Actions Status][vm-actions-badge]][vm-actions-link]                     | [![Code Coverage][vm-coverage-badge]][vm-coverage-link]                     |
 | [@ethereumjs/wallet][wallet-package]             | [![NPM Package][wallet-npm-badge]][wallet-npm-link]             | [![StateManager Issues][wallet-issues-badge]][wallet-issues-link]             | [![Actions Status][wallet-actions-badge]][wallet-actions-link]             | [![Code Coverage][wallet-coverage-badge]][wallet-coverage-link]             |
 
@@ -235,6 +236,15 @@ Most packages are [MPL-2.0](<https://tldrlegal.com/license/mozilla-public-licens
 [statemanager-actions-link]: https://github.com/ethereumjs/ethereumjs-monorepo/actions?query=workflow%3A%22StateManager%22
 [statemanager-coverage-badge]: https://codecov.io/gh/ethereumjs/ethereumjs-monorepo/branch/master/graph/badge.svg?flag=statemanager
 [statemanager-coverage-link]: https://codecov.io/gh/ethereumjs/ethereumjs-monorepo/tree/master/packages/statemanager
+[verkle-package]: ./packages/verkle
+[verkle-npm-badge]: https://img.shields.io/npm/v/@ethereumjs/verkle.svg
+[verkle-npm-link]: https://www.npmjs.com/package/@ethereumjs/verkle
+[verkle-issues-badge]: https://img.shields.io/github/issues/ethereumjs/ethereumjs-monorepo/package:%20verkle?label=issues
+[verkle-issues-link]: https://github.com/ethereumjs/ethereumjs-monorepo/issues?q=is%3Aopen+is%3Aissue+label%3A"package%3A+verkle"
+[verkle-actions-badge]: https://github.com/ethereumjs/ethereumjs-monorepo/workflows/Verkle/badge.svg
+[verkle-actions-link]: https://github.com/ethereumjs/ethereumjs-monorepo/actions?query=workflow%3A%22Verkle%22
+[verkle-coverage-badge]: https://codecov.io/gh/ethereumjs/ethereumjs-monorepo/branch/master/graph/badge.svg?flag=verkle
+[verkle-coverage-link]: https://codecov.io/gh/ethereumjs/ethereumjs-monorepo/tree/master/packages/verkle
 [vm-package]: ./packages/vm
 [vm-npm-badge]: https://img.shields.io/npm/v/@ethereumjs/vm.svg
 [vm-npm-link]: https://www.npmjs.com/package/@ethereumjs/vm

diff --git a/package-lock.json b/package-lock.json
diff --git a/packages/util/src/bytes.ts b/packages/util/src/bytes.ts
@@ -444,5 +444,61 @@ export const concatBytes = (...arrays: Uint8Array[]): Uint8Array => {
   return result
 }
 
+/**
+ * @notice Convert a Uint8Array to a 32-bit integer
+ * @param {Uint8Array} bytes The input Uint8Array from which to read the 32-bit integer.
+ * @param {boolean} littleEndian True for little-endian, undefined or false for big-endian.
+ * @throws {Error} If the input Uint8Array has a length less than 4.
+ * @return {number} The 32-bit integer read from the input Uint8Array.
+ */
+export function bytesToInt32(bytes: Uint8Array, littleEndian: boolean = false): number {
+  if (bytes.length < 4) {
+    throw new Error('The input Uint8Array must have at least 4 bytes.')
+  }
+  const dataView = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
+  return dataView.getInt32(0, littleEndian)
+}
+
+/**
+ * @notice Convert a Uint8Array to a 64-bit bigint
+ * @param {Uint8Array} bytes The input Uint8Array from which to read the 64-bit bigint.
+ * @param {boolean} littleEndian True for little-endian, undefined or false for big-endian.
+ * @throws {Error} If the input Uint8Array has a length less than 8.
+ * @return {bigint} The 64-bit bigint read from the input Uint8Array.
+ */
+export function bytesToBigInt64(bytes: Uint8Array, littleEndian: boolean = false): bigint {
+  if (bytes.length < 8) {
+    throw new Error('The input Uint8Array must have at least 8 bytes.')
+  }
+  const dataView = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
+  return dataView.getBigInt64(0, littleEndian)
+}
+
+/**
+ * @notice Convert a 32-bit integer to a Uint8Array.
+ * @param {number} value The 32-bit integer to convert.
+ * @param {boolean} littleEndian True for little-endian, undefined or false for big-endian.
+ * @return {Uint8Array} A Uint8Array of length 4 containing the integer.
+ */
+export function int32ToBytes(value: number, littleEndian: boolean = false): Uint8Array {
+  const buffer = new ArrayBuffer(4)
+  const dataView = new DataView(buffer)
+  dataView.setInt32(0, value, littleEndian)
+  return new Uint8Array(buffer)
+}
+
+/**
+ * @notice Convert a 64-bit bigint to a Uint8Array.
+ * @param {bigint} value The 64-bit bigint to convert.
+ * @param {boolean} littleEndian True for little-endian, undefined or false for big-endian.
+ * @return {Uint8Array} A Uint8Array of length 8 containing the bigint.
+ */
+export function bigInt64ToBytes(value: bigint, littleEndian: boolean = false): Uint8Array {
+  const buffer = new ArrayBuffer(8)
+  const dataView = new DataView(buffer)
+  dataView.setBigInt64(0, value, littleEndian)
+  return new Uint8Array(buffer)
+}
+
 // eslint-disable-next-line no-restricted-imports
 export { bytesToUtf8, equalsBytes, utf8ToBytes } from 'ethereum-cryptography/utils.js'
diff --git a/packages/verkle/.c8rc.json b/packages/verkle/.c8rc.json
@@ -0,0 +1,4 @@
+{
+  "extends": "../../config/.c8rc.json",
+  "include": ["src/**/*.ts"]
+}
diff --git a/packages/verkle/.eslintrc.cjs b/packages/verkle/.eslintrc.cjs
@@ -0,0 +1,14 @@
+module.exports = {
+  extends: '../../config/eslint.cjs',
+  parserOptions: {
+    project: ['./tsconfig.json', './tsconfig.benchmarks.json'],
+  },
+  overrides: [
+    {
+      files: ['benchmarks/*.ts'],
+      rules: {
+        'no-console': 'off',
+      },
+    },
+  ],
+}
diff --git a/packages/verkle/.gitignore b/packages/verkle/.gitignore
@@ -0,0 +1 @@
+
diff --git a/packages/verkle/.npmignore b/packages/verkle/.npmignore
@@ -0,0 +1,2 @@
+test/
+src/
diff --git a/packages/verkle/.prettierignore b/packages/verkle/.prettierignore
@@ -0,0 +1,6 @@
+node_modules
+.vscode
+dist
+.nyc_output
+*.json
+docs
diff --git a/packages/verkle/CHANGELOG.md b/packages/verkle/CHANGELOG.md
@@ -0,0 +1,11 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
+(modification: no type change headlines) and this project adheres to
+[Semantic Versioning](http://semver.org/spec/v2.0.0.html).
+
+## [0.0.1] - 2023-10-15
+
+- Initial development release
diff --git a/packages/verkle/README.md b/packages/verkle/README.md
@@ -0,0 +1,112 @@
+# @ethereumjs/verkle
+
+[![NPM Package][verkle-npm-badge]][verkle-npm-link]
+[![GitHub Issues][verkle-issues-badge]][verkle-issues-link]
+[![Actions Status][verkle-actions-badge]][verkle-actions-link]
+[![Code Coverage][verkle-coverage-badge]][verkle-coverage-link]
+[![Discord][discord-badge]][discord-link]
+
+| Implementation of [Verkle Trees](https://ethereum.org/en/roadmap/verkle-trees/) as specified in [EIP-6800](https://eips.ethereum.org/EIPS/eip-6800) |
+| --------------------------------------------------------------------------------------------------------------------------------------------------- |
+
+> Verkle trees are a cryptographic data structure proposed for use in Ethereum to optimize storage and transaction verification. They combine features of Merkle Patricia Tries and Vector Commitment Trees to offer efficient data verification with smaller proof sizes. The goal is to improve scalability and efficiency in Ethereum's network operations.
+
+This package is currently in early alpha and is a work in progress. It is not intended for use in production environments, but rather for research and development purposes. Any help in improving the package is very much welcome.
+
+## Installation
+
+To obtain the latest version, simply require the project using `npm`:
+
+```shell
+npm install @ethereumjs/verkle
+```
+
+## Usage
+
+This class implements the basic [Modified Merkle Patricia Trie](https://ethereum.org/en/developers/docs/data-structures-and-encoding/patricia-merkle-trie/) in the `Trie` base class, which you can use with the `useKeyHashing` option set to `true` to create a trie which stores values under the `keccak256` hash of its keys (this is the Trie flavor which is used in Ethereum production systems).
+
+Checkpointing functionality to `Trie` through the methods `checkpoint`, `commit` and `revert`.
+
+It is best to select the variant that is most appropriate for your unique use case.
+
+### Initialization and Basic Usage
+
+```typescript
+import { VerkleTrie } from '@ethereumjs/verkle'
+import { bytesToUtf8, utf8ToBytes } from 'ethereumjs/util'
+
+const trie = new VerkleTrie()
+
+async function test() {
+  await trie.put(utf8ToBytes('test'), utf8ToBytes('one'))
+  const value = await trie.get(utf8ToBytes('test'))
+  console.log(value ? bytesToUtf8(value) : 'not found') // 'one'
+}
+
+test()
+```
+
+## Proofs
+
+### Verkle Proofs
+
+The EthereumJS Verkle package is still in its infancy, and as such, it does not currently support Verkle proof creation and verification. Support for Verkle proofs will be added eventually.
+
+## Examples
+
+You can find additional examples complete with detailed explanations [here](./examples/README.md).
+
+## Browser
+
+With the breaking release round in Summer 2023 we have added hybrid ESM/CJS builds for all our libraries (see section below) and have eliminated many of the caveats which had previously prevented frictionless browser usage.
+
+It is now easily possible to run a browser build of one of the EthereumJS libraries within a modern browser using the provided ESM build. For a setup example see [./examples/browser.html](./examples/browser.html).
+
+## API
+
+### Docs
+
+Generated TypeDoc API [Documentation](./docs/README.md)
+
+### Hybrid CJS/ESM Builds
+
+With the breaking releases from Summer 2023 we have started to ship our libraries with both CommonJS (`cjs` folder) and ESM builds (`esm` folder), see `package.json` for the detailed setup.
+
+If you use an ES6-style `import` in your code, files from the ESM build will be used:
+
+```typescript
+import { EthereumJSClass } from '@ethereumjs/[PACKAGE_NAME]'
+```
+
+If you use Node.js-specific `require`, the CJS build will be used:
+
+```typescript
+const { EthereumJSClass } = require('@ethereumjs/[PACKAGE_NAME]')
+```
+
+Using ESM will give you additional advantages over CJS beyond browser usage like static code analysis / Tree Shaking, which CJS cannot provide.
+
+## References
+
+- Wiki
+  - [Overview of verkle tries](https://ethereum.org/en/roadmap/verkle-trees/)
+  - [Verkle tries General Resource](https://verkle.info/)
+
+## EthereumJS
+
+See our organizational [documentation](https://ethereumjs.readthedocs.io) for an introduction to `EthereumJS` as well as information on current standards and best practices. If you want to join for work or carry out improvements on the libraries, please review our [contribution guidelines](https://ethereumjs.readthedocs.io/en/latest/contributing.html) first.
+
+## License
+
+[MPL-2.0](<https://tldrlegal.com/license/mozilla-public-license-2.0-(mpl-2)>)
+
+[discord-badge]: https://img.shields.io/static/v1?logo=discord&label=discord&message=Join&color=blue
+[discord-link]: https://discord.gg/TNwARpR
+[verkle-npm-badge]: https://img.shields.io/npm/v/@ethereumjs/verkle.svg
+[verkle-npm-link]: https://www.npmjs.com/package/@ethereumjs/verkle
+[verkle-issues-badge]: https://img.shields.io/github/issues/ethereumjs/ethereumjs-monorepo/package:%20verkle?label=issues
+[verkle-issues-link]: https://github.com/ethereumjs/ethereumjs-monorepo/issues?q=is%3Aopen+is%3Aissue+label%3A"package%3A+verkle"
+[verkle-actions-badge]: https://github.com/ethereumjs/ethereumjs-monorepo/workflows/Trie/badge.svg
+[verkle-actions-link]: https://github.com/ethereumjs/ethereumjs-monorepo/actions?query=workflow%3A%22Trie%22
+[verkle-coverage-badge]: https://codecov.io/gh/ethereumjs/ethereumjs-monorepo/branch/master/graph/badge.svg?flag=verkle
+[verkle-coverage-link]: https://codecov.io/gh/ethereumjs/ethereumjs-monorepo/tree/master/packages/verkle
diff --git a/packages/verkle/package.json b/packages/verkle/package.json
@@ -0,0 +1,60 @@
+{
+  "name": "@ethereumjs/verkle",
+  "version": "0.0.1",
+  "description": "Implementation of verkle tries as used in Ethereum.",
+  "keywords": [
+    "verkle",
+    "trie",
+    "ethereum"
+  ],
+  "homepage": "https://github.com/ethereumjs/ethereumjs-monorepo/tree/master/packages/verkle#readme",
+  "bugs": {
+    "url": "https://github.com/ethereumjs/ethereumjs-monorepo/issues?q=is%3Aissue+label%3A%22package%3A+verkle%22"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/ethereumjs/ethereumjs-monorepo.git"
+  },
+  "license": "MPL-2.0",
+  "author": "EthereumJS Team",
+  "contributors": [
+    {
+      "name": "Gabriel Rocheleau",
+      "url": "https://github.com/gabrocheleau"
+    }
+  ],
+  "main": "dist/cjs/index.js",
+  "type": "commonjs",
+  "module": "dist/esm/index.js",
+  "exports": {
+    ".": {
+      "import": "./dist/esm/index.js",
+      "require": "./dist/cjs/index.js"
+    }
+  },
+  "files": [
+    "dist",
+    "src"
+  ],
+  "scripts": {
+    "build": "../../config/cli/ts-build.sh",
+    "clean": "../../config/cli/clean-package.sh",
+    "coverage": "npx vitest run --coverage.enabled --coverage.reporter=lcov",
+    "docs:build": "typedoc --options typedoc.cjs",
+    "examples": "ts-node ../../scripts/examples-runner.ts -- verkle",
+    "lint": "../../config/cli/lint.sh",
+    "lint:diff": "../../config/cli/lint-diff.sh",
+    "lint:fix": "../../config/cli/lint-fix.sh",
+    "prepublishOnly": "../../config/cli/prepublish.sh",
+    "test": "npx vitest run",
+    "tsc": "../../config/cli/ts-compile.sh"
+  },
+  "dependencies": {
+    "@ethereumjs/rlp": "5.0.0",
+    "@ethereumjs/util": "9.0.0",
+    "lru-cache": "^10.0.0"
+  },
+  "engines": {
+    "node": ">=18"
+  }
+}