Agnostic Database Implementation

[faustbrian]

Not sure if this is already planned or was discussed but what's the status on an agnostic database implementation? Right now LevelDB is hardcoded through the db.ts file, so if you want to use something like https://github.com/DoctorEvidence/lmdb-js you will need to fork and modify that file.

The least invasive approach would probably be to change https://github.com/ethereumjs/ethereumjs-monorepo/blob/master/packages/trie/src/baseTrie.ts#L64 to this.db = db ?? new DB() and https://github.com/ethereumjs/ethereumjs-monorepo/blob/master/packages/trie/src/baseTrie.ts#L60 to constructor(db?: DB | null, root?: Buffer, deleteFromDB: boolean = false) { where DB would then become an interface. This is obviously a small breaking change but would allow the use of custom DB implementations.

Are there any unique characteristics that make LMDB a requirement/best-in-class for how the package is designed or would you be open to a pull-request to make it agnostic and allow for swapping out through the trie constructor?

Example of an LMDB driver

import lmdb from 'lmdb';

export interface IDatabase {
  get(key: Buffer): Promise<Buffer | null>;
  put(key: Buffer, value: Buffer): Promise<void>;
  del(key: Buffer): Promise<void>;
  batch(opStack: BatchDBOp[]): Promise<void>;
  copy(): IDatabase;
}

export class LMDB implements IDatabase {
  _db: lmdb.Database;

  constructor(database?: lmdb.Database) {
    this._db =
      database ??
      lmdb.open({
        compression: true,
        name: 'ethereumjs',
        path: '',
      });
  }

  async get(key: Buffer): Promise<Buffer | null> {
    return this._db.get(key);
  }
  
  async put(key: Buffer, value: Buffer): Promise<void> {
    await this._db.put(key, value);
  }
  
  async del(key: Buffer): Promise<void> {
    await this._db.remove(key);
  }
  
  async batch(opStack: BatchDBOp[]): Promise<void> {
    await this._db.batch(async () => {
      for (const stack of opStack) {
        if (stack.type === 'put') {
          this._db.put(stack.key, stack.value);
        }

        if (stack.type === 'del') {
          this._db.remove(stack.key);
        }
      }
    });
  }

  copy(): DB {
    return new DB(this._db);
  }
}

[holgerd77]

Hi there,
it would be really great to have this additional abstraction level to allow to swap out the underlying data store, so a PR on this would be highly appreciated!! 😀 💯

We are currently (actually: already for the last couple of months) preparing breaking releases on most of the libraries (including the MPT library) and releases are - relatively - imminent, within a couple of weeks (let's say 4-6 weeks roughly), so this would be a good time to do such kind of work.

We are doing all breaking work towards the develop branch, so if you want to prepare a PR please make sure to target that branch!

Be aware that there are already a lot of breaking changes merged, so the code on this branch deviates substantially from the master branch.

In the Trie library breaking changes are limited though, one major thing is that options are now passed in as an options dictionary - as it is being done in the other libraries - which in doubt should make your life rather easier though.

[faustbrian]

Great! Will take a look this week because there's some spots that leak the underlying DB, like for example https://github.com/ethereumjs/ethereumjs-monorepo/blob/develop/packages/trie/src/checkpointDb.ts#L131. Will work on a PR once I've figured out how to best isolate that without creating circular dependencies.