crc32 speed boost

[fuweichin]

I'm trying to speed up conflux by replacing crc32 implementation with thirdparty high performance crc32 lib (which uses asm.js/wasm/simd), and I did some tests:

metric \ crc32 impl	in-package crc.js	hash-wasm crc32
crc32 speed when running standalone	440.62 MB/s	1312.82 MB/s
conflux writer speed when used by conflux	163.61 MB/s	462.58 MB/s

Note: tested with 1GB in-memory file, 2MB chunk size.

So my proposal is to adjust conflux to allow developers use/configure custom crc32 implementaion.

[fuweichin]

The code to test conflux writer speed:

window.testWriterSpeed=function(){
  let memFile=new File([new ArrayBuffer(1*1024**3)],'1GB.bin', {type:'application/octet-stream'});
  console.time('zip');
  let iterator=[memFile].values();
  let entryStream=new ReadableStream({
    async pull(controller) {
      const {done, value:entry} = iterator.next();
      if (done)
        return controller.close();
      return controller.enqueue(entry);
    },
  });

  let nullStream=new WritableStream({
    async write(chunk) {},
    close() {}
  });

  
  entryStream.pipeThrough(new Writer()).pipeTo(nullStream).finally(()=>{
    console.timeEnd('zip');
  });
};

and here is my Crc32 implementation based on hash-wasm:

import 'hash-wasm/dist/crc32.umd.min.js'; /* global hashwasm */

class Crc32 {
  constructor() {
  }

  async init(){
    let crc32=await hashwasm.createCRC32();
    crc32.init();
    this.crc32 = crc32;
  }

  append(data) {
    this.crc32.update(data);
  }

  get() {
    return new DataView(this.crc32.digest('binary').buffer).getUint32(0);
  }
}

export default Crc32;

Note: this Crc32 implementation defined a method async init(){...} and the method must be called and fulfilled before any append() call.