Adds Neutrino Client #1163

manavdesai27 · 2023-07-10T23:47:54Z

This pr implements neutrino client mode. This has the commit history cleaned for better reviewing.

Added Neutrino node which will communicate with the network.
Added
- sendGetCFilters - Gets compact filters from peer.
- sendGetCFHeaders - Gets Compact filter headers from peer.
- sendGetCFCheckpt - Gets Filters using checkpoints
Added saveFilter and saveFilterHeader methods in filterIndexer. While saving filterHeader I am keeping the filter field empty.
Added NeutrinoState which helps to keep track of the filterHeaderHeight and the filterHeight
Added
- Headers-sync: This skips downloading the blocks and instead saves the block headers and once filter headers are synced, emits event called headersFull, which indicates that we can start fetching filter headers
- Filter Headers Sync: Syncs filter headers and saves them after certain assertions. After all cfheaders are synced we emit cfheaders event, which indicates that we can start fetching filters.
- Filters Sync: Syncs filters and saves them after certain assertions. After saving the filter, we emit cfilter event, which sends the filter to the wallet and marks beginning of filter matching for the particular filter.
Added checkFilter: Checks the filter against all address hashes in the wallet db. Address hashes are first converted to output scripts and that is then matched against the filter.
Added getBlockFromNode and getBlockPeer: These methods help the client to download a particular block when any filter matches on the wallet side.
Added RPC methods:
- getfiltercount: Gets the filter height in NeutrinoState
- getfilterheadercount: Gets the filter header height in NeutrinoState
- getblockfilterheader: Retrieves filter header for a particular block

codecov · 2023-07-10T23:58:00Z

Codecov Report

Patch coverage: 76.47% and project coverage change: +0.49 🎉

Comparison is base (a4bf281) 70.26% compared to head (8e0bec0) 70.75%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1163      +/-   ##
==========================================
+ Coverage   70.26%   70.75%   +0.49%     
==========================================
  Files         174      175       +1     
  Lines       27367    27833     +466     
==========================================
+ Hits        19229    19693     +464     
- Misses       8138     8140       +2

Impacted Files	Coverage Δ
lib/bcoin-browser.js	`0.00% <0.00%> (ø)`
lib/bcoin.js	`0.00% <0.00%> (ø)`
lib/blockchain/layout.js	`100.00% <ø> (ø)`
lib/client/node.js	`25.86% <0.00%> (-1.42%)`	⬇️
lib/node/fullnode.js	`85.42% <ø> (+0.88%)`	⬆️
lib/protocol/networks.js	`100.00% <ø> (ø)`
lib/wallet/client.js	`0.00% <0.00%> (ø)`
lib/wallet/nullclient.js	`87.50% <ø> (ø)`
lib/node/http.js	`58.20% <12.50%> (-1.74%)`	⬇️
lib/node/rpc.js	`42.04% <52.00%> (+0.17%)`	⬆️
... and 10 more

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

bin/bcoin

bin/neutrino

lib/node/neutrino.js

pinheadmz · 2023-07-14T19:44:05Z

lib/indexer/indexer.js

@@ -50,6 +50,8 @@ class Indexer extends EventEmitter {
    this.blocks = this.options.blocks;
    this.chain = this.options.chain;

+    this.neutrino = this.options.neutrino;


this should probably just live in FilterIndexer and leave the base class alone

pinheadmz · 2023-07-14T19:44:44Z

lib/indexer/indexer.js

+    if (this.neutrino) {
+      if (!this.batch)
+        this.start();
+      return true;
+    }


instead of doing this here, I think what we may need to do is in FilterIndexer.indexBlock() just check if neutrino, return false

If I implement it this way, the logStatus at line 458 in indexer.js gives an error

And also, the indexBlock is expected to return a value. Return false gives out an error

lib/blockchain/chaindb.js

pinheadmz · 2023-07-14T19:46:44Z

lib/blockchain/chaindb.js

@@ -1001,7 +1008,7 @@ class ChainDB {
   */

  async getRawBlock(block) {
-    if (this.options.spv)
+    if (this.options.spv && !this.options.neutrino)


hmm what's wrong with this line? We can't get raw blocks in neutrino mode

We would need this in order to check if a requested block by wallet already exists in the database or not, in order to skip fetching it if it exists.

pinheadmz · 2023-07-14T19:49:57Z

lib/blockchain/chaindb.js

@@ -1478,7 +1493,7 @@ class ChainDB {
  async saveBlock(entry, block, view) {
    const hash = block.hash();

-    if (this.options.spv)
+    if (this.options.spv && !this.neutrinoSave)


same question here - SPV mode does just fine without calling saveBlock() Neutrino should be the same. Don't we save ChainEntry (headers) outside this function?

When we get a pruned block on request by wallet, without the neutrinoSave flag, the chain skips saving the block.

lib/node/fullnode.js

lib/wallet/wallet.js

test/wallet-neutrino-test.js

pinheadmz · 2023-07-14T20:04:36Z

test/p2p-test.js

@@ -73,58 +72,6 @@ describe('P2P', function () {
    await node2.close();
  });

-  describe('BIP157', function () {


Interesting that this change triggered a bunch of code coverage warnings, will have to investigate further.

bin/neutrino

lib/blockchain/chain.js

lib/net/peer.js

lib/node/http.js

lib/node/neutrino.js

lib/node/rpc.js

test/neutrino-test.js

test/p2p-test.js

feat<sync>: completed HeadersSync, CFHeaderSync, CFilterSync

…header methods

manavdesai27 marked this pull request as ready for review July 11, 2023 00:26

pinheadmz reviewed Jul 14, 2023

View reviewed changes

bin/bcoin Outdated Show resolved Hide resolved

bin/neutrino Outdated Show resolved Hide resolved

lib/node/neutrino.js Outdated Show resolved Hide resolved

pinheadmz reviewed Jul 14, 2023

View reviewed changes

masterchief164 suggested changes Jul 16, 2023

View reviewed changes

manavdesai27 force-pushed the neutrino-client branch 3 times, most recently from 059de6e to 7d2635c Compare July 17, 2023 05:11

manavdesai27 requested review from pinheadmz and masterchief164 July 17, 2023 05:14

manavdesai27 force-pushed the neutrino-client branch 2 times, most recently from a8bb572 to 934bdcc Compare July 17, 2023 06:24

manavdesai27 added 11 commits July 17, 2023 23:46

feat<node>: added Neutrino node

c979112

feat<node>: added sendGetCFilters, sendGetCFHeaders, sendGetCFCheckpt

e8f7df8

feat<indexer>: added saveFilter, saveFilterHeader methods

7fab563

feat<node>: added NeutrinoState

c75d4ca

feat<sync>: completed HeadersSync, CFHeaderSync, CFilterSync

feat<client>: added getBlockFromNode, getBlockPeer

41009b8

feat<rpc>: added getfiltercount, getfilterheadercount, getblockfilter…

0f4519e

…header methods

feat<wallet>: added checkFilter

5fdee1b

test<node>: added neutrino-test

c717a9b

test<wallet>: added wallet-neutrino-test

d0bf737

test<p2p>: fix p2p-test

7b78379

fix<review>

62dfdbd

manavdesai27 force-pushed the neutrino-client branch 2 times, most recently from 1dc89dc to bffb60a Compare July 20, 2023 06:10

feat<cli>: added filterheader

8e0bec0

manavdesai27 force-pushed the neutrino-client branch from bffb60a to 8e0bec0 Compare July 20, 2023 14:48

pinheadmz removed their request for review July 1, 2024 19:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Neutrino Client #1163

Adds Neutrino Client #1163

manavdesai27 commented Jul 10, 2023

codecov bot commented Jul 10, 2023 •

edited

Loading

pinheadmz Jul 14, 2023

pinheadmz Jul 14, 2023

manavdesai27 Jul 14, 2023

manavdesai27 Jul 15, 2023

pinheadmz Jul 14, 2023

manavdesai27 Jul 14, 2023

pinheadmz Jul 14, 2023

manavdesai27 Jul 14, 2023

pinheadmz Jul 14, 2023

Adds Neutrino Client #1163

Are you sure you want to change the base?

Adds Neutrino Client #1163

Conversation

manavdesai27 commented Jul 10, 2023

codecov bot commented Jul 10, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 10, 2023 •

edited

Loading