How to speed up BSC syncing

[billyadelphia]

After 2 days of syncing, I only got 10 blocks per second on syncing.

6/16 Execution] Executed blocks         number=7018832 blk/s=10.4 tx/s=3536.2 Mgas/s=433.8 gasState=0.11 batch=138.9Mb alloc=2.1Gb sys=5.6Gb

It will take around 2 weeks to fully synced, it's too long.
How to improve the syncing speed for BSC ?

erigon version 2022.99.99-dev-2ec9733c (just got updated this morning)

Here the command :

/root/bin/erigon --syncmode=snap --torrent.download.rate=500mb --datadir=/root/erigon_data --chain=bsc --maxpeers 300 --http --ws --http.api=eth,erigon,web3,net,debug,trace,txpool

Here my bare-metal machine :

AMD Ryzen™ 9 3900 12 Core
128 GB DDR4 ECC
2x 2TB NVME SSD with RAID 0 (Software)
1Gbps Internet

Here the complete logs :

May 22 15:47:19 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:19.764] Initialising Ethereum protocol           network=56
May 22 15:47:19 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:19.764] Effective                                prune_flags= snapshot_flags="--syncmode=snap"
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.072] Starting private RPC server              on=127.0.0.1:9090
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.072] new subscription to logs established
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.072] rpc filters: subscribing to Erigon events
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.072] New txs subscriber joined
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.072] new subscription to newHeaders established
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.073] HTTP endpoint opened                     url=localhost:8545 ws=true ws.compression=true grpc=false
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.077] Started P2P networking                   version=66 self=enode://ffab6ba00f094b4361afff7a99ca903a3ea965129b0c76f55f7c7ec773ce7e599e88d6cdba1d82f1d6b7aa700066a6cca57ae5b9df4cc1bc22bafdb5883725e5@127.0.0.1:30303 name=erigon/v2022.99.99-dev-2ec9733c/linux-amd64/go1.18.2
May 22 15:47:20 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:20.082] [Snapshots] Fetching torrent files metadata
May 22 15:47:21 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:21.632] [Snapshots] Stat                         blocks=16999999 alloc=2.0Gb sys=3.6Gb
May 22 15:47:21 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:21.633] [1/16 Headers] Waiting for headers...    from=18011383
May 22 15:47:41 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:41.633] [1/16 Headers] Wrote block headers       number=18011383 blk/second=0.000 alloc=2.3Gb sys=3.6Gb
May 22 15:47:41 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:41.633] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=89 respMin=18013434 respMax=18024769
May 22 15:47:41 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:41.634] {18013434-18024769 links=11336 (18013434-18024769)} => 5d6418fe72b7b3d44c0abb6791827cac6016980913034556f47ca3e8f21a15e2
May 22 15:47:49 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:49.496] [1/16 Headers] Processed                 highest inserted=18024772 age=1s
May 22 15:47:51 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:47:51.850] [4/16 Bodies] Processing bodies...       from=18011383 to=18024772
May 22 15:48:11 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:48:11.851] [4/16 Bodies] Wrote block bodies         block_num=18014071 delivery/sec=29.5Mb wasted/sec=0b alloc=2.7Gb sys=4.2Gb
May 22 15:48:31 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:48:31.851] [4/16 Bodies] Wrote block bodies         block_num=18018424 delivery/sec=499.7Kb wasted/sec=0b alloc=3.0Gb sys=4.2Gb
May 22 15:48:51 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:48:51.851] [4/16 Bodies] Wrote block bodies         block_num=18018424 delivery/sec=0b wasted/sec=0b alloc=3.1Gb sys=4.2Gb
May 22 15:48:59 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:48:59.491] [4/16 Bodies] Processed                  highest=18024772
May 22 15:48:59 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:48:59.491] [4/16 Bodies] DONE                       in=1m7.640695327s
May 22 15:48:59 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:48:59.491] [5/16 Senders] Started                   from=18011383 to=18024772
May 22 15:49:10 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:49:10.211] [6/16 Execution] Blocks execution        from=7017311 to=18024772
May 22 15:49:12 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:49:12.894] [p2p] GoodPeers                          eth66=44
May 22 15:49:30 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:49:30.246] [6/16 Execution] Executed blocks         number=7017526 blk/s=10.7 tx/s=4018.8 Mgas/s=451.8 gasState=0.02 batch=17.6Mb alloc=2.8Gb sys=5.6Gb
May 22 15:49:50 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:49:50.269] [6/16 Execution] Executed blocks         number=7017739 blk/s=10.6 tx/s=4180.5 Mgas/s=435.0 gasState=0.03 batch=36.9Mb alloc=2.2Gb sys=5.6Gb
May 22 15:50:10 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:50:10.219] [6/16 Execution] Executed blocks         number=7017969 blk/s=11.5 tx/s=3748.7 Mgas/s=434.8 gasState=0.05 batch=57.0Mb alloc=3.4Gb sys=5.6Gb
May 22 15:50:30 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:50:30.331] [6/16 Execution] Executed blocks         number=7018197 blk/s=11.3 tx/s=3891.4 Mgas/s=430.0 gasState=0.06 batch=77.2Mb alloc=2.7Gb sys=5.6Gb
May 22 15:50:50 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:50:50.316] [6/16 Execution] Executed blocks         number=7018413 blk/s=10.8 tx/s=3735.8 Mgas/s=428.9 gasState=0.08 batch=97.4Mb alloc=3.8Gb sys=5.6Gb
May 22 15:51:10 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:51:10.277] [6/16 Execution] Executed blocks         number=7018623 blk/s=10.5 tx/s=3952.6 Mgas/s=436.1 gasState=0.10 batch=117.9Mb alloc=3.0Gb sys=5.6Gb
May 22 15:51:12 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:51:12.894] [p2p] GoodPeers                          eth66=76
May 22 15:51:30 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:51:30.288] [6/16 Execution] Executed blocks         number=7018832 blk/s=10.4 tx/s=3536.2 Mgas/s=433.8 gasState=0.11 batch=138.9Mb alloc=2.1Gb sys=5.6Gb
May 22 15:51:50 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:51:50.323] [6/16 Execution] Executed blocks         number=7019070 blk/s=11.9 tx/s=3711.5 Mgas/s=440.2 gasState=0.13 batch=160.9Mb alloc=3.5Gb sys=5.6Gb
May 22 15:52:10 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:52:10.266] [6/16 Execution] Executed blocks         number=7019319 blk/s=12.5 tx/s=3501.3 Mgas/s=443.1 gasState=0.14 batch=183.2Mb alloc=2.6Gb sys=5.6Gb
May 22 15:52:30 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:52:30.227] [6/16 Execution] Executed blocks         number=7019561 blk/s=12.1 tx/s=3639.9 Mgas/s=433.1 gasState=0.16 batch=204.3Mb alloc=3.7Gb sys=5.6Gb
May 22 15:52:50 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:52:50.336] [6/16 Execution] Executed blocks         number=7019755 blk/s=9.6 tx/s=3665.8 Mgas/s=428.7 gasState=0.17 batch=222.7Mb alloc=2.7Gb sys=5.6Gb
May 22 15:53:10 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:53:10.280] [6/16 Execution] Executed blocks         number=7019994 blk/s=12.0 tx/s=4350.2 Mgas/s=430.1 gasState=0.19 batch=241.1Mb alloc=3.7Gb sys=5.6Gb
May 22 15:53:12 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:53:12.893] [p2p] GoodPeers                          eth66=87
May 22 15:53:30 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:53:30.319] [6/16 Execution] Executed blocks         number=7020244 blk/s=12.5 tx/s=3877.1 Mgas/s=432.9 gasState=0.21 batch=261.0Mb alloc=2.7Gb sys=5.6Gb
May 22 15:53:50 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:53:50.245] [6/16 Execution] Executed blocks         number=7020470 blk/s=11.3 tx/s=3906.1 Mgas/s=431.0 gasState=0.22 batch=281.0Mb alloc=3.7Gb sys=5.6Gb
May 22 15:54:10 Ubuntu-2004-focal-64-minimal erigon[3379332]: [INFO] [05-22|15:54:10.328] [6/16 Execution] Executed blocks         number=7020658 blk/s=9.4 tx/s=4028.5 Mgas/s=427.5 gasState=0.24 batch=301.2Mb alloc=2.6Gb sys=5.6Gb

[AskAlexSharov]

1. blocks are not equal in size
2. yep, it's not fast process
3. general recommendation: can speedup by giving Disc with lower latency (not throughput), or more RAM
4. for now: execution of all blocks - it's only way of syncing

[billyadelphia]

@AskAlexSharov does adding --state.cache can improve the syncing ? Since only 30GB RAM is used out of 128GB
And I think the NVME disk latency is good enough, since it's a bare-metal server, not VM / Cloud disk like AWS

[AskAlexSharov]

• "does adding --state.cache can improve the syncing ?" - no. --batchSize maybe a bit.
• in our Makefile you can set MDBX_FORCE_ASSERTIONS=0. also may help a bit.
• "NVME disk latency is good enough" - true. but Raid can be miss-configured.
• I think your node is ok, my advise: wait
• "30GB RAM is used out of 128GB" - read next section to know more about Erigon's DB and RAM: https://github.com/ledgerwatch/erigon#htop-shows-incorrect-memory-usage

[mwpastore]

I know this is an old issue, but I would have asked: How is your RAID0 configured? What filesystem are you using? It could be a stripe width problem. Or one or both NVMe drives are attached to the PCH (instead of directly to the CPU) and are taking a latency hit and/or bottlenecking at the chipset. Or the NVMe controllers are thermal-throttling. And as Alex said, any kind of software RAID is going to impact latency. Oh, and be sure that your firewall is forwarding the correct ports for both TCP and UDP.

I've been struggling with performance issues as well. I'm mired around 20 blk/s for the last ~1M blocks on a Ryzen 5 5600G with 32GB DDR4-3200 CL22 and high-end SATA SSDs. I'm feeling impatient because I had to re-sync from scratch due to some bad pruning options and now my client (i.e. a paying customer) is getting slashed. It's frustrating that the CPU is mostly idle, the disks are mostly idle, the network uplink is mostly idle, etc. and it's moving at a snail's pace. And of course tuning is challenging because every time you interrupt erigon (and/or the CL client) to try something new you risk having to re-process something you've already processed.

I'm using LVM2 and XFS. LVM2 can be a little frustrating (e.g. no round-robin reading from mirrors) but it's flexible and it has lots of features and knobs. It uses mdraid, dm-cache, etc. on the backend. XFS can automatically determine sectors sizes, stripe widths, etc. from the underlying block device(s) and in general it's a great filesystem for databases. My primary block device for the erigon database and snapshots is an LVM2 mirror.

There's a pair of fast Gen3 NVMe drives in the system as well; they're not big enough to hold the entire database, but I've got a dedicated NVMe-backed LV mounted at mainnet/etl-temp and I've played around with write-back and -through caching. I've tried pinning processes using taskset, and switching from NVMe interrupt to polling mode. I've also tried increasing --batchSize and --rpc.batch.concurrency. As of yet, unfortunately, nothing has really helped speed up that first "7/16 Execution" phase.

Ultimately, it seems like the biggest factors are, in order: 1. memory capacity, throughput, and latency (and by extension L3 cache size and performance); 2. disk latency; and 3. lightly-threaded CPU performance. That's the best explanation for what I'm seeing on this particular hardware, which is limited on all three factors. In retrospect I should have spun up an archive node on a more capable system and then played around with forking a full node from it. Once my client is back up and validating I might revisit this.

EDIT: Welp, I upgraded the RAM to a much faster XMP CL16 kit, and it didn't help at all. So... 🤷‍♂️

[billyadelphia]

I left the idea of syncing from scratch, it's take too long. So I just downloaded blocks snapshoot from this repository https://github.com/BNB48Club/bsc-snapshots

[journeytosilius]

hi billy, does something like this exist for ETH ? Archive node snapshots for Erigon ?

[muyinliu]

Is there any way to speed up sync with an already synced node with same config on local network ?

[AskAlexSharov]

Too abstract question - we don’t know your bottleneck. Can disable something (—txpool.disable), can be more nice with erigon (https://www.linuxfordevices.com/tutorials/linux/nice-command-in-linux), can force warmup something after cold start of server (integration warmup —table=PlainState), can setup disk with lower latency, can give more ram, can add staticpeer, …

[muyinliu]

The bottleneck is the network speed, e.g. only 10MB/s. Is there any method to force a new node sync from the synced node and keep the synced node running? The local network's speed is 2.5 Gbps(320 MB/s, in this speed only takes 2 hour to sync a 2 TB node). Already try option --staticpeers/--trustedpeers of Erigon, the new node just can't sync from the synced node.

[thorstenhirsch]

Since I also tried the warmup to speed up my node performance - the correct command is a little bit different: integration warmup --bucket=PlainState. In a containerized environment you use this command in an init container that runs before the regular erigon container starts. Both containers need to mount the same storage for $DATADIR. In order to switch from the erigon executable (default) to the integration executable you need to set --entrypoint=integration.