Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fulcrum sync with BTC Core slows to a crawl then times out on Windows 11 PC with ext SSD #177

Open
edjanx opened this issue Jun 18, 2023 · 0 comments

Comments

@edjanx
Copy link

edjanx commented Jun 18, 2023

I am having terrible difficulty syncing the Fulcrum DB to Bitcoin Core. I tired several configurations, both with and without fastsync which provides memory issues. I am using a old HP PC upgraded to 16 GB memory, and a 2TB external SSD which stores blockchain data as well as folder for the fulcrum DB, with the programs for both are installed on the PC internal SSD drive. When it does sync, once it gets into the 40% range it only adds .1% about every 4 hours, which equates to a full sync in a few months! After starting over with several configuration tweaks based on reading docs and troubleshooting posts, I doubt this is expected behavior, so I gave in and decided to ask for assistance. Thanks in advance!

Config:
datadir = f:\FulcrumDB
bitcoind = 127.0.0.1:8332
rpcuser = ****
rpcpassword = ****
tcp = 127.0.0.1:50001
peering = false
announce = false
public_tcp_port = 50001
admin = 8000
stats = 8080
db_max_open_files = 200
bitcoind_timeout = 600
MALLOC_CONF = tcache:false

BTC Core conf:
server=1
txindex=1
rpcbind=127.0.0.1
rpcallow=127.0.0.1
rpcallowip=127.0.0.1
rpcuser=****
rpcpassword=****
zmqpubhashblock=tcp://127.0.0.1:8433

current effort:
C:\Fulcrum>"c:\Fulcrum\Fulcrum.exe" c:\Fulcrum\Fulcrum.conf
[2023-06-15 09:48:30.959] Enabled JSON parser: simdjson
[2023-06-15 09:48:30.959] simdjson implementations:
[2023-06-15 09:48:30.959] haswell: Intel/AMD AVX2 [not supported]
[2023-06-15 09:48:30.959] westmere: Intel/AMD SSE4.2 [supported]
[2023-06-15 09:48:30.959] fallback: Generic fallback implementation [supported]
[2023-06-15 09:48:30.959] active implementation: westmere
[2023-06-15 09:48:30.959] jemalloc: version 5.2.1-0-gea6b3e9
[2023-06-15 09:48:30.959] Qt: version 5.15.2
[2023-06-15 09:48:30.959] rocksdb: version 6.14.6-@@
[2023-06-15 09:48:30.959] simdjson: version 0.6.0
[2023-06-15 09:48:30.959] ssl: OpenSSL 3.0.1 14 Dec 2021
[2023-06-15 09:48:30.959] zmq: libzmq version: 4.3.3, cppzmq version: 4.7.1
[2023-06-15 09:48:30.959] Fulcrum 1.9.1 (Release 713d2d7) - Thu Jun 15, 2023 09:48:30.959 Eastern Daylight Time - starting up ...
[2023-06-15 09:48:30.959] Loading database ...
[2023-06-15 09:48:39.362] DB memory: 512.00 MiB
[2023-06-15 09:48:39.362] Verifying headers ...
[2023-06-15 09:48:39.362] DB version: v2
[2023-06-15 09:48:39.362] BitcoinDMgr: starting 3 bitcoin RPC clients ...
[2023-06-15 09:48:39.362] BitcoinDMgr: started ok
[2023-06-15 09:48:39.362] Stats HTTP: starting 1 server ...
[2023-06-15 09:48:39.362] Starting listener service for HttpSrv 127.0.0.1:8080 ...
[2023-06-15 09:48:39.378] Service started, listening for connections on 127.0.0.1:8080
[2023-06-15 09:48:39.394] Coin: BTC
[2023-06-15 09:48:39.489] Chain: main
[2023-06-15 09:48:39.489] Block height 794467, downloading new blocks ...
[2023-06-15 09:48:39.489] fast-sync: Not enabled
[2023-06-15 09:48:41.655] Processed height: 1000, 0.1%, 460.7 blocks/sec, 469.4 txs/sec, 481.4 addrs/sec
[2023-06-15 09:48:43.383] Processed height: 2000, 0.3%, 580.7 blocks/sec, 587.1 txs/sec, 633.0 addrs/sec
[2023-06-15 09:48:46.235] Processed height: 3000, 0.4%, 351.5 blocks/sec, 358.5 txs/sec, 372.2 addrs/sec
[2023-06-15 09:48:48.014] Processed height: 4000, 0.5%, 558.7 blocks/sec, 563.1 txs/sec, 608.9 addrs/sec
[2023-06-15 09:48:48.768] Processed height: 5000, 0.6%, 1322.8 blocks/sec, 1329.4 txs/sec, 1407.4 addrs/sec[2023-06-15 09:48:50.832] Processed height: 6000, 0.8%, 487.6 blocks/sec, 490.5 txs/sec, 516.3 addrs/sec
[2023-06-15 09:48:53.515] Processed height: 7000, 0.9%, 371.6 blocks/sec, 374.6 txs/sec, 399.1 addrs/sec
[2023-06-15 09:48:56.527] Processed height: 8000, 1.0%, 332.9 blocks/sec, 334.6 txs/sec, 360.2 addrs/sec
[2023-06-15 09:48:59.437] Processed height: 9000, 1.1%, 342.6 blocks/sec, 344.0 txs/sec, 361.1 addrs/sec
[2023-06-15 09:49:02.624] Processed height: 10000, 1.3%, 314.6 blocks/sec, 316.5 txs/sec, 337.2 addrs/sec
SNIP
[2023-06-16 05:00:00.203] Processed height: 348000, 43.8%, 1.03 blocks/sec, 696.8 txs/sec, 2874.0 addrs/sec
[2023-06-16 05:12:57.525] Processed height: 349000, 43.9%, 1.29 blocks/sec, 880.3 txs/sec, 3489.2 addrs/sec
[2023-06-16 05:33:22.094] Processed height: 350000, 44.1%, 49.0 blocks/min, 517.8 txs/sec, 2063.2 addrs/sec
[2023-06-16 06:27:04.235] Processed height: 351000, 44.2%, 18.6 blocks/min, 224.1 txs/sec, 869.4 addrs/sec
[2023-06-16 09:23:48.282] Processed height: 352000, 44.3%, 5.66 blocks/min, 72.1 txs/sec, 278.1 addrs/sec
[2023-06-16 12:19:31.813] Processed height: 353000, 44.4%, 5.69 blocks/min, 70.2 txs/sec, 262.9 addrs/sec
[2023-06-16 15:40:10.457] Processed height: 354000, 44.6%, 4.98 blocks/min, 62.1 txs/sec, 241.8 addrs/sec
[2023-06-16 19:21:40.062] Processed height: 355000, 44.7%, 4.51 blocks/min, 55.5 txs/sec, 214.1 addrs/sec
[2023-06-16 20:42:46.754] <Task.DL 1 -> 794467> 722917: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 0 -> 794467> 722920: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 1 -> 794467> 722923: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 1 -> 794467> 722927: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 1 -> 794467> 722918: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 2 -> 794467> 722922: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 0 -> 794467> 722924: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 2 -> 794467> 722925: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 0 -> 794467> 722926: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 2 -> 794467> 722928: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 0 -> 794467> 722921: FAIL: bitcoind request timed out
[2023-06-16 20:42:54.260] <Task.DL 2 -> 794467> 722919: FAIL: bitcoind request timed out
[2023-06-16 21:14:34.238] Task errored: Task.DL 1 -> 794467, error: bitcoind request timed out
[2023-06-16 21:14:34.238] Failed to synch blocks and/or mempool
[2023-06-16 21:14:36.385] Block height 794691, downloading new blocks ...
[2023-06-16 21:14:36.385] fast-sync: Not enabled
[2023-06-16 22:38:03.099] Processed height: 356000, 44.8%, 4.91 blocks/min, 58.3 txs/sec, 242.3 addrs/sec
[2023-06-17 01:25:31.934] Processed height: 357000, 44.9%, 5.97 blocks/min, 66.0 txs/sec, 268.3 addrs/sec
[2023-06-17 03:34:24.642] Processed height: 358000, 45.0%, 7.76 blocks/min, 92.9 txs/sec, 376.5 addrs/sec
[2023-06-17 05:39:31.552] Processed height: 359000, 45.2%, 7.99 blocks/min, 109.2 txs/sec, 427.1 addrs/sec
[2023-06-17 08:21:00.499] Processed height: 360000, 45.3%, 6.19 blocks/min, 75.1 txs/sec, 300.8 addrs/sec
[2023-06-17 12:41:04.545] Processed height: 361000, 45.4%, 3.85 blocks/min, 49.8 txs/sec, 197.1 addrs/sec
[2023-06-17 18:26:44.153] Processed height: 362000, 45.6%, 2.89 blocks/min, 40.7 txs/sec, 158.8 addrs/sec
[2023-06-17 23:11:11.167] Processed height: 363000, 45.7%, 3.52 blocks/min, 45.6 txs/sec, 177.5 addrs/sec
[2023-06-18 01:35:01.212] Processed height: 364000, 45.8%, 6.95 blocks/min, 103.4 txs/sec, 378.8 addrs/sec
[2023-06-18 05:58:23.392] Processed height: 365000, 45.9%, 3.80 blocks/min, 80.0 txs/sec, 211.3 addrs/sec
[2023-06-18 08:41:06.197] <Task.DL 355591 -> 794691> 768278: FAIL: bitcoind request timed out
[2023-06-18 08:41:58.706] <Task.DL 355593 -> 794691> 768290: FAIL: bitcoind request timed out
[2023-06-18 08:42:43.763] <Task.DL 355592 -> 794691> 768308: FAIL: bitcoind request timed out
[2023-06-18 08:43:13.780] <Task.DL 355593 -> 794691> 768317: FAIL: bitcoind request timed out
[2023-06-18 08:43:43.805] <Task.DL 355591 -> 794691> 768327: FAIL: bitcoind request timed out
[2023-06-18 08:44:06.300] <Task.DL 355592 -> 794691> 768334: FAIL: bitcoind request timed out

Previous results with fastsync:
[2023-06-12 04:01:06.547] <BitcoinD.1> 503 (header): Date: Mon, 12 Jun 2023 08:01:06 GMT
[2023-06-12 04:01:06.547] <BitcoinD.1> 503 (header): Content-Length: 25
[2023-06-12 04:01:06.547] <BitcoinD.1> 503 (header): Content-Type: text/html; charset=ISO-8859-1
[2023-06-12 04:01:06.547] <BitcoinD.1> Got unexpected content type: text/html; charset=ISO-8859-1; will log the rest of this HTTP response
[2023-06-12 04:01:06.547] <BitcoinD.1> 503 (header):
[2023-06-12 04:01:06.547] <BitcoinD.1> 503 (content): Work queue depth exceeded
[2023-06-12 04:01:06.547] <BitcoinD.1> Error processing data coming in: Failed to parse Json from string: Work queue depth exceeded
[2023-06-12 04:01:06.547] <BitcoinD.1> Lost connection to bitcoind, will retry every 5 seconds ...
[2023-06-12 04:01:11.555] <BitcoinD.2> Got HTTP status 503 Service Unavailable; will log the rest of this HTTP response
[2023-06-12 04:01:11.555] <BitcoinD.2> 503 (header): Date: Mon, 12 Jun 2023 08:01:11 GMT
[2023-06-12 04:01:11.555] <BitcoinD.2> 503 (header): Content-Length: 25
[2023-06-12 04:01:11.555] <BitcoinD.2> 503 (header): Content-Type: text/html; charset=ISO-8859-1
[2023-06-12 04:01:11.555] <BitcoinD.2> Got unexpected content type: text/html; charset=ISO-8859-1; will log the rest of this HTTP response
[2023-06-12 04:01:11.555] <BitcoinD.2> 503 (header):
[2023-06-12 04:01:11.555] <BitcoinD.2> 503 (content): Work queue depth exceeded
[2023-06-12 04:01:11.555] <BitcoinD.2> Error processing data coming in: Failed to parse Json from string: Work queue depth exceeded
[2023-06-12 04:01:11.555] <BitcoinD.3> Got HTTP status 503 Service Unavailable; will log the rest of this HTTP response
[2023-06-12 04:01:11.555] <BitcoinD.3> 503 (header): Date: Mon, 12 Jun 2023 08:01:11 GMT
[2023-06-12 04:01:11.555] <BitcoinD.3> 503 (header): Content-Length: 25
[2023-06-12 04:01:11.555] <BitcoinD.3> 503 (header): Content-Type: text/html; charset=ISO-8859-1
[2023-06-12 04:01:11.555] <BitcoinD.3> Got unexpected content type: text/html; charset=ISO-8859-1; will log the rest of this HTTP response
[2023-06-12 04:01:11.555] <BitcoinD.2> Lost connection to bitcoind, will retry every 5 seconds ...
[2023-06-12 04:01:11.555] <BitcoinD.3> 503 (header):
[2023-06-12 04:01:11.555] <BitcoinD.3> 503 (content): Work queue depth exceeded
[2023-06-12 04:01:11.555] <BitcoinD.3> Error processing data coming in: Failed to parse Json from string: Work queue depth exceeded
[2023-06-12 04:01:11.555] <BitcoinD.3> Lost connection to bitcoind, will retry every 5 seconds ...
[2023-06-12 04:01:11.555] <BitcoinD.1> Got HTTP status 503 Service Unavailable; will log the rest of this HTTP response
[2023-06-12 04:01:11.555] <BitcoinD.1> 503 (header): Date: Mon, 12 Jun 2023 08:01:11 GMT
[2023-06-12 04:01:11.555] <BitcoinD.1> 503 (header): Content-Length: 25
[2023-06-12 04:01:11.555] <BitcoinD.1> 503 (header): Content-Type: text/html; charset=ISO-8859-1
[2023-06-12 04:01:11.555] <BitcoinD.1> Got unexpected content type: text/html; charset=ISO-8859-1; will log the rest of this HTTP response
[2023-06-12 04:01:11.555] <BitcoinD.1> 503 (header):
SNIP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
@edjanx and others