Stop shutting down on intermittent issues

[tonyimpervious]

I am trying to get ION working again with a recent ION and IPFS version and it is going through some complete refresh. I'm not sure if it's from some change or from the syncing/state issues that I've constantly run into.

But one thing is obvious to me is that it should really stop shutting down so often. It's hammering my local bitcoind testnet instance and if it doesn't respond immediately it crashes. This should not happen.

 reply: { n: 1, ok: 1 }
}
Sending jRPC request: id: 4v6sfm8qn2j, method: getrawtransaction
Sending jRPC request: id: 28lfj7o8cp4, method: getrawtransaction
Sending jRPC request: id: 5k1rhce96vv, method: getrawtransaction
Sidetree transaction found; adding {"transactionNumber":2104811000008,"transactionTime":2104811,"transactionTimeHash":"0000000000000016237972af7fbc895552df143c10df89d901bc13206b774c80","anchorString":"4.QmVzG9v3HkGCRsdgjcJdcWyEk6g7WmT5fQ1QjMcPoh4iem","transactionFeePaid":1340,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}
CommandSucceededEvent {
  connectionId: 'localhost:27017',
  requestId: 13104,
  commandName: 'insert',
  duration: 0,
  reply: { n: 1, ok: 1 }
}
Sending jRPC request: id: 157et486iok, method: getrawtransaction
Sending jRPC request: id: 440khjqooit, method: getrawtransaction
Sending jRPC request: id: 6r5skvjr12, method: getrawtransaction
Attempt 0 timed-out.
Retrying attempt count: 1 with request timeout of 600 ms...
Attempt 1 timed-out.
Retrying attempt count: 2 with request timeout of 1200 ms...
Attempt 2 timed-out.
Retrying attempt count: 3 with request timeout of 2400 ms...
Sidetree transaction found; adding {"transactionNumber":2104812000007,"transactionTime":2104812,"transactionTimeHash":"00000000000000231f3cfb05ca1ee4e11ccbd5b3bf401ff0cd17860d862baad2","anchorString":"2.QmckdA53ePxJNmj2RjtnihGtJtAM3GQFZEkGLsbZDRWbtK","transactionFeePaid":1340,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}
CommandSucceededEvent {
  connectionId: 'localhost:27017',
  requestId: 13105,
  commandName: 'insert',
  duration: 1,
  reply: { n: 1, ok: 1 }
}
Sending jRPC request: id: 60j55892l35, method: getrawtransaction
Attempt 0 timed-out.
Retrying attempt count: 1 with request timeout of 600 ms...
Attempt 1 timed-out.
Retrying attempt count: 2 with request timeout of 1200 ms...
Attempt 2 timed-out.
Retrying attempt count: 3 with request timeout of 2400 ms...
Attempt 3 timed-out.
Max retries reached without success.
Error while trying to get outputIdx: 1 from transaction: 80583b1ae2b143c5231e15c1c128de652c246d43553f35a1a7d26affc296dd64. Error: {"message":"network timeout at: http://localhost:18332/","type":"request-timeout","stack":"FetchError: network timeout at: http://localhost:18332/\n    at Timeout.<anonymous> (/Users/imp/Dev/Bitcoin/ion/ion/node_modules/node-fetch/lib/index.js:1484:13)\n    at listOnTimeout (node:internal/timers:557:17)\n    at processTimers (node:internal/timers:500:7)"}
An error happened when trying to add sidetree transaction to the store. Moving on to the next transaction. Inputs: {"blockHeight":2104812,"blockHash":"00000000000000231f3cfb05ca1ee4e11ccbd5b3bf401ff0cd17860d862baad2","transactionIndex":8}
Full error: {"message":"network timeout at: http://localhost:18332/","type":"request-timeout","stack":"FetchError: network timeout at: http://localhost:18332/\n    at Timeout.<anonymous> (/Users/imp/Dev/Bitcoin/ion/ion/node_modules/node-fetch/lib/index.js:1484:13)\n    at listOnTimeout (node:internal/timers:557:17)\n    at processTimers (node:internal/timers:500:7)"}
Sidetree-Bitcoin node initialization failed with error: {"message":"network timeout at: http://localhost:18332/","type":"request-timeout","stack":"FetchError: network timeout at: http://localhost:18332/\n    at Timeout.<anonymous> (/Users/imp/Dev/Bitcoin/ion/ion/node_modules/node-fetch/lib/index.js:1484:13)\n    at listOnTimeout (node:internal/timers:557:17)\n    at processTimers (node:internal/timers:500:7)"}
❯ 
   

Furthermore, there's no way for me to get past this. It gets to block dat 202 or so and then just keeps crashing and then restarts from the top. Why does it not save state?

[thehenrytsai]

A couple of questions for understanding the full context:

1. Are you using "fast-initialization" (bitcoinDataDirectory param)? Sounds like you are as you mentioned "block dat 202".
2. Are using the docker-compose setup or the full manual setup guide?

In theory, ~5 seconds total time for an jPRC call to bitcoind should be sufficient, but if the bitcoind is consistently having trouble responding, you can try setting the following config params in mainnet/testnet-bitcoin-config.json:

requestTimeoutInMilliseconds and requestMaxRetries

[tonyimpervious]

Are you using "fast-initialization" (bitcoinDataDirectory param)? Sounds like you are as you mentioned "block dat 202".

Yes. It always stopped on the same transaction. I have since removed that fast initialization option and it proceeded to work, slowly.

Are using the docker-compose setup or the full manual setup guide?

Manual setup

requestTimeoutInMilliseconds and requestMaxRetries

I don't see those options listed anywhere? Are you referencing this? https://github.com/decentralized-identity/ion/blob/master/json/mainnet-bitcoin-config.json

[thehenrytsai]

Yes. It always stopped on the same transaction. I have since removed that fast initialization option and it proceeded to work, slowly.

@tonyimpervious, sounds like there is probably a bug in fast-init path that parses blocks files directly, but unfortunately it is dependent on the specific block files as the download order of blocks is different in each machine. If you are able to share those files from the end blocks to the file that causes the issue (parsing is in reverse order) that would help diagnose the issue, though I understand if it's too much data.

I don't see those options listed anywhere? Are you referencing this?

Yes, they are not in the default config because there was no need to change them. But I just create an issue for documenting all config parameters.