Loss transactions in full node database

[tdnguyenND]

Problem

Some transactions are missing (not from latest block)

Closing criteria

Deploy full node successfully with no missing transactions

Problem details

When I deploy a full node cosmos, I found that some transactions are missing from my database (i.e 02940919FFC8E9A2B153654EA104AA041B603719C2A50A73F8A28BED1EAF109D - which I found exists in mintscan. For data analyzing purpose, I expect my database to have all the neccessary transaction.
So the question is how can I download the missing transaction?

How to reproduce this problems?

1. Install gaiad & cosmovisor. I'm using version 13.0.2
   
2. Download archieved data source from quicksync. At the moment I downloaded, it was 8.5TB, sadly I don't remember the latest block when downloading it
3. Unzip downloaded file and put in into the correct database folder
4. Start gaiad app
5. Run search for transaction

curl http://localhost:26657/tx?hash=0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

Expected result: Return transaction detail.
Actual result: {"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"tx (AA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD) not found"}}

In fact, all 3 transactions from block 7412749 are missed. Although they're all searchable at mintscan, I can't find it from any other explorer platform (maybe mintscan just simply decode from block data, i guess?)

[mmulji-ic]

Thanks @tdnguyenND for reporting, I'll check with the Comet team why this has occurred, they're supporting the db and p2p layers. I'm assuming that you have transactions before and after this transaction. Also how did you stumble across this transaction that was missing?

[andynog]

Hi @tdnguyenND, I'm Andy from the CometBFT team, in order to help @mmulji-ic troubleshoot this, could you please let us know how you're querying for this transaction using your full node ?

[tdnguyenND]

@andynog I have provide more detail in problem description. Hope it help

[andynog]

Thanks @tdnguyenND, what do you get when you query the block ?

curl http://localhost:26657/block?height=7412749

[tdnguyenND]

The block is still fine with all txns included
7412749.json

[mmulji-ic]

@andynog any further thoughts?

[andynog]

https://www.mintscan.io/cosmos/block/7412749 doesn't have the tx hash you mentioned 0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

[tdnguyenND]

https://www.mintscan.io/cosmos/block/7412749 doesn't have the tx hash you mentioned 0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

Maybe I was wrongly referring the command, but I'm sure that some transaction were somehow removed from quicksync's archieved file - i.e 02940919FFC8E9A2B153654EA104AA041B603719C2A50A73F8A28BED1EAF109D (I have a list of transactions that are missed)

Unfortunately this happened quite long and I decided to ignore them, my fullnode no longer exists so I can't recreate the scenario.

[a26nine]

I am seeing this as well. For example, 8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6 exisats on https://www.mintscan.io/cosmos/tx/8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6?height=15213800 , but the node gives : {"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"tx (8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6) not found"}}