-
Notifications
You must be signed in to change notification settings - Fork 649
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
after restart, bbolt db failed to get all reachable pages #778
Comments
Did the machine/VM restarted? Do you have a detailed steps to reproduce this issue by any chance?
Yes, it would be helpful if you could provide the db file.
We have surgery commands, but again we need to the db file firstly. |
This seems like also a potential issue of the freelist management, e.g a page was allocated twice, accordingly it's referenced by multiple places. |
I could not get an approval to provide the db file. sorry for the trouble. |
Hi @timshannon just want to confirm that the issue show up after the host or vm where the process run has been restarted? Thanks |
@fuweid I assume you meant to call out @timannguyen not @timshannon right? |
Right, sorry for the confusion. |
sorry @timshannon for the confusion. I should ping @timannguyen |
@fuweid sorry for the delay, the process was restarted and the issue occurs. |
Thanks for the feedback but the questions isn't answered. The question is " Also please also provide the following info,
Could you dump the db with all the value redacted?
|
|
My machine met unexpected power down, then bbolt db say:
Here is my db file: |
Thanks. I will take a look later. |
The last transaction before the unexpected poweroff somehow corrupted the db. One possible reason is some page (i.e. 1079) wasn't successfully synced to disk even the The workaround to fix the corrupted db is to rollback the last transaction,
Reference
|
@timshannon @lnnt Could you also share your systeminfo using https://github.com/ahrtr/systeminfo? |
cpu.IsBigEndian: false |
As mentioned in previous comment, the last transaction (10789638) before the unexpected poweroff somehow corrupted the db. After rolling back the last transaction, the db is back to normal again. The max revision under the last good state is
The
The problem is the leaf page 4737 and root page 5150 were persisted successfully, but some branch pages were not. As far as I can tell, most likely it's the file system's issue.
As a followup, it's helpful if we can figure out the missing intermediate branch page between |
@ahrtr 2 questions:
|
We already introduced dmflakey, which is similar to LazyFS, and without the possible downside of LazyFS, which (LazyFS) may not respect the fdatasync semantics. Please refer to fuweid/go-dmflakey#1
Yes, It's kind of good point. I already thought about it, but not automatically done by bbolt. Based on the data corruption cases in the past around 3 years, IIRC, only minor parts could be fixed by It also has downside... there may be data loss after reverting to previous TXN. |
Looks like issue is closed. Do you mean #568? Repeating the ask, could we cover this exact scenario using dmflakey?
Just asking for this to be considered, because I would want to better understand the risk of revert. I think high level question is, is there a chance of previous transaction to be silently broken if the latest transaction was corrupted? Don't think this needs to be a default option, might just be a feature in etcd as I think it would benefit it's ability to recover. For systems like etcd, bbolt is not the source of truth, just the kv database. We can still recover the last transaction from WAL. For cases where only the latest transaction was corrupted, instead of etcd giving up the bootstrap, we would just revert it and replay the transaction again from the WAL. If the last transaction can be trusted I would propose that etcd should always try to revert meta page if the state of btree on the latest meta page is broken. But before implementing this we should verify and test health of previous transaction under different corruption scenarios. |
After restarting the process running with open telemetry file extension, we get
I'll see if i can get the db pulled from the running machine. Sorry for no db blob.
is there any means of recovery from this issue ?
The text was updated successfully, but these errors were encountered: