-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reapply #8644 #9242
Open
aakselrod
wants to merge
14
commits into
lightningnetwork:master
Choose a base branch
from
aakselrod:reapply-8644
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+186
−151
Open
Reapply #8644 #9242
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
4a614e4
Reapply "kvdb/postgres: remove global application level lock"
aakselrod 76db02f
go.mod: use local kvdb to reapply removal of global postgres lock
aakselrod 4e7eb16
batch: handle serialization errors correctly
aakselrod 63525a0
channeldb: handle errors in `delChannelEdgeUnsafe()`
aakselrod ac494e3
channeldb: handle errors in `putChanEdgePolicy()`
aakselrod cd81377
sqldb: handle "in aborted tx" errors as serialization errors
aakselrod df3f608
sqldb: add "deadlock detected" to serialization errors
aakselrod a75a17a
lncfg: change default postgres maxconnections setting to 20
aakselrod fffb55d
Makefile: tune params for db-instance for postgres itests
aakselrod 6a1b28f
Makefile: log to file instead of console
aakselrod 306273f
github workflow: save postgres log to zip file
aakselrod fad1fa4
Makefile: update postgres to v17
aakselrod f3c8290
batch: don't batch requests in single tx for postgres
aakselrod 01f8aec
itest: fix flake in multi-hop payments
aakselrod File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| //go:build !kvdb_postgres | ||
| // +build !kvdb_postgres | ||
|
|
||
| package batch | ||
|
|
||
| import ( | ||
| "github.com/lightningnetwork/lnd/kvdb" | ||
| ) | ||
|
|
||
| // run executes the current batch of requests. If any individual requests fail | ||
| // alongside others they will be retried by the caller. | ||
| func (b *batch) run() { | ||
| // Clear the batch from its scheduler, ensuring that no new requests are | ||
| // added to this batch. | ||
| b.clear(b) | ||
|
|
||
| // If a cache lock was provided, hold it until the this method returns. | ||
| // This is critical for ensuring external consistency of the operation, | ||
| // so that caches don't get out of sync with the on disk state. | ||
| if b.locker != nil { | ||
| b.locker.Lock() | ||
| defer b.locker.Unlock() | ||
| } | ||
|
|
||
| // Apply the batch until a subset succeeds or all of them fail. Requests | ||
| // that fail will be retried individually. | ||
| for len(b.reqs) > 0 { | ||
| var failIdx = -1 | ||
| err := kvdb.Update(b.db, func(tx kvdb.RwTx) error { | ||
| for i, req := range b.reqs { | ||
| err := req.Update(tx) | ||
| if err != nil { | ||
| failIdx = i | ||
| return err | ||
| } | ||
| } | ||
| return nil | ||
| }, func() { | ||
| for _, req := range b.reqs { | ||
| if req.Reset != nil { | ||
| req.Reset() | ||
| } | ||
| } | ||
| }) | ||
|
|
||
| // If a request's Update failed, extract it and re-run the | ||
| // batch. The removed request will be retried individually by | ||
| // the caller. | ||
| if failIdx >= 0 { | ||
| req := b.reqs[failIdx] | ||
|
|
||
| // It's safe to shorten b.reqs here because the | ||
| // scheduler's batch no longer points to us. | ||
| b.reqs[failIdx] = b.reqs[len(b.reqs)-1] | ||
| b.reqs = b.reqs[:len(b.reqs)-1] | ||
|
|
||
| // Tell the submitter re-run it solo, continue with the | ||
| // rest of the batch. | ||
| req.errChan <- errSolo | ||
| continue | ||
| } | ||
|
|
||
| // None of the remaining requests failed, process the errors | ||
| // using each request's OnCommit closure and return the error | ||
| // to the requester. If no OnCommit closure is provided, simply | ||
| // return the error directly. | ||
| for _, req := range b.reqs { | ||
| if req.OnCommit != nil { | ||
| req.errChan <- req.OnCommit(err) | ||
| } else { | ||
| req.errChan <- err | ||
| } | ||
| } | ||
|
|
||
| return | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| //go:build kvdb_postgres | ||
| // +build kvdb_postgres | ||
|
|
||
| package batch | ||
|
|
||
| import ( | ||
| "github.com/lightningnetwork/lnd/kvdb" | ||
| ) | ||
|
|
||
| // run executes the current batch of requests. It avoids doing actual batching | ||
| // to prevent excessive serialization errors and deadlocks. | ||
| func (b *batch) run() { | ||
| // Clear the batch from its scheduler, ensuring that no new requests are | ||
| // added to this batch. | ||
| b.clear(b) | ||
|
|
||
| // If a cache lock was provided, hold it until the this method returns. | ||
| // This is critical for ensuring external consistency of the operation, | ||
| // so that caches don't get out of sync with the on disk state. | ||
| if b.locker != nil { | ||
| b.locker.Lock() | ||
| defer b.locker.Unlock() | ||
| } | ||
|
|
||
| // Apply each request in the batch in its own transaction. Requests that | ||
| // fail will be retried by the caller. | ||
| for _, req := range b.reqs { | ||
| err := kvdb.Update(b.db, func(tx kvdb.RwTx) error { | ||
| return req.Update(tx) | ||
| }, func() { | ||
| if req.Reset != nil { | ||
| req.Reset() | ||
| } | ||
| }) | ||
| switch { | ||
| case err != nil: | ||
| req.errChan <- errSolo | ||
| case req.OnCommit != nil: | ||
| req.errChan <- req.OnCommit(err) | ||
| default: | ||
| req.errChan <- nil | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively, we can also limit
sqlpackage params for lnd as well.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah this was just me setting it back to the original config. In the end, I've been running tests locally with the existing config plus the parameters suggested by @djkazic above. I did set the default max connections to 20 in LND. Will push cleaned up/latest code tomorrow.