Reapply #8644

[aakselrod]

Change Description

Fix #9229 by reapplying #8644 and

• correctly handling serialization errors in the batch package
• correctly handling previously-ignored errors in the channeldb package
• handling current transaction is aborted errors as serialization errors in case we hit a serialization error and ignore it, and get this error in a subsequent call to postgres
• tuning the db-instance postgres flags in Makefile per @djkazic's recommendations
• setting the maxconnections parameter for postgres DBs to 20 instead of 50 by default

Steps to Test

See the failing itests prior to the fix, and the passing itests after the fix.

Pull Request Checklist

Testing

• Your PR passes all CI checks.
• Tests covering the positive and negative (error paths) are included.
• Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

• The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
• Commits follow the Ideal Git Commit Structure.
• Any new logging statements use an appropriate subsystem and logging level.
• Any new lncli commands have appropriate tags in the comments for the rpc in the proto file.
• There is a change description in the release notes, or [skip ci] in the commit message for small changes.

📝 Please see our Contribution Guidelines for further guidance.

[coderabbitai]

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

• llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[aakselrod]

Waiting to push fix commit until CI completes for the reapplication.

[aakselrod]

Note the postgres itests fail mostly on opening, announcing, and closing channels. This is due to the use of the batch package to batch announcement update writes, while batch doesn't handle serialization errors correctly.

[aakselrod]

Looks like there are still a couple of itests failing. Will keep working on this next week.

[aakselrod]

The error message not enough elements in RWConflictPool to record a read/write conflict tells me that postgres might not be running with enough resources to handle the itests with 8644 reapplied.

[Roasbeef]

This looks relevant, re some of the errors I see in the latest CI run: https://stackoverflow.com/a/42303225

[Roasbeef]

Perhaps part of the issue is with the ON CONFLICT clause in many of the queries:

• ON CONFLICT Clause with Partial Indexes: Your use of ON CONFLICT with a partial unique index and a WHERE clause may not be matching the index as expected. This can lead to unexpected behavior in conflict resolution and contribute to serialization failures.
• Conflict Resolution: The ON CONFLICT clause relies on unique indexes or constraints to detect conflicts. If the clause doesn't perfectly match an existing unique index or constraint, PostgreSQL cannot efficiently perform conflict resolution, leading to increased chances of serialization failures.
• Match ON CONFLICT Clause to Unique Index: Ensure that the ON CONFLICT clause exactly matches the unique index or constraint without additional WHERE clauses that might prevent PostgreSQL from recognizing the conflict.

Based on the SO link above, we might also be lacking some needed indexes.

[aakselrod]

With closing the channel and a couple of other tests, I'm seeing logs similar to:

2024-11-01 22:13:15.503 [ERR] GRPH builder.go:1020: unable to prune routing table: unknown postgres error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
2024-11-01 22:13:15.504 [ERR] GRPH builder.go:849: unable to prune graph with closed channels: unknown postgres error: ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)

when I reproduce locally, as well as in the CI logs. I'm going to pull on that thread first...

On the test config side, also seeing these:

2024-11-02 04:28:59.361 UTC [19976] ERROR:  out of shared memory
2024-11-02 04:28:59.361 UTC [19976] HINT:  You might need to increase max_pred_locks_per_transaction.

I think the first issue above is with the code, the second is a config issue, and the other config issue in my comment above are the three major failures still happening. I think the ON CONFLICT/index audit is also a good idea to ensure we minimize serialization errors, but I'd like to solve at least the code issue with the graph pruning first.

[Roasbeef]

2024-11-01 22:13:15.504 [ERR] GRPH builder.go:849: unable to prune graph with closed channels

This looks like a case where we continue when we get an error, instead of checking it for a serialization error, and returning if it is.

[aakselrod]

This looks like a case where we continue when we get an error, instead of checking it for a serialization error, and returning if it is.

Yep, looking into why that isn't caught by the panic/recover mechanism.

[aakselrod]

It was actually a lack of error checking in delChannelEdgeUnsafe() when calling updateEdgePolicyDisabledIndex(). This caused the serialization error to be ignored, and the next error is the one I pasted above. Submitting a fix commit below.

[aakselrod]

Looks better as far as the errors on closing channels. Will keep working tomorrow to eliminate the other errors.

[Roasbeef]

2024-11-05 03:38:04.408 UTC [14305] ERROR: out of shared memory
2024-11-05 03:38:04.408 UTC [14305] HINT: You might need to increase max_pred_locks_per_transaction.

Hmm, so we don't have great visibility into how much memory these CI machines have. Perhaps we need to modify the connection settings to reduce the number of active connections, and also tune params like max_pred_locks_per_transaction.

@djkazic has been working on a postgres+lnd tuning/perf guide, that I think we can eventually check directly into lnd.

[Roasbeef]

This is also very funky:

lnd/kvdb/sqlbase/readwrite_bucket.go

Lines 336 to 363 in e3cc4d7

	// Check to see if a bucket with this key exists.
	var dummy int
	row, cancel := b.tx.QueryRow(
	"SELECT 1 FROM "+b.table+" WHERE "+parentSelector(b.id)+
	" AND key=$1 AND value IS NULL", key,
	)
	defer cancel()
	err := row.Scan(&dummy)
	switch {
	// No bucket exists, proceed to deletion of the key.
	case err == sql.ErrNoRows:
	case err != nil:
	return err
	// Bucket exists.
	default:
	return walletdb.ErrIncompatibleValue
	}
	_, err = b.tx.Exec(
	"DELETE FROM "+b.table+" WHERE key=$1 AND "+
	parentSelector(b.id)+" AND value IS NOT NULL",
	key,
	)
	if err != nil {
	return err
	}

We do two queries to just delete: select to see if exists, then delete. Instead of just trying to delete.

Stepping back a minute: perhaps the issue is with this flawed KV abstraction we have. Perhaps we should just re-create a better hierarchical KV table from scratch. We use sqlc elsewhere so we can gain by having a unified set of light abstractions over what we want to do on the SQL layer.

[Roasbeef]

Here's another instance of duplicated work in CreateBucket:

lnd/kvdb/sqlbase/readwrite_bucket.go

Lines 149 to 187 in e3cc4d7

	// Check to see if the bucket already exists.
	var (
	value *[]byte
	id int64
	)
	row, cancel := b.tx.QueryRow(
	"SELECT id,value FROM "+b.table+" WHERE "+parentSelector(b.id)+
	" AND key=$1", key,
	)
	defer cancel()
	err := row.Scan(&id, &value)
	switch {
	case err == sql.ErrNoRows:
	case err == nil && value == nil:
	return nil, walletdb.ErrBucketExists
	case err == nil && value != nil:
	return nil, walletdb.ErrIncompatibleValue
	case err != nil:
	return nil, err
	}
	// Bucket does not yet exist, so create it. Postgres will generate a
	// bucket id for the new bucket.
	row, cancel = b.tx.QueryRow(
	"INSERT INTO "+b.table+" (parent_id, key) "+
	"VALUES($1, $2) RETURNING id", b.id, key,
	)
	defer cancel()
	err = row.Scan(&id)
	if err != nil {
	return nil, err
	}
	return newReadWriteBucket(b.tx, &id), nil
	}

We select to see if it exists, then potentially do the insert again. Instead, we can just do an UPSERT, then use RETURNING to give us the bucket id and key we care about, so a single query.

[Roasbeef]

I think the way the sequence is implemented may also be problematic: we have the sequence field directly in the table, which means table locks may need to be held. The sequence gets incremented a lot for stuff like payments, or invoice. We may be able to instead split that out into another table that can be updated independently of the main table:

lnd/kvdb/sqlbase/readwrite_bucket.go

Lines 412 to 437 in e3cc4d7

	// Sequence returns the current sequence number for this bucket without
	// incrementing it.
	func (b *readWriteBucket) Sequence() uint64 {
	if b.id == nil {
	panic("sequence not supported on top level bucket")
	}
	var seq int64
	row, cancel := b.tx.QueryRow(
	"SELECT sequence FROM "+b.table+" WHERE id=$1 "+
	"AND sequence IS NOT NULL",
	b.id,
	)
	defer cancel()
	err := row.Scan(&seq)
	switch {
	case err == sql.ErrNoRows:
	return 0
	case err != nil:
	panic(err)
	}
	return uint64(seq)
	}

[aakselrod]

I've been able to reduce (but not fully eliminate) the out of shared memory and not enough elements in RWConflictPool errors locally by changing the -N parameter of the lnd-postgres container to 200, and changing the default maxconnections value in lnd to 20. This follows from this comment about how the RWConflictPool is allocated.

I've also tried treating these errors and current transaction is aborted as serialization errors, since they generally happen when too many transactions are conflicting, and that seemed to reduce the number of test failures.

In addition, I've found one more place where we get the current transaction is aborted errors due to lack of error handling, and added error handling there.

I pushed these changes above for discussion. My next step is to try to reduce the number of conflicts based on @Roasbeef's suggestions above. I'm going on vacation for the rest of the week until next Tuesday, so will keep working on this then.

[aakselrod]

I think treating the OOM errors as serialization errors ended up being a mistake. Going to take that out and push when this run is done. In addition, I'm trying doubling the max_pred_locks_per_transaction value from the default (64->128).

[aakselrod]

Got some errors like:

ERROR:  could not access status of transaction
DETAIL:  Could not open file "pg_serial/0002": No such file or directory.
...
LOG:  could not truncate directory "pg_serial": apparent wraparound

Looking into those, they could also be from my change of treating the OOM errors as retryable.

[aakselrod]

Set -N 500 again while leaving the default per-node max connections at 20. This seems to improve things. Now starting work on reducing the number of actual serialization errors per suggestions above, rather than just optimizing settings/fixing our responses to them.

[aakselrod]

Only 1 test failure last run, looks like no instances of RWConflictPool or out of shared memory but one cluster of pg_serial right after the failure. Looking more closely into this.

[djkazic]

I think treating the OOM errors as serialization errors ended up being a mistake. Going to take that out and push when this run is done. In addition, I'm trying doubling the max_pred_locks_per_transaction value from the default (64->128).

I'm running with

max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024
jit = off
checkpoint_timeout = 10min
work_mem = 8MB

[Roasbeef]

Alternatively, we can also limit sql package params for lnd as well.

[aakselrod]

Ah this was just me setting it back to the original config. In the end, I've been running tests locally with the existing config plus the parameters suggested by @djkazic above. I did set the default max connections to 20 in LND. Will push cleaned up/latest code tomorrow.

[aakselrod]

Looking at the last failure, I think it's a flake that's unrelated to the DB (we encountered it in our tests often before I added a hacky fix locally, it's due to an issue in txnotifier). When I run locally, I get some test framework related issues restarting nodes that I think I have a fix for, but I'll finish writing it (this seems to have been caused by my local system, and rebooting seems to have fixed it) and have the PR cleaned up tomorrow. Otherwise, I think it's generally passing tests despite the serialization failures.

[aakselrod]

I'm still getting excessive serialization errors causing failures. Looks like that's hopefully the last place to fix. I'm trying out some schema/query changes, and it's likely to be pretty different than the existing schema. I'm going to try to do it all in the kvdb/sqlbase for the purpose of testing the changes for now. If/when I can get it to work better in itests, we can talk about migration support for existing nodes.

[aakselrod]

I have a somewhat incomplete attempt at rebuilding the KV SQL but haven't fixed the failures. I have a couple of avenues to explore next.

[aakselrod]

Pushed last update for the week:

• Updated postgres from v13 to v17
• Stopped batching multiple requests per tx in postgres in the batch package, and instead use 1 tx per request
• Removed the SQL schema/query changes for now as they don't seem to make much difference

I've had better results with those, but am bottlenecked by how long it takes the itests to run in terms of testing which changes are making the most difference.

Tests are still not fully passing. In particular, this failure is also a fairly consistent failure for me locally, so I'll look into that next week.

--- FAIL: TestLightningNetworkDaemon (400.71s)
    harness_setup.go:25: Setting up HarnessTest...
    harness_setup.go:38: Prepare the miner and mine blocks to activate segwit...
    harness_setup.go:46: Connecting the miner at 127.0.0.1:10023 with the chain backend...
    harness.go:317: Setting up standby nodes Alice and Bob...
    harness.go:356: Finished the setup, now running tests...
    --- FAIL: TestLightningNetworkDaemon/tranche03/68-of-174/bitcoind/multi-hop_payments (70.05s)
        harness_assertion.go:2379: 
            	Error Trace:	/home/runner/work/lnd/lnd/lntest/harness_assertion.go:2379
            	            				/home/runner/work/lnd/lnd/itest/lnd_multi-hop-payments_test.go:227
            	            				/home/runner/work/lnd/lnd/lntest/harness.go:392
            	            				/home/runner/work/lnd/lnd/itest/lnd_test.go:139
            	Error:      	Not equal: 
            	            	expected: 5
            	            	actual  : 4
            	Test:       	TestLightningNetworkDaemon/tranche03/68-of-174/bitcoind/multi-hop_payments
            	Messages:   	num of forwards mismatch
        harness.go:448: finished test: multi-hop_payments, start height=483, end height=503, mined blocks=20
        harness.go:454: test failed, skipped cleanup
    lnd_test.go:147: Failure time: 2024-11-16 00:46:35.052
    lnd_test.go:155: =========> tests finished for tranche: 3, tested 21 cases, end height: 483
FAIL

[aakselrod]

Looking into the failure above, I can reproduce it every few times running the test by itself locally with the postgres backend. It looks like we get 6 settle and 4 forward events rather than 5 forward and 5 settle. From a first glance, it looks like we're reusing/overwriting some parts of a struct incorrectly, but I'm narrowing down what/where.