feat(clustering): introduce incremental sync for clustering #13157
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
core/proxy
core/db/migrations
core/clustering
core/templates
core/db
core/configuration
schema-change-noteworthy
cherry-pick kong-ee
dndx
force-pushed
the
feat/incremental
branch
3 times, most recently
from
88e177a to
e64c085
Compare
chronolaw
force-pushed
the
feat/incremental
branch
from
e64c085 to
2692ea9
Compare
dndx
force-pushed
the
feat/incremental
branch
2 times, most recently
from
856ca04 to
50528fd
Compare
3 tasks
dndx
force-pushed
the
feat/incremental
branch
from
dbe6259 to
573852d
Compare
chronolaw
force-pushed
the
feat/incremental
branch
2 times, most recently
from
a558f38 to
b11e659
Compare
dndx
commented
Sep 26, 2024
chronolaw
changed the title
chronolaw
changed the title
dndx
commented
Sep 27, 2024
chronolaw
force-pushed
the
feat/incremental
branch
from
bebd9ed to
68849c4
Compare
This reverts commit 95f5cc6.
* fix(tests): enable inc sync for 01-sync_spec.lua * fix(incremental): dont delete old entity from LMDB if we clear it already * dont modify crud type even though we dont delete old entity from lmdb * localize ns_delta.wipe * localize ns_delta.wipe - part 2 * fix that cp will not init clustering * refactor old_entity logic * cp should always support full sync protocol * Revert "cp should always support full sync protocol" This reverts commit 6803958. * Revert "fix that cp will not init clustering" This reverts commit b84f261. * Revert "fix(tests): enable inc sync for 01-sync_spec.lua" This reverts commit 54d0f54. --------- Co-authored-by: chronolaw <[email protected]>
This reverts commit adcbbf0.
dndx
force-pushed
the
feat/incremental
branch
from
0df05a4 to
7c02759
Compare
|
Cherry-pick failed for Please cherry-pick the changes locally. git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13157-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13157-to-master-to-upstream
git checkout -b cherry-pick-13157-to-master-to-upstream
ancref=$(git merge-base 9ae36382db4b9485f8889b8cd80d66a0885c86e7 7c027596914938766d7c93d74406f894dc19aa96)
git cherry-pick -x $ancref..7c027596914938766d7c93d74406f894dc19aa96 |
github-actions
bot
added
the
incomplete-cherry-pick
kikito
removed
the
incomplete-cherry-pick
lhanjian
pushed a commit
that referenced
this pull request
Dec 23, 2024
* Revert "fix(rpc): disable cluster_rpc for 3.7" This reverts commit ddda6a1. This commit introduces a freshly redesigned DB-less mode that is more efficient when storing data, more efficient when accessing data and incrementally update able. This commit also introduces the "incremental sync" RPC server/client. It introduces the `kong.sync.v2` RPC capability that is used by CP to notify new version updates to DP and for DP to pull config delta. DP applies these config delta into the LMDB incrementally and transactionally which avoids the expensive config flip and cache wipe majority of the time. This commit also modifies the DAO so that for the CP, config diff logs are generated transactionally whenever entity modification occurs. Finally, this commit modifies the `off` strategy so that it works with the redesigned DB-less mode and storage format. Incremental sync is not yet enabled by default, it can be enabled by setting `cluster_incremental_sync = on` via `kong.conf`. KAG-4865 KAG-2986 KAG-2987 KAG-3502 KAG-3258 KAG-5283 --------- Co-authored-by: Chrono <[email protected]> Co-authored-by: Xiaochen Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
KAG-4865
KAG-2986
KAG-2987
KAG-3502
KAG-3258
KAG-5283
NOTE:
cluster_rpcnow isonby defaultcluster_incremental_syncnow isoffby default, we will continue to fix tests after this PR then turn it on.Flaky (perhaps need rerun):