Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(clustering): CP/DP RPC framework & dynamic log level RPC #12320

Merged
merged 39 commits into from
Apr 24, 2024
Merged

Conversation

dndx
Copy link
Member

@dndx dndx commented Jan 9, 2024

This PR implements a generic and extensible RPC framework that can be used to implement complex CP/DP interactions. This PR also implements a simple RPC "kong.debug.log_level.v1" that supports dynamic log level over RPC.

  • JSON RPC library

  • RPC framework

  • Compressions support

  • Authentication and encryption

  • Inert switch

  • Set log level over RPC

  • Get log level over RPC

  • Clear log level over RPC

[KAG-623]
KAG-3751
[KAG-623]: https://konghq.atlassian.net/browse/KAG-623?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

@github-actions github-actions bot added the cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee label Mar 21, 2024
@dndx dndx force-pushed the feat/cp_dp_rpc branch from 78d2f57 to febac1f Compare April 3, 2024 07:49
@dndx dndx force-pushed the feat/cp_dp_rpc branch 3 times, most recently from af8113e to 35426dd Compare April 8, 2024 09:31
@chronolaw chronolaw added this to the 3.7.0 milestone Apr 8, 2024
kong/clustering/rpc/socket.lua Show resolved Hide resolved
kong/clustering/rpc/queue.lua Show resolved Hide resolved
kong/clustering/rpc/socket.lua Show resolved Hide resolved
kong/clustering/rpc/socket.lua Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/utils.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/future.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
@chobits chobits self-requested a review April 9, 2024 06:22
kong/clustering/services/debug.lua Outdated Show resolved Hide resolved
kong/clustering/services/debug.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/socket.lua Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
kong/clustering/rpc/socket.lua Outdated Show resolved Hide resolved
@dndx
Copy link
Member Author

dndx commented Apr 24, 2024

After discussion internally we have decided to address some of the styling reviews in a separate PR in order to keep the conversation manageable. The PR should be in a mergable state as long as it does not contain DoA type bugs that affects other components.

Copy link
Contributor

@nowNick nowNick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm glad to see progress on RPC CP/DP framework 🚀
I just wish we had more time to add more tests and refactor it a bit - let's prioritize it as the next step 🙏


while not exiting() do
repeat
local n, err = notifications_queue:pop(0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense - thanks!

@dndx dndx merged commit b7d591d into master Apr 24, 2024
32 checks passed
@dndx dndx deleted the feat/cp_dp_rpc branch April 24, 2024 15:51
@team-gateway-bot
Copy link
Collaborator

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-12320-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-12320-to-master-to-upstream
git checkout -b cherry-pick-12320-to-master-to-upstream
ancref=$(git merge-base 9db89af20ac5b594b26e3a794fdb87e34db0a246 21b34f6c0aa8ce4c44eb0243667d2c5d29240481)
git cherry-pick -x $ancref..21b34f6c0aa8ce4c44eb0243667d2c5d29240481

@github-actions github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Apr 24, 2024
@locao locao removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Apr 26, 2024
locao pushed a commit that referenced this pull request Jun 21, 2024
This PR implements an extensible mechanism where CP and DP nodes can bidirectionally communicate
with each other and perform RPC calls. Which can be used for building complex interactions
such as config sync, debugging, etc.

This PR also implements a simple RPC service for CP to inspect and change the dynamic log level
of the connected DP nodes. The functionality is exposed via a new Admin API endpoint
`/clustering/data-planes/<node-id>/log-level`.

KAG-623
KAG-3751
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants