Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs spam from openraft::engine::handler::vote_handler #1236

Closed
sebadob opened this issue Aug 27, 2024 · 3 comments
Closed

Logs spam from openraft::engine::handler::vote_handler #1236

sebadob opened this issue Aug 27, 2024 · 3 comments
Assignees

Comments

@sebadob
Copy link

sebadob commented Aug 27, 2024

Describe the bug
Since v0.9.14 I see endless log spamming on the info level coming from the openraft::engine::handler::vote_handler. These seem to happen on follower nodes only. I have not seen them on the leader so far. They look like this:

2024-08-27T07:00:02.130351Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:02.207132Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:02.356268Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:02.433548Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:02.582226Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:02.659156Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:02.808812Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:02.885670Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:03.034084Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed
2024-08-27T07:00:03.112075Z  INFO openraft::engine::handler::vote_handler: updated last_seen_vote from T1-N1:committed to T1-N1:committed

This happens on fresh clusters and on older already existing ones as well.
When I pin the version to openraft = "=0.9.13", these logs disappear.

To Reproduce
Just start up any cluster, I guess. I saw these when working on Hiqlite and you could simply just just test for instance to see these messages, although the tests emit many logs. I also see these from my test deployment on a real environment.

Expected behavior
I guess these logs are simply in the wrong log level and should probably be pushed into trace or something like this, because they produce a very huge log output in a short amount of time.

Actual behavior
See Describe the bug, it is pretty simple.

Env (please complete the following information):

  • openraft = { version = "0.9.14", features = ["serde", "storage-v2"] }
  • any additional information is probably pointless here, since its just logging

Additional files:
This is a screenshot from the log output of a 3 node cluster started inside a K8s test env, nodes 1, 2, 3 from top to bottom. Node 1 became the leader in this case and the followers started logs spamming:

grafik

Copy link

👋 Thanks for opening this issue!

Get help or engage by:

  • /help : to print help messages.
  • /assignme : to assign this issue to you.

@drmingdrmer drmingdrmer self-assigned this Aug 28, 2024
drmingdrmer added a commit to drmingdrmer/openraft that referenced this issue Aug 28, 2024
A node's `vote` may be updated when a leader observes a higher vote.
In such cases, the leader updates its local vote and steps down.
However, this vote update does not imply that the node accepts the
higher vote as valid for leadership, as it has not yet compared their
logs.

In this commit, re-enable `VoteResponse.vote_granted` to indicate a vote
is granted.

This commit also fix:

- Fix: databendlabs#1236
@drmingdrmer
Copy link
Member

Thank you for bringing this issue to my attention.

This should be resolved in version v0.9.15. Please upgrade and let me know if the issue is addressed.

Note that version 0.9.14 has been yanked due to a bug that could potentially cause a brain split. This bug has also been fixed in v0.9.15.

@sebadob
Copy link
Author

sebadob commented Aug 28, 2024

I just tested with 0.9.15 and the issue is gone, thank you!

@sebadob sebadob closed this as completed Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants