fix: prune subscriptions trie when topics are unsubscribed #1671
+103
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Binary incompatibility detected for commit e17d003. com.aws.greengrass.util.NucleusPaths is binary incompatible and is source incompatible because of CONSTRUCTOR_REMOVED Produced by binaryCompatability.py |
|
Unit Tests Coverage Report
Minimum allowed coverage is Generated by 🐒 cobertura-action against e17d003 |
|
Integration Tests Coverage Report
Minimum allowed coverage is Generated by 🐒 cobertura-action against e17d003 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
#1670
Description of changes:
Background
PubSubIPCEventStreamAgenthandles all the IPC local pubsub requests. It implements the handlers forSubscribeToTopicRequestandPublishToTopicRequestwhere the topics are stored in aSubscriptionTrietrie.When a topic is subscribed to, it is added to the
SubscriptionTrietrie where the topic levels as represented as nodes of the trie. When you subscribe to a topic with a handler, the handler is registered as a listener of that topic where the last topic level node of that topic increments its listeners count by 1. So, for a valid topic in a trie, the number of subscriptions count at the last topic level of that topic is greater than or equal to one.Eg. When topics like a/b, a/b/c, a/# and d/+ are registered with a callback, the trie representation is as follows where the nodes
b,c,#and+maintain the number of subscription handlers held by a topic that ends at their topic level.Problem
When an IPC subscription for a topic is closed, it is expected that topic references are removed from the trie if there are no others listeners for the same topic. But, currently, we only remove the registered subscription handler from the list of handlers maintained at that topic level, leaving behind all the nodes as is. That means, once a topic is subscribed to, it is never removed from the heap memory even when it is closed/unsubscribed to and no longer used.
Changes
When a topic is unsubscribed to, we clear up the references from the
SubscriptionTrie, which manages topics along with their listeners for matching and filtering.If a topic node has no children or no callbacks registered, it means that the path to that topic node is not a complete topic or a prefix of other topic(s). So, it can be removed.
We modify the trie in the following way when a topic is unsubscribed to or a subscription is closed:
Why is this change necessary:
#1670
The heap memory keeps growing by retaining the object references even when they're no longer needed. If the heap size continues to grows due to
SubscribeToTopicRequestobject, the IPC event loop thread which handles IPC requests is killed due to out of memory error leaving the JVM in a bad state.How was this change tested:
Any additional information or context required to review the change:
Documentation Checklist:
Compatibility Checklist:
any deprecated method or type.
Refer to Compatibility Guidelines for more information.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.