(ipc): resources are not completely freed when closing subscription

[felixeriksson]

Describe the bug
Memory leaks when an IPC subscription is created and closed.

Heap dumps from affected machines show that the memory is hogged by the object Map<String, SubscriptionTrie<K>> children in an instance of the class SubscriptionTrie (see source). The SubscriptionTrie instance that hogs the memory is called listeners which is a member of the class PubSubIPCEventStreamAgent (source).

In PubSubIPCEventStreamAgent.handleUnsubscribeToTopicRequest() (which I believe is invoked when calling close() on SubscribeToTopicOperation in the Python SDK, which we're using in our application), a call to remove() on listeners is made. When inspecting the source for remove(), it seems that the member subscriptionCallbacks is cleaned up, but not children. subscriptionCallbacks and children are the only non-static final members of SubscriptionTrie.
To Reproduce
Using the Python SDK:

ipc_client = GreengrassCoreIPCClientV2()

count = 0
while True:
    request_id = uuid.uuid1()
    response_topic = f"dummy_method-response-{request_id}"
    def response_listener(message: SubscriptionResponseMessage) -> None:
        log.info("Response listener")
    def error_listener(_: Exception) -> Union[None, bool]:
        log.info("Error listener")
        return True
    _, subscribe_to_topic_operation = ipc_client.subscribe_to_topic(
        topic=response_topic,
        on_stream_event=response_listener,
        on_stream_error=error_listener,
    )
    count += 1
    if count % 1000 == 0:
        log.info("Created %s subscriptions", count)
        time.sleep(0.1)
    subscribe_to_topic_operation.close()

The above example seemingly causes unbounded heap usage growth.

Expected behavior
The memory allocated when setting up a subscription should be completely freed when calling subscribe_to_topic_operation.close().

Actual behavior
The memory allocated when setting up a subscription is not completely freed despite calling subscribe_to_topic_operation.close().

Environment

• OS: Ubuntu 22.04.3 LTS
• JDK version:
  • openjdk version "11.0.25" 2024-10-15
  • OpenJDK Runtime Environment (build 11.0.25+9-post-Ubuntu-1ubuntu122.04)
  • OpenJDK 64-Bit Server VM (build 11.0.25+9-post-Ubuntu-1ubuntu122.04, mixed mode, sharing)
• Nucleus version: 2.11.2

Additional context
A related issue was opened a few months ago (#1650), but closed due to the belief that this is a problem in the Python SDK (and also due to incorrect usage of GreengrassCoreIPCClientV2 in the example, which is fixed in the example in this issue).

An issue was therefore opened in the Python SDK repo: aws/aws-iot-device-sdk-python-v2#589
But given the above reasoning, I don't think it is within the power of the Python SDK to clean up the subscription-related data in SubscriptionTrie.children, as it does not seem like remove() (or any other method) removes data from the children member upon unsubscription - therefore I opened this issue.

This bug impacts us because our application creates new subscriptions regularly. The devices we build are intended to run for months or years, but often they run out of memory and require manual intervention after a few weeks.

[junfuchen99]

Hello, thank you for you detailed investigation. We are taking a closer look at the issue and will get back to you.