Can the Red Bird based in-memory logger, enabled by default, be purged? #201
Unanswered
netopsengineer
asked this question in
Q&A
Replies: 1 comment
-
I had a similar problem in prod where a 1-second schedule quickly OOM'd a long-running FastAPI microservice. I created a couple of custom repos (since I didn't need the rocketry logs as I had other logging mechanisms in place). Both are based on the redbird MemoryRepo. One limits the memory consumption, and the other simply mocks all functions and won't keep any logs (careful as it might break log-retrieving functionalities of rocketry) from redbird.repos import MemoryRepo
DEFAULT_MAX_MEMORY_ENTRIES = 1000
DEFAULT_CLEANUP_ENTRIES = 10
class LimitedMemoryRepo(MemoryRepo):
"""
This expands the naive MemoryRepo from redbird.
It will purge logs after a certain length has been reached
"""
max_memory_entries: int = DEFAULT_MAX_MEMORY_ENTRIES # will run cleanup when this number of logs is reached
cleanup_last_entries: int = DEFAULT_CLEANUP_ENTRIES # will delete this many logs from list on each cleanup
def __init__(
self, max_memory_entries: int = DEFAULT_MAX_MEMORY_ENTRIES, cleanup_last_entries: int = DEFAULT_CLEANUP_ENTRIES
):
super().__init__()
self.max_memory_entries = max_memory_entries
self.cleanup_last_entries = cleanup_last_entries
def insert(self, item):
super().insert(item)
# run cleanup
if len(self.collection) >= self.max_memory_entries:
self.collection = self.collection[self.cleanup_last_entries:]
class NullRepo(MemoryRepo):
"""
This functions like a redbird log repo, but won't log anything. Cannot perform quries either
Can be used as a Rocketry logger repo where logging of tasks is not needed
"""
def insert(self, item):
return
def query_data(self, query):
return []
def query_update(self, query: dict, values: dict):
return
def query_delete(self, query: dict):
return I create my rocketry scheduler like this. from rocketry import Rocketry
from memory_limited_repo import MemoryLimitedRepo
"""
Rocketry execution
main: no parallelization
async: run async if can (default in the future)
thread: on separate thread
process: on separate process (default)
"""
# rocketry_app = Rocketry(execution="thread", logger_repo=NullRepo())
rocketry_app = Rocketry(logger_repo=MemoryLimitedRepo(max_memory_entries=1000, cleanup_last_entries=10)
) |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
The observed behavior of the Red Bird based in-memory logging, which is enabled by default when another option is not configured, seems to consume RAM until the program is restarted, or goes OOM. Is there an option to purge it after a certain size or time period, without redeploying the app? I have not been able to locate any documentation around that specific feature, aside from configuring a different logging method, which disables the in-memory logging.
Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions