Backend container seems to stop processing randomly

[kchristensen]

I've experienced this pretty reliably about 1-2x a week since using BirdCAGE and I haven't actually tracked down the cause yet, so this issue is to hopefully brainstorm what is going on.

The only thing I see in my backend container logs before things stop processing are:

[2023-06-27 07:51:00,806: WARNING/ForkPoolWorker-6] Recording successful. File saved to: /app/../tmp/75476791e76c4d7a8f85b5e9faeefa52.wav Now setting metadata
[2023-06-27 07:51:00,809: WARNING/ForkPoolWorker-6] Metadata set for: /app/../tmp/75476791e76c4d7a8f85b5e9faeefa52.wav
[2023-06-27 07:51:01,153: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/75476791e76c4d7a8f85b5e9faeefa52.wav
[2023-06-27 07:51:01,696: WARNING/ForkPoolWorker-8] Detected: House Finch Action: record
[2023-06-27 07:51:01,963: WARNING/ForkPoolWorker-8] Adding detection
[2023-06-27 07:51:01,966: WARNING/ForkPoolWorker-8] Detection added
[2023-06-27 07:51:01,967: WARNING/ForkPoolWorker-8] Notifying
[2023-06-27 07:51:01,967: ERROR/ForkPoolWorker-8] There are no service(s) to notify
[2023-06-27 07:51:01,967: WARNING/ForkPoolWorker-8] MQTT not being used. Skipping publication.
[2023-06-27 07:51:01,975: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:51:01,977: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:51:15,044: WARNING/ForkPoolWorker-7] Recording successful. File saved to: /app/../tmp/48b4d97f936b43b8a3a514e8b8ef24d0.wav Now setting metadata
[2023-06-27 07:51:15,047: WARNING/ForkPoolWorker-7] Metadata set for: /app/../tmp/48b4d97f936b43b8a3a514e8b8ef24d0.wav
[2023-06-27 07:51:16,031: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/48b4d97f936b43b8a3a514e8b8ef24d0.wav
[2023-06-27 07:51:16,621: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:51:16,624: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:51:17,108: WARNING/ForkPoolWorker-6] Recording successful. File saved to: /app/../tmp/e4eec42c234e42a98713b088988897c6.wav Now setting metadata
[2023-06-27 07:51:17,112: WARNING/ForkPoolWorker-6] Metadata set for: /app/../tmp/e4eec42c234e42a98713b088988897c6.wav
[2023-06-27 07:51:17,629: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/e4eec42c234e42a98713b088988897c6.wav
[2023-06-27 07:51:18,189: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:51:18,191: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:51:31,347: WARNING/ForkPoolWorker-7] Recording successful. File saved to: /app/../tmp/f141526ab0a34591bb70e9d5996b5656.wav Now setting metadata
[2023-06-27 07:51:31,350: WARNING/ForkPoolWorker-7] Metadata set for: /app/../tmp/f141526ab0a34591bb70e9d5996b5656.wav
[2023-06-27 07:51:32,245: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/f141526ab0a34591bb70e9d5996b5656.wav
[2023-06-27 07:51:32,816: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:51:32,820: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:51:33,142: WARNING/ForkPoolWorker-6] Recording successful. File saved to: /app/../tmp/83450105207747b789f3e570d8c43369.wav Now setting metadata
[2023-06-27 07:51:33,146: WARNING/ForkPoolWorker-6] Metadata set for: /app/../tmp/83450105207747b789f3e570d8c43369.wav
[2023-06-27 07:51:33,824: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/83450105207747b789f3e570d8c43369.wav
[2023-06-27 07:51:34,387: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:51:34,389: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:51:47,550: WARNING/ForkPoolWorker-7] Recording successful. File saved to: /app/../tmp/8e92edd130e14a5fbf2c42adc0de17d9.wav Now setting metadata
[2023-06-27 07:51:47,554: WARNING/ForkPoolWorker-7] Metadata set for: /app/../tmp/8e92edd130e14a5fbf2c42adc0de17d9.wav
[2023-06-27 07:51:48,444: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/8e92edd130e14a5fbf2c42adc0de17d9.wav
[2023-06-27 07:51:49,020: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:51:49,023: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:51:49,680: WARNING/ForkPoolWorker-6] Recording successful. File saved to: /app/../tmp/a23b5e5677e344e0ba28bcb206af44c6.wav Now setting metadata
[2023-06-27 07:51:49,684: WARNING/ForkPoolWorker-6] Metadata set for: /app/../tmp/a23b5e5677e344e0ba28bcb206af44c6.wav
[2023-06-27 07:51:50,027: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/a23b5e5677e344e0ba28bcb206af44c6.wav
[2023-06-27 07:51:50,556: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:51:50,559: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:52:03,851: WARNING/ForkPoolWorker-7] Recording successful. File saved to: /app/../tmp/fbfeee420c874d3088c31cbe7e2caf42.wav Now setting metadata
[2023-06-27 07:52:03,855: WARNING/ForkPoolWorker-7] Metadata set for: /app/../tmp/fbfeee420c874d3088c31cbe7e2caf42.wav
[2023-06-27 07:52:04,627: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/fbfeee420c874d3088c31cbe7e2caf42.wav
[2023-06-27 07:52:05,188: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:52:05,191: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:52:05,914: WARNING/ForkPoolWorker-6] Recording successful. File saved to: /app/../tmp/8af7051efc3941b8828220bd5233fd2d.wav Now setting metadata
[2023-06-27 07:52:05,917: WARNING/ForkPoolWorker-6] Metadata set for: /app/../tmp/8af7051efc3941b8828220bd5233fd2d.wav
[2023-06-27 07:52:06,196: WARNING/ForkPoolWorker-8] Requesting analysis for /app/../tmp/8af7051efc3941b8828220bd5233fd2d.wav
[2023-06-27 07:52:06,731: WARNING/ForkPoolWorker-8] Deleting metadata
[2023-06-27 07:52:06,734: WARNING/ForkPoolWorker-8] Metadata deleted
[2023-06-27 07:57:11,193: WARNING/ForkPoolWorker-8] There is no metadata and the file is old. Deleted file: 86970b09eb8f4705a37f70762c2ca1a6.wav
[2023-06-27 07:57:21,237: WARNING/ForkPoolWorker-8] There is no metadata and the file is old. Deleted file: 7198642279614db28f60d298317e4918.wav

7:57 corresponds when processing activity ceases. A container restart and it'll pick back up again. Unsure if this is related to #35 in any way, but that's all I have to work off of so far.

[UFOSmuggler]

i have the same issue. have been meaning to take a deep dive into it, but before i do, have you made any progress?

[kchristensen]

I ended up just setting up a crontab to restart the containers nightly when birds are sleeping, which did the trick. It's not ideal, but it seems to avoid the issue of everything locking up.

[UFOSmuggler]

thanks for your response mate. that is exactly what i have done as a temporary workaround as well.

if i get around to figuring out the problem, will mention it here.

[quimnut]

#79