DataAccessException triggered when handling a large number of CBZ files

[Kuan-Lun]

Steps to reproduce

1. Place 2,788 CBZ files, totaling 59 GB, in a single directory.
2. Add the directory to libraries, unchecking all settings.
3. Click on "Scan library files" (the scan does not start automatically after adding).
4. A certain number of CBZ files will be scanned, and the scanned files will appear as "To be analyzed" in Komga.
5. The scan interrupts, and an error is displayed in the logs (see Logs).
   *. Repeating steps 3 to 5 multiple times eventually allows the entire library to be scanned.

Expected behavior

The scan should complete after a single click of the scan button.

Actual behavior

Please refer to the Steps to reproduce.

Logs

mylogs.csv

Komga version

v1.11.3-master

Operating system

Linux

Installation method

Docker

Other details

No.

Acknowledgements

• I have searched the existing issues (open AND closed) and this is a new ticket, NOT a duplicate or related to another open issue.
• I have written a short but informative title.
• I have checked the FAQ.
• I have updated the app to the latest version.
• I will fill out all of the requested information in this form.

[gotson]

your database is not reachable or too slow:

SqliteMainPool - Connection is not available, request timed out after 30000ms

[Kuan-Lun]

I understand. However, could this possibly be due to a one-time insertion of too much data into the "local" database?

[gotson]

no, Komga processes data sequentially, people have been successfully processing TBs of data in Komga without any problem. Most likely the issue lies in how you host Komga.

Can you share your dockerfile, and provide details on how the drives are mapped, and what is the underlying hardware and storage infrastructure ?

[Kuan-Lun]

I understand that Komga can handle TBs of data without issues, as I successfully processed 2 TB of data with the older version (v1.10.4-master) without any problems. I believe this issue might be related to the new version of Komga.

Dockerfile: docker-compose.yml
Drives: Synology DS1520+

[gotson]

Nothing has changed on that front for quite some time. On Synology you may need to check that the config folder is not indexed / managed by Synology in any way. Syno media indexing can be very taxing on some folders.