[nightly] Library scan doesn’t remove non-existent series

[newyorker123]

What happened?

To reproduce:

1. Delete a series folder from the file system.
2. Scan the library in Kavita.

The issue:

The series will not be automatically removed from Kavita. To delete the series in Kavita, you must either scan it from its detail page or perform a force scan on the library. A normal scan does not remove the series.

What did you expect?

Library scan can automatically remove non-existent series.

Kavita Version Number - If you don not see your version number listed, please update Kavita and see if your issue still persists.

Nightly Testing Branch

What operating system is Kavita being hosted from?

None

If the issue is being seen on Desktop, what OS are you running where you see the issue?

None

If the issue is being seen in the UI, what browsers are you seeing the problem on?

No response

If the issue is being seen on Mobile, what OS are you running where you see the issue?

None

If the issue is being seen on the Mobile UI, what browsers are you seeing the problem on?

No response

Relevant log output

No response

Additional Notes

No response

[DieselTech]

Kavita works by going off the filesystem events to detect changes. If your underlying filesystem doesn't tell kavita something has changed, then we don't know about it.

I just removed a folder and ran a scan on the latest nightly to test this:

[Kavita] [2024-11-16 20:05:50.229 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library Manga Step 5: Remove Deleted Series
[Kavita] [2024-11-16 20:05:50.231 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Removing series that were not found during the scan
[Kavita] [2024-11-16 20:05:50.455 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Found 1 series to remove: A Couple of Cuckoos
[Kavita] [2024-11-16 20:05:50.701 -05:00  4] [Debug] API.Services.Tasks.ScannerService [ScannerService] Series removal process completed

[newyorker123]

For some reason, Kavita isn’t detecting changes after I delete a folder.

Good to know it’s not a bug in the nightly release. I’ll see what I can do to fix this on my end. Thanks!

[majora2007]

I also can't reproduce as I've been deleting a lot of things during my testing.

[newyorker123]

After the latest stable release, Kavita still doesn't detect when a series folder is removed during a regular scan from the side nav.

I'm running my Kavita server with all files stored on a QNAP NAS. When I delete a file, the QNAP log center logs it as "Deleted." However, Kavita still fails to recognize the change.

Currently, my Kavita only detects removed files in the following scenarios:

1. Scanning from the series detail page.
2. Performing a force scan.
3. Conducting a regular scan from the side nav, but only if a new series is added. In this case, Kavita also removes the deleted series.

Can anyone help me identify the problem? It's really frustrating. Thanks in advance!

[majora2007]

Let's do some targeted testing. Delete your log file for the day, then perform the scan where something was deleted. Then attach the logs here.

[newyorker123]

Thank you for your help.

I have two series folders, test1 and test2, in a library directory test. I delete test2 from the filesystem and perform a regular scan for the library. The sereis test2 still exists in kavita. Here is the log:

[Kavita] [2024-11-23 12:33:38.958 +00:00  13] [Information] API.Services.TaskScheduler Enqueuing library scan for: 16
[Kavita] [2024-11-23 12:33:38.989 +00:00  13] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP POST /api/library/scan?libraryId=16&force=false responded 200 in 77.1454 ms
[Kavita] [2024-11-23 12:33:39.098 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Beginning file scan on test
[Kavita] [2024-11-23 12:33:39.130 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 1: Scan & Parse Files
[Kavita] [2024-11-23 12:33:39.175 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 1.A: Process 1 folders
[Kavita] [2024-11-23 12:33:39.177 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.B: Scan files in /media/test
[Kavita] [2024-11-23 12:33:39.186 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Step 1.C Found 1 directories to process for /media/test
[Kavita] [2024-11-23 12:33:39.189 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ProcessFiles] Skipping /media/test/test1 as it hasn't changed since last scan
[Kavita] [2024-11-23 12:33:39.189 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.C: Process files in /media/test
[Kavita] [2024-11-23 12:33:39.192 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Skipped File Scan for /media/test/test1 as it hasn't changed
[Kavita] [2024-11-23 12:33:39.193 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.D: Merge any localized series with series /media/test
[Kavita] [2024-11-23 12:33:39.196 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.E: Group all parsed data into logical Series
[Kavita] [2024-11-23 12:33:39.201 +00:00  7] [Debug] API.Services.Tasks.ScannerService 	[ScannerService] Library test Step 1.F: Generate Sort Order for Series and Finalize
[Kavita] [2024-11-23 12:33:39.213 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 2: Process and Update Database
[Kavita] [2024-11-23 12:33:39.215 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Found 0 Series that need processing in 79 ms
[Kavita] [2024-11-23 12:33:39.216 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Finished file scan in 79 milliseconds. Updating database
[Kavita] [2024-11-23 12:33:39.240 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 3: Save Library
[Kavita] [2024-11-23 12:33:39.275 +00:00  7] [Information] API.Services.Tasks.ScannerService [ScannerService] Finished library scan of 0 series in 290 milliseconds for test. There were no changes
[Kavita] [2024-11-23 12:33:39.275 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Library test Step 5: Remove Deleted Series
[Kavita] [2024-11-23 12:33:39.277 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Removing series that were not found during the scan
[Kavita] [2024-11-23 12:33:39.279 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Found 0 series to remove: 
[Kavita] [2024-11-23 12:33:39.280 +00:00  7] [Debug] API.Services.Tasks.ScannerService [ScannerService] Series removal process completed
[Kavita] [2024-11-23 12:33:39.285 +00:00  43] [Debug] API.Controllers.LibraryController Caching libraries for library_zehao
[Kavita] [2024-11-23 12:33:39.362 +00:00  43] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP GET /api/library/libraries responded 200 in 138.8310 ms
[Kavita] [2024-11-23 12:33:39.405 +00:00  43] [Debug] API.Controllers.LibraryController Caching libraries for library_zehao
[Kavita] [2024-11-23 12:33:39.408 +00:00  43] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP GET /api/library/libraries responded 200 in 9.3204 ms
[Kavita] [2024-11-23 12:33:39.412 +00:00  41] [Debug] API.Controllers.LibraryController Caching libraries for library_zehao
[Kavita] [2024-11-23 12:33:39.415 +00:00  41] [Information] Serilog.AspNetCore.RequestLoggingMiddleware HTTP GET /api/library/libraries responded 200 in 3.8171 ms
[Kavita] [2024-11-23 12:33:39.543 +00:00  37] [Information] API.Services.Tasks.CleanupService Performing cleanup of Cache directories
[Kavita] [2024-11-23 12:33:39.543 +00:00  37] [Information] API.Services.Tasks.CleanupService Cache directory purged

[majora2007]

Can you also show me whats in the test1 and test2 directories (before you removed test2)?

Were they the same series?

[newyorker123]

No, they are different series, with one file in each folder. The second one is the series in test2, which should be removed.

[majora2007]

Please run this python script before the deletion of the folder and attach the json that comes from it. This will just create a map of the folders for me to put into a unit test framework I built around this script.

Please also give me the library type you have setup.

import os
import json


def map_files(root_dir):
    files_map = []

    for dirpath, dirnames, filenames in os.walk(root_dir):
        # Skip directories that start with "."
        dirnames[:] = [d for d in dirnames if not d.startswith('.')]

        for filename in filenames:
            # Skip files that start with "."
            if not filename.startswith('.'):
                # Get the relative path of the file
                relative_path = os.path.relpath(os.path.join(dirpath, filename), root_dir)
                files_map.append(relative_path)

    # Export the map to a JSON file
    with open('files_map.json', 'w') as outfile:
        json.dump(files_map, outfile, indent=4)


if __name__ == "__main__":
    root_dir = os.getcwd()
    map_files(root_dir)
    print("File map generated and saved to files_map.json.")

[newyorker123]

I run the python script in the test library root folder, here is the json, but I don't think that provide much information

[
    "try.py",
    "test1\\test1.epub",
    "test2\\test2.epub"
]

I'm using Book type library for testing. But It also happens to every library type I use (Manga, light novel and Book).

[majora2007]

Thanks, that's all I need from you. When I have time I will mock this up in the code and see if I can reproduce. On my system, deleting folders has not been an issue, hence why this was marked previously as can't reproduce.

Time is extremely limited for me, so it will be a bit before I can try to reproduce.

Also does a force scan not get the series deleted?

[newyorker123]

Thank you for taking the time to look into this. I completely understand how busy you are with everything related to Kavita. If there’s anything else I can assist with or clarify, please don’t hesitate to let me know.

A force scan will delete the sereis. The series will be removed in the following scenarios:

1. Scanning from the series detail page.
2. Performing a force scan.
3. Conducting a regular scan from the side nav, but only if a new series is added. In this case, Kavita also removes the deleted series.