sdcv: Don't walk into "res/" subdirectories

[georgeto]

When searching for .ifo files use the same optimization as we do in the koreader frontend in getIfosInDir(path) in readerdictionary.lua.

Fortunately sdcv uses the for_each_file function exclusively to search for .ifo files, making the patch trivial.

I have several dictionaries on my Kobo Aura One, some of which contain 'res' subdirectories with many files. After a restart or cold boot, the first dictionary lookup would take around 45s, while subsequent lookups would only take merely around 0.5s. My theory for this massive difference is that after reboot, the file system cache is empty, and therefore the file system structure for all the resource directories must first be loaded from the flash, which seems to be quite slow.

With the patched sdcv there is now no difference between the first and the following lookups anymore. In addition, the average lookup time on my device drops from 0.5s to about 0.3s.

---

This change is 

[benoit-pierre]

Wouldn't it be better to contribute this upstream?

[georgeto]

Wouldn't it be better to contribute this upstream?

I have my doubts that upstream would accept it in the current form, because it does not handle the edge case that there is a dictionary named res. For koreader that is not a problem however, because the check in readerdictionary.lua has the same limitation. But with a patch that doesn't have this limitation I could try it upstream. I will think about it.

[georgeto]

But with a patch that doesn't have this limitation I could try it upstream. I will think about it.

I looked at the code again, but I currently don't see how to implement this nicely in a generalized way, i.e. outside the controlled conditions that we have in koreader.

A possible approach could the following, however I wouldn't call it nice: The directory iteration could remember when a res subdirectory is encountered, and skip it at first. Then at the end if there was no .ifo file in the directory, enter the res subdirectory. However, it remains that for_each_file is actually a generic function that could theoretically also be used for data types other than .ifo.

[Frenzie]

What concretely is many files?

[georgeto]

What concretely is many files?

In my data/dict directory there are a total of 10500 files which are located in res subdirectories .
Not necessarily a number where I would expect the measured delay of 45s.

[Frenzie]

On an SD card with FAT32 that can sometimes be surprisingly slow, even if I don't necessarily mean quite that slow. You could run a simple time ls to compare the numbers.

[Frenzie]

Or since I just realized you said subdirectories plural, perhaps that should be something like time find . -type f > /dev/null.

[georgeto]

Or since I just realized you said subdirectories plural, perhaps that should be something like time find . -type f > /dev/null.

Basically the same result as observed with sdcv:

[root@kobo koreader]# time find data/dict -type f > /dev/null
real    0m 43.31s
user    0m 0.14s
sys     0m 43.04s
[root@kobo koreader]# time find data/dict -type f > /dev/null
real    0m 0.23s
user    0m 0.04s
sys     0m 0.18s
[root@kobo koreader]# find data/dict -type f | wc
    10558     10595    869759

[Frenzie]

Right, so sdcv presumably isn't accidentally doing something inefficient, other than going through res. That's not very useful per se, but it's always good to double check. ^_^