You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To reduce the number of jobs being scheduled, I propose we move multi-root support into the tool itself, peraps as a separate command that delegates to regular index
That's quite wasteful, not only because of CPU cycles, but also because of lack of persistent dependency cache.
Instead, we can add index-batch [options] root1 root2 root3 command which can index each root independently and possible in parallel, and then merge the resulting SCIP indexes.
Merging should be done in simplest way possible - basically concatenating lists of documents, assuming that subfolders are independent of each other. This can be lossy, but good enough.
The text was updated successfully, but these errors were encountered:
To reduce the number of jobs being scheduled, I propose we move multi-root support into the tool itself, peraps as a separate command that delegates to regular
index
For example, in this repo: https://github.com/in28minutes/spring-boot-examples we are currently inferencing 1 indexer job per folder.
That's quite wasteful, not only because of CPU cycles, but also because of lack of persistent dependency cache.
Instead, we can add
index-batch [options] root1 root2 root3
command which can index each root independently and possible in parallel, and then merge the resulting SCIP indexes.Merging should be done in simplest way possible - basically concatenating lists of documents, assuming that subfolders are independent of each other. This can be lossy, but good enough.
The text was updated successfully, but these errors were encountered: