Indexing: use one ctags process per shard #702
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jtibshirani
commented
Nov 20, 2023
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package ctags |
There was a problem hiding this comment.
I made sure to separate the rename from parser_map.go -> parser_factory.go into its own commit, to make it easier to see what changed.
jtibshirani
commented
Nov 20, 2023
| continue | ||
| } | ||
| parsers[parserKind] = parser | ||
| defer parser.Close() |
There was a problem hiding this comment.
This is a bit messy. In a follow up I plan to:
- Clean up
lockedParser, removing unneeded synchronization and simplifying timeout logic - Abstract out this multiplexing logic into a parser wrapper, so it delegates to the right sub-parser process
jtibshirani
force-pushed
the
jtibs/parallel-ctags
branch
from
2cbe5ec to
5d480d8
Compare
jtibshirani
added a commit
to sourcegraph/sourcegraph-public-snapshot
that referenced
this pull request
Nov 27, 2023
In sourcegraph/zoekt#702, we updated indexserver to parse symbols in parallel by spawning a new ctags process per shard. By default, indexing uses all available CPUs to create shards in parallel, so now it will create many more processes than before. As a safeguard, we're exposing a site config setting to reduce the indexing concurrency. It's not intended to be set by users, but will let us experiment and make sure the defaults are solid. As part of this change, I bumped the Zoekt dependency to pull in the change to IndexOptions.
jtibshirani
added a commit
that referenced
this pull request
Dec 6, 2023
This change cleans up the Go ctags parser wrapper as a follow-up to #702. Specific changes: * Remove synchronization in `lockedParser` and rename it to `CTagsParser` * Push delegation to universal vs. SCIP ctags into parser wrapper * Simplify document timeout logic * Rename some files
vovakulikov
pushed a commit
to sourcegraph/sourcegraph-public-snapshot
that referenced
this pull request
Dec 12, 2023
In sourcegraph/zoekt#702, we updated indexserver to parse symbols in parallel by spawning a new ctags process per shard. By default, indexing uses all available CPUs to create shards in parallel, so now it will create many more processes than before. As a safeguard, we're exposing a site config setting to reduce the indexing concurrency. It's not intended to be set by users, but will let us experiment and make sure the defaults are solid. As part of this change, I bumped the Zoekt dependency to pull in the change to IndexOptions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, we use a single ctags process for indexing an entire repository.
Even though we build shards in parallel, they all share the same (single
threaded) ctags process. Since ctags is one of the most expensive parts of
shard building, this creates a bottleneck that can really slow down indexing.
This change proposes to launch a new ctags process per shard. For
sgtest/megarepo, this speeds up indexing by almost 2x (enabling scip-ctagsand setting
-parallelism=4):Addresses https://github.com/sourcegraph/sourcegraph/issues/58112