This repository has been archived by the owner on Sep 30, 2024. It is now read-only.
Search: add index shard concurrency to site config #58514
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Codenotify: Notifying subscribers in CODENOTIFY files for diff 03b86e3...d86085f.
|
keegancsmith
approved these changes
Nov 23, 2023
schema/site.schema.json
Outdated
| "description": "The number of threads each indexserver should use to index shards. If not set, indexserver will use the number of available CPUs. This is exposed as a safeguard and should usually not require being set.", | ||
| "type": "integer", | ||
| "group": "Search", | ||
| "examples": [["10"]] |
There was a problem hiding this comment.
shouldn't this be
Suggested change
| "examples": [["10"]] | |
| "examples": [10] |
vovakulikov
pushed a commit
that referenced
this pull request
Dec 12, 2023
In sourcegraph/zoekt#702, we updated indexserver to parse symbols in parallel by spawning a new ctags process per shard. By default, indexing uses all available CPUs to create shards in parallel, so now it will create many more processes than before. As a safeguard, we're exposing a site config setting to reduce the indexing concurrency. It's not intended to be set by users, but will let us experiment and make sure the defaults are solid. As part of this change, I bumped the Zoekt dependency to pull in the change to IndexOptions.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In sourcegraph/zoekt#702, we updated indexserver to parse symbols in parallel
by spawning a new ctags process per shard. By default, indexing uses all
available CPUs to create shards in parallel, so now it will create many more
processes than before.
As a safeguard, we're exposing a site config setting to reduce the indexing
concurrency. It's not intended to be set by users, but will let us experiment
and make sure the defaults are solid.
As part of this change, I bumped the Zoekt dependency:
sourcegraph/zoekt@0959170...5c51794
Relates to #58112
Test plan
New unit test, also existing coverage from proto round trip tests