Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shards: only trigger rescan on .zoekt files changing #801

Merged
merged 1 commit into from
Aug 2, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions shards/watcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ func versionFromPath(path string) (string, int) {
}

func (s *DirectoryWatcher) scan() error {
// NOTE: if you change which file extensions are read, please update the
// watch implementation.
fs, err := filepath.Glob(filepath.Join(s.dir, "*.zoekt"))
if err != nil {
return err
Expand Down Expand Up @@ -216,21 +218,38 @@ func (s *DirectoryWatcher) watch() error {
signal := make(chan struct{}, 1)

go func() {
notify := func() {
select {
case signal <- struct{}{}:
default:
}
}

ticker := time.NewTicker(time.Minute)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems fairly frequent for a fail-safe? Isn't this roughly in the order of magnitude we scanned before this PR? I might be totally off though ;-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are always scanning right now on dotcom. IE as soon as one scan is done another starts. The scanning doesn't take long (~50ms?) but effectively we do this for { scan() }. So once a minute seems good?


for {
select {
case <-watcher.Events:
select {
case signal <- struct{}{}:
default:
case event := <-watcher.Events:
// Only notify if a file we read in has changed. This is important to
// avoid all the events writing to temporary files.
if strings.HasSuffix(event.Name, ".zoekt") || strings.HasSuffix(event.Name, ".meta") {
notify()
}

case <-ticker.C:
// Periodically just double check the disk
notify()

case err := <-watcher.Errors:
// Ignore ErrEventOverflow since we rely on the presence of events so
// safe to ignore.
if err != nil && err != fsnotify.ErrEventOverflow {
log.Println("watcher error:", err)
}

case <-s.quit:
watcher.Close()
ticker.Stop()
close(signal)
return
}
Expand Down
Loading