Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable docValues on sort fields #180

Open
1 of 7 tasks
hackartisan opened this issue Jan 10, 2020 · 0 comments
Open
1 of 7 tasks

Enable docValues on sort fields #180

hackartisan opened this issue Jan 10, 2020 · 0 comments

Comments

@hackartisan
Copy link
Member

hackartisan commented Jan 10, 2020

What maintenance needs to be done?

Enable docValues for sort fields like author_sort, title_sort, for the catalog

Level of urgency

  • High
  • Moderate
  • Low

Why is this maintenance needed?

Sort fields are being created at runtime and stored in the cache. We could create these during indexing and potentially save memory and improve performance by creating them during indexing

Acceptance criteria

  • The cache size is reduced
  • This change does not significantly increase the time it takes to index
  • Run both read and write solr load tests and make sure there are no major regressions
  • If this is successful create issues to extend this to other applications

Implementation notes, if any

docValues documentation: https://lucene.apache.org/solr/guide/7_7/docvalues.html

@tpendragon did some research into this, saying:

Some things I'm reading (https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html and https://risdenk.github.io/2018/10/21/apache-solr-out-of-memory-symptoms-and-solutions.html) seems to imply that making sure our fields have docvalues enabled will make the heap much smaller.

Each replica's field cache for the catalog is ~ 3 GB, so that's 12 GB Heap.
author_sort is 21 MB, title_sort is ~ 160 MB
If we enable docValues on author_sort/title_sort etc, it'll use those at index time instead of populating that huge FieldCache
Figgy's field cache is also 2.8 GB
Er, or not? It looks like these metrics are showing other core fields

@rladdusaw rladdusaw changed the title Consider enabling docValues on sort fields Enable docValues on sort fields Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants