SAI-4604 : Add proper metrics around distributed request delay / http2 client bandwidth #148

noblepaul · 2023-09-25T14:15:36Z

The new metric for delayed requests added to path [metrics].[solr.node].[QUERY.httpShardHandler.delayedInterNodeRequests]

looks as follows

  "QUERY.httpShardHandler.delayedInterNodeRequests": {
        "count": 0,
        "meanRate": 0.0,
        "1minRate": 0.0,
        "5minRate": 0.0,
        "15minRate": 0.0
      }

Add support for HLL params for testing various accuracy levels but also to unlock options for cheap unique estimations for facet sorting.

* added lucne submodule to point the fs/branch_9_4_1 * tmp lucene branch * git checkout with submodules * updated lucene repo ref * updated pre-commit yaml to ignore some tasks * updated precommit.yml * pointing fs/branch_9_4_1 back

…Processor whitelist params (#26)

#29) * LUCENE-9302,SOLR-14381: Integer overflow in total count in grouping results * updated lucene version * updated lucene branch and interface * updated * spotless apply * updated test Co-authored-by: Ishan Chattopadhyaya <[email protected]>

* added system property to disable zk config watch

* Adding solr core and solrj tests to precommit checks * Updating to run tests as separate step * Fixing solr test name * Adding submodule checkout * Running all tests * Running all tests

* Ability to compress the collection state [sc-214780] * Running gradle tidy * Fixing errorprone issues * Removing unused import * Running gradle tidy

* Port over BitDocSet max calculation. https://fullstory.atlassian.net/browse/SAI-3925 * format fix

* Applying changes from https://github.com/fullstorydev/lucene-solr/pull/102/files * ./gradlew tidy

* Range doc values. (#107) * Range doc values. Add initial json facet range doc values support. Only support single value numerics to start. * Use DocSetBuilder. * Use long directly. Update comment referring to interval tree. * Add interval tree. * Range docvalues sub stats. (#109) Forgot to handle sub stats/functions. Bucket based subfacets worked fine. Needed a few additions to complete. * Doc set builder cost estimate. (#124) Improve the cost estimate for range query doc values processing. The cost estimate should scale with the number of documents being processed. This will allow the doc set builder to be more efficient as it won't need to resize as frequently. * updated test compilatin fix * actually bump tdigest dependency version also revert changes to stddev and variance that likely result from full-line/partial-line merge conflict resolution * fixed class cast exception * fixed test issue * test fix - type conversion * more test fix - type conversion issue Co-authored-by: Clay Goddard <[email protected]> Co-authored-by: Michael Gibney <[email protected]>

…ossible

…ions

…name`

…anch_9x (#63)

…se to fix (#64)

* Changed indent size for state.json file from 2 to 0 * updated test. Set default indent to 0 * Update test to remove log message. Co-authored-by: Hitesh Khamesra <[email protected]> Co-authored-by: hitesh <[email protected]>

Shard flag is only needed when a given facet or subfacet has refinement enabled. Do not set the flag otherwise as it has an overflow possibility. Also do not set a bucket num unless refinement is enabled (this also helps with avoiding overflow). Co-authored-by: Clay Goddard <[email protected]>

* Fixed missing rid in SearchHandler log message * ./gradlew tidy

…utImpl) (#111)

ports changes from: fullstorydev/lucene-solr#204 (60964ece8abc1e7e838b9a8f1e54ff7f4505d91d) fullstorydev/lucene-solr#207 (ee264220729490da81916af8c2ae23c5e4e52718) fullstorydev/lucene-solr#208 (20b2b694c223723c6e80c2344b401772c67221ce) fullstorydev/lucene-solr#209 (8835c4d0b4d90de90559c380dab8e51a0ee895a6) along with updates to compile against branch_9x

… branch_9x (#118) * fix remaining compile errors after merge * fix forbiddenAPIs after merge

…ode (apache#1794)

…CoordinatorHttpSolrCall init (apache#1800) * Synchronize to avoid race condition in CoordinatorHttpSolrCall * ./gradlew tidy

… property (#124)

…#125) (cherry picked from commit 2e7959e)

… value retrieval (#127) (cherry picked from commit 75f01bf)

* Handling IOExceptions from Lucene queries which are generally exceptions caused by an invalid query so they should be returned as 400s * Updating to handle specific exceptions and adding more tests * Tidying code and fixing imports * Improving error handling in test * Simplifying test code * Fixing indentation and test system property

(cherry picked from commit 2b98621)

…as true (#139) * Handling BAD_REQUEST exceptions as failures even with shard.tolerant as true * Fixing forbiddenapi use * Updating the set of error codes to just be a Set

backport unmerged upstream PR at commit 9eeb7e0 * include updates to accommodate FS custom CacheConfig ctor signature

(cherry picked from commit 110a296)

* fix improper ramBytes reporting for wrapped OrdinalMap * for reliable testing of OrdinalMap building, must set NoMergePolicyFactory * tidy

…st filesystem (#143) (cherry picked from commit 60dd5c8)

Justin Sweeney and others added 30 commits November 1, 2022 11:48

Updating Github Workflow to run on all branches, including with slash

0ac02d7

Add back PrometheusMetricsServlet. (#37)

c603741

Added support for HLL params. (#113)

e1d4349

Add support for HLL params for testing various accuracy levels but also to unlock options for cheap unique estimations for facet sorting.

Add facet parser factories and plugin support. (#122)

3a9f6ae

Merge branch 'branch_9x' into fs/branch_9x

4359482

failOnVersionConflicts param should be added to the DistributedUpdate…

7063d46

…Processor whitelist params (#26)

Added getRequestId from 8.4.2 branch (#8)

ea9a160

Bump tdigest for latest updates and add a percentile configurable (#34)

a209409

add warn log for ShardRequest Exceptions (#37)

56fbf89

Added FacetParserFactory to staticlist (#38)

640851a

Add java system property to disable config watch (#41)

6b399e3

* added system property to disable zk config watch

Add java system property to overseer queue size (#43)

e5fb6b5

updated lucene reference (#51)

a1e82a4

Adding solr tests to precommit checks (#52)

b77737f

* Adding solr core and solrj tests to precommit checks * Updating to run tests as separate step * Fixing solr test name * Adding submodule checkout * Running all tests * Running all tests

Ability to compress the collection state [sc-214780] (#47)

b7da5ae

* Ability to compress the collection state [sc-214780] * Running gradle tidy * Fixing errorprone issues * Removing unused import * Running gradle tidy

Port over BitDocSet max calculation. (#58)

1f59231

* Port over BitDocSet max calculation. https://fullstory.atlassian.net/browse/SAI-3925 * format fix

sc 214640 : log cleanup and req tracing [from lucene-solr] (#56)

4e764bc

* Applying changes from https://github.com/fullstorydev/lucene-solr/pull/102/files * ./gradlew tidy

doclet group is arbitrary; use solr namespace instead to avoid conflicts

c3f2b73

allow reading lucene.dev.path from gradle.properties

f23e712

use luceneBaseVersionProvider for lucene dependency versions, where p…

1d86646

…ossible

it makes no sense to validate jar checksums for SNAPSHOT (local) vers…

74315f0

…ions

prevent validation tasks from recursing into lucene subdir

3908207

name of included local Lucene build should match Lucene `rootProject.…

1131870

…name`

Merge branch 'michaelgibney/gradle-ide-lucene-root-subdir' into fs/br…

ce03e2b

…anch_9x (#63)

CI times out on large test suite; make CI logging slightly more verbo…

14d6953

…se to fix (#64)

patsonluk and others added 28 commits June 29, 2023 13:29

SAI-4528 : Missing rid in SearchHandler log (#113)

6a71341

* Fixed missing rid in SearchHandler log message * ./gradlew tidy

support dynamic ngram enable/disable (after pattern of SolrQueryTimeo…

a693e20

…utImpl) (#111)

Merge branch 'branch_9x' into fs/branch_9x

ea08703

Updating lucene version after merge

dcdbfc9

Fixing merge conflicts

14ba886

remaining fixes (compile and forbiddenAPIs) after merge from upstream…

d589e93

… branch_9x (#118) * fix remaining compile errors after merge * fix forbiddenAPIs after merge

Merge branch 'branch_9x' into fs/branch_9x

86a4418

Merge branch 'branch_9_3' into fs/branch_9_3

482a153

Update lucene hash to point at fs/branch_9_7

b93ec20

Remove a println statement from JSONTestUtil (apache#1795)

d18f73d

SOLR-16871: Fix for duplicated replica added from first coordinator n…

44eedac

…ode (apache#1794)

SOLR-16871: Synchronize on a larger block to avoid race condition in …

1625860

…CoordinatorHttpSolrCall init (apache#1800) * Synchronize to avoid race condition in CoordinatorHttpSolrCall * ./gradlew tidy

make Http2SolrClient max outstanding requests configurable via system…

6fa4971

… property (#124)

SAI-4608: extend non-automaton prefix query optimization to TextField (…

c0109f0

…#125) (cherry picked from commit 2e7959e)

direct prefix query need only compare a single index (#126)

fa10f8e

SAI-4620: (FS-specific) pre-filter dynamic docValues fields for field…

597ba26

… value retrieval (#127) (cherry picked from commit 75f01bf)

Re-adding 0 indentation of doc colleciton JSON and fixing test (#131)

3155c7a

indentation test fix (followup to #131) (#137)

7268085

(cherry picked from commit 2b98621)

Handling BAD_REQUEST exceptions as failures even with shard.tolerant …

4ada0be

…as true (#139) * Handling BAD_REQUEST exceptions as failures even with shard.tolerant as true * Fixing forbiddenapi use * Updating the set of error codes to just be a Set

SOLR-16966: Add a first-class cache for OrdinalMaps (#119)

1246c69

backport unmerged upstream PR at commit 9eeb7e0 * include updates to accommodate FS custom CacheConfig ctor signature

add hook for Directory impls to directly supply their size (#141)

23a266b

(cherry picked from commit 110a296)

minor ordMapCache fixes: ramBytes accounting and a test fix (#142)

bcfbd4b

* fix improper ramBytes reporting for wrapped OrdinalMap * for reliable testing of OrdinalMap building, must set NoMergePolicyFactory * tidy

add ability to explicitly reconcile heap directory size account again…

9e094bd

…st filesystem (#143) (cherry picked from commit 60dd5c8)

reuse DocValues iterators in SolrDocumentFetcher/udvas (#144)

9bb42b8

port test from upstream PR; fix multivalued reuse bug (#146)

77f2cf6

Added a new metric for delayedInterNodeRequests

90a3dfe

noblepaul closed this Sep 25, 2023

noblepaul deleted the noble/SAI-4604 branch September 25, 2023 14:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SAI-4604 : Add proper metrics around distributed request delay / http2 client bandwidth #148

SAI-4604 : Add proper metrics around distributed request delay / http2 client bandwidth #148

noblepaul commented Sep 25, 2023

SAI-4604 : Add proper metrics around distributed request delay / http2 client bandwidth #148

SAI-4604 : Add proper metrics around distributed request delay / http2 client bandwidth #148

Conversation

noblepaul commented Sep 25, 2023