Merge branch_9x into fs/branch_9x

[patsonluk]

Closing this

Using #170 instead, as this PR merge beyond the merge-base of 9x and 9.4, which could complicate our future work.

Attempt to periodically update our fs/branch_9x to be in-sync with branch_9x from apache

Highlights

Http2SolrClient

In the process of resolving conflicts for Http2SolrClient, added the asyncListener to trigger onStart of request.

However, this resulted as some pretty weird method signature becomes weird (need to add extra param at the end at here and here. Do we really need asyncListener to listen onBegin anymore?

CircuitBreaker

Our existing fs/branch_9x seems to have older changes on CircuitBreakerManager which is deprecated and restrucuted in the latest branch_9x. Taking changes from fs/branch_9_3 (newer changes on the CircuitBreaker directly) instead

Node level cache

Since our fs version CacheConfig.newInstance(core) takes a core as the param, the newly added node level cache call CacheConfig.newInstance() in CoreContainer is flagged as missing arg.

Based on the code in CacheConfig#newInstance, it should be able to handle null org (ie node level cache should not expect a core anyway)

Reuse DocValues iter in SolrDocumentFetcher

The current fs/branch_9x version seems to be missing the updates in fs/branch_9_3 of #144 and #146

Cherry picked #144 and #146 from fs/branch_9_3 to fs/branch_9_x, this was necessary as similar changes are also in branch_9x, which the merge had some conflicts.

Lucene version

Lucene reference has been updated from 9.7.0 to 9.8.0.

Lucene fs/branch_9_8

We do not yet have such branch, followed steps in https://www.notion.so/OSS-Solr-Development-Guide-41a77cf7bc7246d0bf56177e6986c174?pvs=4#7dc4bc1c52ad40c5ad33d77aba46789b (but replaced with lucene) to create a new fs/branch_9_8 branch for fullstory lucene

Gradle version conflict

Solr branch 9_x now uses gradle 8.4.0, however Lucene is still using gradle 7.6.0. Manually updated Lucene to use 8.4.0. Lucene builds fine when using ./gradew build on it with 8.4.0

For completeness, also created branch_9_8 in our repo

Outstanding issue

Commits in fs/branch_9_3 that are not in fs/branch_9x

Below is the list of commits (local to FS) that appear in fs/branch_9_3 but not fs/branch_9x, we might want to consider including them later on in fs/branch_9x if applicable:

• SOLR-16966: Add a first-class cache for OrdinalMaps (introduce first-class cache for OrdinalMaps #119) 2023-09-12, 8:42 a.m. 1246c69
• minor ordMapCache fixes: ramBytes accounting and a test fix (minor ordMapCache fixes: ramBytes accounting and a test fix #142) 2023-09-14, 4:17 a.m. bcfbd4b
• reuse DocValues iterators in SolrDocumentFetcher/udvas (reuse DocValues iterators in SolrDocumentFetcher/udvas #144) 2023-09-15, 7:53 a.m. 9bb42b8 (cherry picked in this PR)
• port test from upstream PR; fix multivalued reuse bug (fix multivalued dv iterator reuse #146) 2023-09-20, 10:23 a.m. 77f2cf6 (cherry picked in this PR)
• SOLR-16962: Restore ability to configure tlog directory (an alternate approach to fixing tlog dir handling #132) 2023-09-26, 11:56 a.m. f4cee66

Build issue

There's current build issue with ./gradlew build . In order to build , we need to skip several tasks ./gradlew build -x :solr:documentation:collectLuceneJavadocs -x :solr:solr-ref-guide:buildLocalSite -x :solr:solr-ref-guide:checkSiteLinks -x :checkUnusedConstraints -x :verifyLocks -x licenses. There's still one test case failure, see next section

This might be okay as I see our build checks also skip those tasks (???) need to double-confirm

Unit test case NPE on RealTimeGetComponent

With test of
./gradlew :solr:core:test --tests "org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testRTGCompositeRouterWithRouterField" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=1EC92724168C5FD -Ptests.file.encoding=ISO-8859-1

It failed with

  2> 74653 ERROR (qtp2033335124-56-null-17) [n:127.0.0.1:57457_solr c:composite_collection_with_routerfield_1 s:shard1 r:core_node6 x:composite_collection_with_routerfield_1_shard1_replica_n2 t:null-17] o.a.s.s.HttpSolrCall 500 Exception
  2>           => java.lang.NullPointerException
  2>    at org.apache.solr.handler.component.RealTimeGetComponent.createSubRequests(RealTimeGetComponent.java:1058)
  2> java.lang.NullPointerException: null
  2>    at org.apache.solr.handler.component.RealTimeGetComponent.createSubRequests(RealTimeGetComponent.java:1058) ~[main/:?]

Take note that this is a FullSolrCloudDistribCmdsTest.testRTGCompositeRouterWithRouterField is a new test method recently introduced.