[RFC] Provide pluggable deciders for concurrent search

[Gankris96]

Is your feature request related to a problem? Please describe

With #14781 we suggested a adaptive approach to use concurrent search based on resource utilization (CPU) and request type. One of the complexity with this adaptive approach is it could lead to a non-deterministic behavior in the system if CPU load varies. With shared resources between other tasks like indexing, or background jobs like snapshots, the increase in CPU utilization will lead to penalizing search request. To keep it simple and deterministic behavior, we are planning to provide a mechanism such that it is easy to configure and still targets specific use cases where we have seen good improvements.

We will introduce a new setting which is flexible and can be extended for other use cases later. We will deprecate the existing concurrent search setting keeping bwc in mind. Instead of having a binary setting we need a way to control using concurrent search based on a request type. Along with this we will provide a pluggable decision making component such that plugins like KNN can also decide for the request type exposed by it.

Describe the solution you'd like

Today we have the following 2 settings at cluster and index level

cluster: 
"search.concurrent_segment_search.enabled": true

index:
"index.search.concurrent_segment_search.enabled": true

The plan for OS_2_17 will be to introduce a new setting which will take one of the following 3 values:

cluster:
search.concurrent_segment_search.mode: ["all", "none", "auto"]

index:
index.search.concurrent_segment_search.mode: ["all", "none", "auto"]

and deprecate the existing boolean settings with bwc.

The meaning of the values for new settings is as follows:

Core decider behavior:

In the back-end we will control what auto mode means for the setting

all → maps to current behavior of setting the value to true. All requests will run with concurrent segment search
none → maps to current behavior of setting the value to false. No request will run with concurrent segment search
auto → only enable concurrent search for aggs. (this applies to index level or cluster level based on what is set. Index level takes precedence over cluster level, for eg. if cluster level is auto and index level is none, then for that index, concurrent search will be disabled and similarly when cluster level setting is none, but index level setting is auto then that takes effect.)

Decision making with Decider logic only comes into picture when the concurrent search mode is auto

Default value in 2.17 →

• Both cluster level and index level setting will continue to remain none which corresponds to the same behavior as today where the boolean setting is defaulted to false. However, if user updates the setting to auto then concurrent search will be enabled on queries with agg operations.

For future releases, we can think of adding support for more operations with auto setting.

Plugin decider behavior:

Now plugins are only concerned with the indices that are relevant in plugin world, for example, knn plugin is only interested in taking the decision for knn index.

There is no plugin-specific setting to indicate a preference of behavior as different plugins may have different default behaviors. Instead, plugins will also follow what is set in above cluster and index level settings; with a slight change in meaning of auto setting.

In plugin world auto can have different meaning:

For example:
In knn plugin world - auto can mean enable concurrent search for all vector operations on the knn index.

In another plugin, the auto setting can mean disable concurrent search for the plugin index.

In all cases, if there is an index specific override via index setting, then take that into account.

So the decision making based on the settings will be controlled by specific plugin-backend using the pluggable deciders.

Related component

Search:Performance

Additional context

Low level implementation details:

Pluggable deciders for concurrent segment search:

Today, the decision to enable concurrent search is made in DefaultSearchContext#evaluateRequestShouldUseConcurrentSearch where the decision is made by looking at cluster and index settings along with few other conditions.

While we continue to use this logic, we will now use a decider framework to provide a decision on concurrent segment search.

SearchPlugin will provide an additional method to create concurrent search decider that plugins will implement to provide their plugin specific decision logic.
SearchModule will register the core and plugin specific deciders into a concurrent search deciders Collection, which can be iterated over to get the decision for concurrent search for each request.
SearchService#evaluateRequestShouldUseConcurrentSearch will now iterate over all registered deciders to provide the decision.

Decider contract will be to take in params required for decision making and provide one of the following decisions:

1. noop (don’t care since this index is not specific to this plugin)
2. true (this is a valid plugin specific index and the decision is YES for concurrent search)
3. false (this is a valid plugin specific index and the decision is NO for concurrent search)

Overall decision flow will be to go over all deciders one by one and collect the decisions. If all the plugins return noop decision then we will fallback to core decider.

[Update] For implementation details see: #15259 (comment)

Co-author: @sohami

[Gankris96]

Updating this with some modifications to the approach.

Modification:

• Implement a visitor to traverse the QueryBuilder tree for the specific operation types

Motivation here is that there will be some deciders that may be interested in only certain types of query operations; for example the knn plugin will only be interested in queries with vector operations. In order to understand what type of query operations are within the query, we need to traverse the query tree. This is achieved using the visitor pattern. At each level of the query tree, the visitor will call all registered deciders to get decision. Deciders can implement mechanisms to look at the queryBuilder at the current level and provide a decision True to enable concurrent search, False, to disable concurrent search or NOOP if it doesn't care about the operation.

how can plugins register their deciders

Each plugin that implements the SearchPlugin can now implement the method getConcurrentSearchRequestDeciderFactories to create and return a ConcurrentSearchRequestDeciderFactory that will be used at each search request to create a new ConcurrentSearchRequestDecider object. This decider object is then used to obtain a concurrent search decision for the search request.

All registered factories will be invoked to create the required decider objects, and then their respective decisions will be collected and combined to obtain the final concurrent search decision for the request.

Plugins can create their plugin specific deciders to extend the abstract class ConcurrentSearchRequestDecider and implement the Factory interface to create corresponding decider objects

public interface Factory {
        default Optional<ConcurrentSearchRequestDecider> create(IndexSettings indexSettings) {
            return Optional.empty();
        }
    }

and also implement the 2 methods that will be used in the decision making process.

public abstract void evaluateForQuery(QueryBuilder queryBuilder, IndexSettings indexSettings);

public abstract ConcurrentSearchDecision getConcurrentSearchDecision();

create --> this method is part of the Factory interface that can be implemented to create the corresponding ConcurrentSearchRequestDecider objects. The indexSettings passed in can help plugin deciders choose to not create any decider if plugin is not interested in decision making for current request. The idea is to avoid additional processing/decision making if some deciders are not interested in decision making for certain requests

evaluateForQuery --> This is called from the visitor ConcurrentSearchDecisionVisitor to be able to visit each node of the queryBuilder tree and check whether a decision needs to be made/updated. The expectation is that classes extending the ConcurrentSearchRequestDecider will keep some sort of state within the object to assist in decision making.

getConcurrentSearchDecision --> This is called after the queryBuilder tree is parsed and evaluateForQuery is called on each node of the tree. This method should return the final decision for concurrent search.

These deciders are then collected and used for decision making within the evaluateRequestShouldUseConcurrentSearch method.

Decision making only comes into picture when the concurrent search mode is auto.

As part of the decision making process in auto mode,
There are 2 steps:

1. the first step is to create the ConcurrentSearchRequestDecider objects using the registered factories. (factories can opt to not return any decider object based on the passed in indexSettings to the ConcurrentSearchRequestDecider.Factory#create method.
2. The second step is only carried out for deciders that created in step 1. As part of this, the query builder tree will be traversed with the visitor pattern and at each level (node of the tree), the evaluateForQuery for all the deciders will be called. After traversal, the getConcurrentSearchDecision method of the deciders will be called to obtain the final concurrent search decision.

To get the final decision from plugins, the collected decisions will be combined with AND condition

1. NOOP AND TRUE → TRUE
2. NOOP AND FALSE → FALSE
3. NOOP AND NOOP → NOOP

Iff final decision from plugin deciders is NOOP, then the core decider's logic of checking for presence of supported aggregations in the query will come into picture to get the decision.