[BUG] Search request with PIT + searchAfter and 2 sort fields loses some hits

[Anastasiia186]

What is the bug?

Executing a search request with a PIT + searchAfter and 2 sort fields (first one is not uniq, second one is uniq) returns less hits in total that for track_total_hits. The smaller the page size, the more data is lost, if you use scrolling, there is no problem. if you change the sorting values (only 1 uniq or first one is uniq and second one is not uniq), the problem also disappears

How can one reproduce the bug?

create a search request with a PIT and 2 sort fields like

 PointInTimeBuilder pointInTimeBuilder = new PointInTimeBuilder(pitId);
 SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
                .pointInTimeBuilder(pointInTimeBuilder);
        searchSourceBuilder.query(<query>);
        searchSourceBuilder.sort(SortBuilders
                .fieldSort(<not uniq field>)
                .order(SortOrder.ASC));
        searchSourceBuilder.sort(SortBuilders
                .fieldSort(<uniq fiels>)
                .order(SortOrder.ASC));
        searchSourceBuilder.size(10_000)

run this query several times using searchAfter like:

searchSourceBuilder.searchAfter(prevSearch.getHits().getHits()[prevSearch.getHits().getHits().length - 1].getSortValues());
SearchRequest searchRequest = new SearchRequest();
searchRequest.source(searchSourceBuilder);
restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT)

and notice that in total you have less hits that returns in track_total_hits:
total using PIT = 1999953
"total": {
"value": 2000000,
"relation": "eq"
}

What is the expected behavior?

A search response in total returns the same number of hits as for track_total_hits

What is your host/environment?

SpringBoot application

[Xtansia]

@opensearch-project/triage Please transfer this to the core repo as this pertains to behaviour of OpenSearch itself (and example is using RHLC).

[msfroh]

[Search Triage] This one needs some investigation. With the unique second sort field acting as a tie-breaker, there should be a total order on the documents, so the search_after should continue exactly after the previous doc. One possibility is that the second field may not be in all documents, such that "missing" is a duplicated value.

@Anastasiia186 -- can you please check if the second field is present in all documents? Also, are you able to produce a minimal test case that reproduces the issue? Thanks!

[Anastasiia186]

@msfroh yes, we rechecked the second field and found some duplicates...we removed them and now it works fine
thank you!