[BUG] Sigmoid rank function - incorrect ranking explanation

[evgeniya00]

1. Create an index with rank_field field
2. Navigate to Dev Tools and run a _search query (using sigmoid rank feature) with explain parameter set to true (_search?explain=true)
3. GET /_search?explain=true

{
    "query": {
        "bool": {
            "should": [
                {
                    "match_phrase_prefix": {
                        "first_name": ""
                    }
                },
                {
                    "rank_feature": {
                        "field": "<rank_field>",
                        "sigmoid": {
                            "pivot": 3,
                            "exponent": 0.5
                        }
                    }
                }
            ]
        }
    }
}

4. Observe the explanation of sigmoid function ranking:
   Both pivot and exponent are filled with pivot value

Expected behavior
Pivot and exponent are displayed correctly

OpenSearch Version
2.11.1

[ananzh]

@opensearch-project/admin could we redirect this issue to opensearch team?

[peternied]

[Triage - attendees 1 2]
@evgeniya00 Thanks for creating this issue, could you creating pull request with a test case that reproduces this bug so its easy to identify if this is fixed, and/or make a fix for the issue?

[hrishikeshdkakkad]

/assign

[hrishikeshdkakkad]

I can reproduce this.
Can I get additional direction?
Some references? What should be the expected behavior?

cc. @peternied @evgeniya00

[dblock]

This looks like a straightforward bug. Start by writing a REST test for it (https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-the-rest-layer), maybe add to https://github.com/opensearch-project/OpenSearch/blob/95fe9cbabcf9da0a5fad6eafd4092b4284520a9a/modules/mapper-extras/src/yamlRestTest/resources/rest-api-spec/test/rank_features/?

[owaiskazi19]

@evgeniya00 this looks like a bug in lucene. Raised a PR there apache/lucene#13691.

[owaiskazi19]

PR is merged and backported. Should be resolved in 9.12 lucene upgrade.

[sandeshkr419]

Search Triage: We should be upgrading Lucene incorporating this fix as part of OS 2.18.
Closing this now.

Thanks @owaiskazi19 for the fix.

[owaiskazi19]

@hrishikeshdkakkad if later you want to add a REST test for it. Here is something I started and you can continue iterating on it

---
"Search with explain=true":
  - do:
      search:
        index: test
        body:
          query:
            bool:
              should:
                - match_phrase_prefix:
                    title: "Document 1"
                - rank_feature:
                    field: pagerank
                    sigmoid:
                      pivot: 3
                      exponent: 0.5
        explain: true

  - match:
      hits.hits.0._id: "1"
  - match:
      hits.hits.1._id: "2"

  # Check pivot and exponent values in the explanation
  - match:
      hits.hits.0._explanation.details.1.details.1.value: 3.0
  - match:
      hits.hits.0._explanation.details.1.details.2.value: 0.5
  - match:
      hits.hits.1._explanation.details.0.details.1.value: 3.0
  - match:
      hits.hits.1._explanation.details.0.details.2.value: 0.5

# Test teardown
---
teardown:
  - do:
      indices.delete:
        index: test
        ignore: 404