[BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results

[kristib]

What is the bug?
We have a cluster with about 60 million documents spread across weekly indexes. The weekly indexes are all added to a "documents" alias. Each weekly index has about 1 million documents, and each doc has a faiss hnsw knn_vector field with 256 dimensions.

We upgraded to OpenSearch 2.17 last week, and some of our k-NN queries no longer return hits. We have isolated the issue to k-NN queries (that are using efficient filtering) that have a range filter for a date field.

Here is the vector field (embeddings.OAI_TE3L_256) mapping:

"embeddings": {
          "properties": {
            "OAI_TE3L_256": {
              "type": "knn_vector",
              "dimension": 256,
              "method": {
                "engine": "faiss",
                "space_type": "l2",
                "name": "hnsw",
                "parameters": {
                  "m": 32
                }
              }
            }

Here is an example query that returns 0 hits:

POST documents/_search
{
  "from": 0,
  "query": {
    "knn": {
      "embeddings.OAI_TE3L_256": {
        "vector": [
          -0.05308306,
          -0.040473435,
          -0.020121736,
          -0.024970116,
          0.017065149,
          0.0007180346,
          -0.041048344,
          0.025315061,
          -0.0696787,
          -0.04986358,
          0.11812417,
          0.004086629,
          -0.121267,
          -0.03683236,
          -0.063508034,
          -0.10087697,
          -0.1159012,
          -0.032597214,
          0.020332536,
          -0.103329904,
          0.15353844,
          -0.05373462,
          -0.000034284803,
          -0.03058504,
          0.04346295,
          0.038748715,
          -0.088458985,
          0.03915115,
          0.066957474,
          0.14510646,
          0.029665189,
          -0.0040171607,
          -0.079260476,
          -0.022766307,
          -0.06937208,
          0.0005449637,
          0.1864231,
          -0.054577816,
          0.010750757,
          0.038729552,
          0.06327807,
          0.048292167,
          0.015273356,
          0.0040890244,
          -0.07489119,
          -0.009126645,
          0.033459574,
          -0.011258591,
          -0.023954447,
          0.009629688,
          -0.03322961,
          -0.03882537,
          -0.070905164,
          0.026349893,
          0.121267,
          -0.096431024,
          0.0071336348,
          0.13782431,
          -0.034226116,
          -0.10509296,
          0.021673985,
          -0.038729552,
          -0.016672296,
          0.061859965,
          -0.06864387,
          0.12824254,
          0.030086787,
          -0.05423287,
          0.004139329,
          -0.046644103,
          0.0077324957,
          0.08370642,
          -0.091985084,
          0.084856234,
          0.017975418,
          0.0033632044,
          0.011910152,
          -0.050630122,
          0.116131164,
          -0.08439631,
          -0.030201769,
          -0.06734075,
          0.104862995,
          0.04043511,
          0.056379188,
          0.0061515025,
          0.064006284,
          -0.23870128,
          0.03117911,
          -0.0060892208,
          0.03867206,
          -0.04162325,
          0.00463279,
          -0.11774091,
          0.0556893,
          -0.07496784,
          -0.0155608095,
          -0.0087960735,
          -0.0011372374,
          -0.0022026114,
          -0.03468604,
          -0.0071384255,
          -0.010175849,
          -0.08907222,
          0.08255661,
          -0.064159594,
          -0.06829892,
          -0.056417517,
          -0.043386295,
          -0.023494523,
          -0.022517182,
          0.00840322,
          -0.038767878,
          0.054616142,
          -0.08439631,
          0.06262651,
          -0.027633851,
          -0.03644909,
          0.039017007,
          -0.05047681,
          0.026714,
          -0.045954213,
          0.10011043,
          -0.03150489,
          -0.026503202,
          0.1031766,
          -0.05756733,
          0.029933479,
          0.060135245,
          -0.17323858,
          -0.04150827,
          0.043194663,
          -0.09367147,
          -0.033574555,
          0.0029607697,
          0.032482233,
          0.034283604,
          0.010607029,
          -0.07044524,
          -0.021942275,
          -0.080486946,
          0.036219127,
          0.021348204,
          -0.005686786,
          -0.074086316,
          -0.041776557,
          -0.033267938,
          -0.056915767,
          -0.07719082,
          0.07742078,
          -0.1773779,
          -0.058295544,
          -0.052431498,
          0.030182606,
          -0.013251601,
          -0.0072294525,
          0.0696787,
          -0.042849716,
          0.019163558,
          0.061131753,
          0.074507914,
          0.095511176,
          -0.039285295,
          0.100187086,
          0.03825046,
          -0.02579415,
          0.13245852,
          0.0035692127,
          -0.022919616,
          0.11375488,
          -0.031485725,
          -0.08133014,
          -0.051818263,
          -0.07059855,
          -0.00053478306,
          0.00737797,
          0.056340862,
          -0.11697436,
          0.071556725,
          0.0053514238,
          0.1167444,
          0.042888045,
          0.025123425,
          -0.0140852155,
          -0.020351699,
          0.0119963875,
          -0.0105207935,
          -0.031428237,
          0.012695857,
          -0.009773415,
          -0.031370744,
          -0.013558217,
          -0.03526095,
          -0.03656407,
          -0.0095913615,
          0.02100326,
          0.053351346,
          0.05825722,
          -0.006208993,
          -0.03665989,
          0.07941379,
          -0.09213839,
          -0.067762345,
          -0.058218893,
          0.07174836,
          0.09443802,
          0.09006873,
          -0.006343138,
          -0.0071911253,
          -0.12555964,
          0.025698332,
          0.014554723,
          0.07339643,
          0.0067072455,
          0.02454852,
          0.030144278,
          0.01432476,
          -0.025947459,
          0.0651561,
          0.039074495,
          0.021463186,
          -0.040090166,
          -0.025104262,
          -0.04346295,
          -0.08753914,
          -0.028400393,
          0.09505125,
          0.14012393,
          -0.055804282,
          0.013059965,
          -0.022306383,
          -0.0034662087,
          -0.0038973887,
          -0.120807074,
          -0.06281815,
          -0.047525626,
          0.009073945,
          0.04411451,
          -0.062281564,
          -0.055114396,
          -0.0583722,
          0.04032013,
          0.00021514088,
          -0.018780287,
          -0.019048577,
          -0.03284634,
          -0.06856721,
          0.069985315,
          0.056839116,
          -0.02127155,
          0.06140004,
          0.028745338,
          -0.0561109,
          0.033363756,
          -0.025621679,
          0.06872052
        ],
        "k": 1000,
        "filter": {
          "bool": {
            "filter": [
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "documentType.keyword": "RAW"
                      }
                    },
                    {
                      "term": {
                        "contentLanguage.keyword": "English"
                      }
                    },
                    {
                      "exists": {
                        "field": "attachment"
                      }
                    },
                    {
                      "match_phrase_prefix": {
                        "topic.country": "Australia"
                      }
                    },
                    {
                      "range": {
                        "dateReceived": {
                          "gte": "2024-12-12T12:30:47.000Z"
                        }
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      }
    }
  }
}

The filtered subset for this query should result in 968 docs, which is less than k, and I believe should result in an exact knn search through the efficient knn filtering algorithm, however it does not seem like we are seeing this behavior. Instead, we are just getting 0 hits.

Overall, we are seeing inconsistent query results. For example:

• If we run this query on a single weekly index rather than the entire documents alias, we do get expected hits back.
• If we remove the range portion of the knn query filter, we get expected hits back
• If we move the range filter outside of the knn query filter (and transform the query into a knn post-filter instead), we do get hits back
• If we use a knn filter range query for a non-date field (such as a float field), we get expected hits back

What is the expected behavior?
K-NN efficient filtering should work as described https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/ for knn range filters on a date field.

What is your host/environment?

• Opensearch 2.17
• k-NN plugin

[heemin32]

Duplication of #2222
This issue is happening without nested field

[navneet1v]

@kristib did you try reproducing on a smaller index with like 10-20 docs? We can try reproducing this with an example index.

Also when you say you upgraded the index, can you please tell from which version of Opensearch you upgraded from?