[BUG] Flint data source cannot read nested field value

[dai-chen]

What is the bug?

Flint data source always return NULL for nested field value.

How can one reproduce the bug?

Create an OpenSearch index with nested field:

POST nested_index/_doc
{
  "a.b.c": 123
}

GET nested_index/_search
    "hits": [
      {
        "_index": "nested_index",
        "_id": "GDV4OI0BWdGHpYUCQ66r",
        "_score": 1,
        "_source": {
          "a.b.c": 123
        }
      }
    ]

Read it using Flint data source in spark-shell:

spark.read.format("flint").load("nested_index").schema
res6: org.apache.spark.sql.types.StructType = 
StructType(StructField(a,StructType(StructField(b,StructType(StructField(c,LongType,true)),true)),true))

spark.read.format("flint").load("nested_index").show
+----+
|   a|
+----+
|null|
+----+

spark.read.format("flint").load("nested_index").select("a.b.c").show
+----+
|   c|
+----+
|null|
+----+

What is the expected behavior?

Flint data source can read nested field value as expected.

[dai-chen]

Did more test. In OpenSearch, index mapping is the same.

POST nested_index/_doc
{
  "a.b.c": 123
}

POST nested_index_2/_doc
{
  "a": {
    "b": {
      "c": 123
    }
  }
}

# Both index has the same mapping as below
GET nested_index/_mapping
{
  "nested_index": {
    "mappings": {
      "properties": {
        "a": {
          "properties": {
            "b": {
              "properties": {
                "c": {
                  "type": "long"
                }
              }
            }
          }
        }
      }
    }
  }
}

But field value in _source is different (what's given when indexing):

GET nested_index/_search
    ...
    "hits": [
      {
        "_index": "nested_index",
        "_id": "GDV4OI0BWdGHpYUCQ66r",
        "_score": 1,
        "_source": {
          "a.b.c": 123
        }
      }
    ]

GET nested_index_2/_search
    ...
    "hits": [
      {
        "_index": "nested_index_2",
        "_id": "04jYPI0BZG4KSy0O6VBi",
        "_score": 1,
        "_source": {
          "a": {
            "b": {
              "c": 123
            }
          }
        }
      }
    ]