[BUG] Fail to generate embedding for ingest document with nested field defined in field map

[yizheliu-amazon]

What is the bug?

When field map is defined with nested field like "a.b.c": "d.e" for ingest pipeline, the ingestion for document with field in a list of "a.b.c" will fail the ingestion

How can one reproduce the bug?

1. Create ingest pipeline

PUT /_ingest/pipeline/nlp-ingest-pipeline-v4

{
  "description": "text embedding pipeline example",
  "processors": [
    {
      "text_embedding": {
        "model_id": "AMT_KJQBWWsqXNqdQAfI",
        "field_map": {
          "nested_field.level1.level2": "level3.level4Embedding" 
				}
      }
    }
  ]
}

2. Simulate

POST /_ingest/pipeline/nlp-ingest-pipeline-v4/_simulate

{
	"docs": [
		{
			"_index": "neural-search-index-v2",
			"_id": "1",
			"_source": {
				"nested_field.level1": [
					{
						"level2": "hello"
					},
					{
						"level2": "world"
					}
				]
			}
		}
	]
}

3. Result

{
  "docs": [
    {
      "doc": {
        "_index": "neural-search-index-v2",
        "_id": "1",
        "_source": {
          "nested_field": [
	   {
              "level1": {
                  "level2": "hello"
               }
            },
	   {
              "level1": {
                  "level2": "world"
               }
            }
          ]
        },
        "_ingest": {
          "timestamp": "2025-01-02T22:45:34.884624Z"
        }
      }
    }
  ]
}

What is the expected behavior?

Doc in nested_field list should have embedding fields like

{
  "docs": [
    {
      "doc": {
        "_index": "neural-search-index-v2",
        "_id": "1",
        "_source": {
          "nested_field": [
            {
              "level1": {
                  "level2": "hello",
                  "level3.level4Embedding": [
                    0.18129997,
                    -0.056219965,
                   ...
                  ]
               }
            },
            {
              "level1": {
                  "level2": "world",
                  "level3.level4Embedding": [
                    0.0358946,
                    0.04183194,
                   ...
                  ]
               }
            }
          ]
        },
        "_ingest": {
          "timestamp": "2025-01-02T22:51:03.541304Z"
        }
      }
    }
  ]
}

What is your host/environment?

Mac OS

Do you have any screenshots?

N/A

Do you have any additional context?

N/A

[heemin32]

Could be duplicated issue with #686?

[yizheliu-amazon]

No. Issue #686 is for bug where source field is overridden by embedding value, but this issue is about embedding value missing for doc containing list of nested objects.

Actually, given change of PR #1040 in my local workspace, I am not able to reproduce #686 now. It might get closed once PR #1040 is merged.