Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set ingest processor can copy the whole object from one field to another #10134

Closed
gaobinlong opened this issue Sep 20, 2023 · 1 comment
Closed
Labels
enhancement Enhancement or improvement to existing feature or request Indexing & Search Search:Query Capabilities Search Search query, autocomplete ...etc

Comments

@gaobinlong
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Currently, the set ingest processor cannot copy the whole object from one field to another, the copied result is just string, which is not practical, the use case is that I want to copy a subField of one field to another root level field, and then delete the original field or overwrite it, but I cannot do it by set processor. See the following cases:

Case 1:

Request:

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "description": "test",
    "processors": [
        {
          "set": {
            "field": "message",
            "value": "{{message.content}}"
          }
        }
    ]
  },
  "docs" : [
    {
            "_routing":123,
            "_id":1,
            "_source" : {
                "message":{
                  "content":{
                    "foo":1,
                    "bar":"2"
                  }
                }
            }
        }
  ]
}

Response:

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "1",
        "_routing": "123",
        "_source": {
          "message": "{bar=2, foo=1}"
        },
        "_ingest": {
          "timestamp": "2023-09-20T05:47:51.936148084Z"
        }
      }
    }
  ]
}

Case 2

Request:

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "description": "test",
    "processors": [
        {
          "set": {
            "field": "content",
            "value": "{{message.content}}"
          }
        },
        {
          "remove": {
            "field": "message"
          }
        }
    ]
  },
  "docs" : [
    {
            "_routing":123,
            "_id":1,
            "_source" : {
                "message":{
                  "content":{
                    "foo":1,
                    "bar":"2"
                  }
                }
            }
        }
  ]
}

Response:

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "1",
        "_routing": "123",
        "_source": {
          "content": "{bar=2, foo=1}"
        },
        "_ingest": {
          "timestamp": "2023-09-20T05:50:03.367657089Z"
        }
      }
    }
  ]
}

The copied result is just string, not an object.

Describe the solution you'd like
Add some enhancement to make set processor can copy the whole object from one field to another, no matter what the data type is, like json object, array, integer, float, all of these types can be copied and keep the same.

Describe alternatives you've considered
Use script processor to do that but it's not easy to use.

@gaobinlong
Copy link
Collaborator Author

Closed by #11870.

@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Search Project Board Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing & Search Search:Query Capabilities Search Search query, autocomplete ...etc
Projects
Archived in project
Development

No branches or pull requests

2 participants