Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gsub Processor Behavior and \n Sequence Issues #170904

Closed
fabs-elastic opened this issue Nov 8, 2023 · 1 comment · Fixed by #175832
Closed

Gsub Processor Behavior and \n Sequence Issues #170904

fabs-elastic opened this issue Nov 8, 2023 · 1 comment · Fixed by #175832
Assignees
@ElenaStoeva
Labels
bug Fixes for quality problems that affect the customer experience Feature:Ingest Node Pipelines Ingest node pipelines management Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more

Comments

@fabs-elastic
Copy link

Kibana version: 8.10.4

Elasticsearch version: 8.10.4

Server OS version:

Browser version: Google Chrome Version 119.0.6045.105

Browser OS version: Mac OS Sonoma 14.1.1

Original install method (e.g. download page, yum, from source, etc.): Download page

Describe the bug:

  1. Gsub Processor Behavior in Console vs. Dev Tools:

  • An Ingest Pipeline was created using the Kibana Dev Tools with a Gsub processor intended to replace newline characters.

  • When checked in the Kibana Console, the \n sequence wasn't displayed as expected, as shown in the attached screenshot (ingest_pipeline_00.png).

image

  1. Issue with Adding \n through Console:
  • Adding the \n sequence directly through the Kibana Console resulted in it being displayed as \n in the API call, indicating a literal backslash followed by 'n' instead of a newline (ingest_pipeline_01.png).
  1. Dev Tools Workaround:
  • A workaround involved using the Kibana Dev Tools to replace \n with \n, aiming to replace a literal \n with an actual newline \n, not another \n.

Steps to reproduce:

  1. Access Kibana (http://localhost:5601/) and navigate to "Dev Tools".
  2. Create an Ingest Pipeline with a Gsub processor using the provided PUT request to replace \n with \n.
    PUT _ingest/pipeline/my_pipeline { "description": "My test pipeline", "processors": [ { "gsub": { "field": "message", "pattern": "\\\\n", "replacement": "\n" } } ] }
  3. Fetch the created pipeline using the GET request and observe the Gsub processor settings.
  4. Navigate to "Stack Management" > "Ingest Node Pipelines" and find the created pipeline (my_pipeline). Observe the Gsub processor settings here.
  5. Attempt to edit the pipeline in this UI, using \n in a replacement.
  6. Save the pipeline and check it again in Dev Tools to see if \n appears as \n.
@fabs-elastic fabs-elastic added the bug Fixes for quality problems that affect the customer experience label Nov 8, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Nov 8, 2023
@alisonelizabeth alisonelizabeth added the Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more label Nov 8, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Nov 8, 2023
@alisonelizabeth alisonelizabeth added Feature:Ingest Node Pipelines Ingest node pipelines management needs-team Issues missing a team label labels Nov 8, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label Nov 8, 2023
@ElenaStoeva ElenaStoeva self-assigned this Jan 24, 2024
@ElenaStoeva ElenaStoeva mentioned this issue Jan 29, 2024
Merged
ElenaStoeva added a commit that referenced this issue Feb 1, 2024
@ElenaStoeva
[Ingest Pipelines] Fix gsub processor's replacement field serializati…
585630d 
…on (#175832)

Fixes #170904

## Summary

This PR adds serialization to the `replacement` field in the Gsub
processor edit form so that the field renders the text the way it is
provided in the Es request, with unescaped special characters, if any
exist. For example, we want `\n` to render the same, not to render as a
new line (empty string) in the `replacement` field in the UI.

### Testing


**Deserialization:**
1. Open Dev tools and create an ingest pipeline:
```
PUT _ingest/pipeline/my_pipeline_1
{ 
  "description": "My test pipeline",
  "processors": [
    {
      "gsub": {
        "field": "message",
        "pattern": "\\\\n",
        "replacement": "\n"
      }
    }]
}
```
3. Run `GET _ingest/pipeline/my_pipeline_1` and observe the
`replacement` fields (should be displayed as a new line).
4. Go to Stack Management -> Ingest Pipelines and start editing
`my_pipeline_1`. The `replacement` field in the edit form should render
as `\n` instead of an empty string.


**Serialization:**
1. Go to Stack Management -> Ingest Pipelines and create a new pipeline
`my_pipeline_2` with a Gsub processor on the field `message`, pattern:
`\\\\n`, and replacement: `\n`. Save the pipeline
2. Go to Dev tools and get the pipeline: `GET
_ingest/pipeline/my_pipeline_2`. The `replacement` field in the response
should be displayed as a new line:
```
{
  "my_pipeline": {
    "description": "My test pipeline",
    "processors": [
      {
        "gsub": {
          "field": "message",
          "pattern": """\\n""",
          "replacement": """
"""
        }
      }
    ]
  }
}
```
3. Edit the pipeline in the UI and change the `replacement` field to
`\\n`.
4. Now the response in Dev tools should display `\n` (the new line
character was escaped).

You can also test with other escape characters. For example, creating a
pipeline with a `my\ttab` replacement field in the UI should display `my
tab` in the response.
fkanout pushed a commit to fkanout/kibana that referenced this issue Feb 7, 2024
@ElenaStoeva @fkanout
[Ingest Pipelines] Fix gsub processor's replacement field serializati…
56ec38a 
…on (elastic#175832)

Fixes elastic#170904

## Summary

This PR adds serialization to the `replacement` field in the Gsub
processor edit form so that the field renders the text the way it is
provided in the Es request, with unescaped special characters, if any
exist. For example, we want `\n` to render the same, not to render as a
new line (empty string) in the `replacement` field in the UI.

### Testing


**Deserialization:**
1. Open Dev tools and create an ingest pipeline:
```
PUT _ingest/pipeline/my_pipeline_1
{ 
  "description": "My test pipeline",
  "processors": [
    {
      "gsub": {
        "field": "message",
        "pattern": "\\\\n",
        "replacement": "\n"
      }
    }]
}
```
3. Run `GET _ingest/pipeline/my_pipeline_1` and observe the
`replacement` fields (should be displayed as a new line).
4. Go to Stack Management -> Ingest Pipelines and start editing
`my_pipeline_1`. The `replacement` field in the edit form should render
as `\n` instead of an empty string.


**Serialization:**
1. Go to Stack Management -> Ingest Pipelines and create a new pipeline
`my_pipeline_2` with a Gsub processor on the field `message`, pattern:
`\\\\n`, and replacement: `\n`. Save the pipeline
2. Go to Dev tools and get the pipeline: `GET
_ingest/pipeline/my_pipeline_2`. The `replacement` field in the response
should be displayed as a new line:
```
{
  "my_pipeline": {
    "description": "My test pipeline",
    "processors": [
      {
        "gsub": {
          "field": "message",
          "pattern": """\\n""",
          "replacement": """
"""
        }
      }
    ]
  }
}
```
3. Edit the pipeline in the UI and change the `replacement` field to
`\\n`.
4. Now the response in Dev tools should display `\n` (the new line
character was escaped).

You can also test with other escape characters. For example, creating a
pipeline with a `my\ttab` replacement field in the UI should display `my
tab` in the response.
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@ElenaStoeva @CoenWarmer
[Ingest Pipelines] Fix gsub processor's replacement field serializati…
6d6a001 
…on (elastic#175832)

Fixes elastic#170904

## Summary

This PR adds serialization to the `replacement` field in the Gsub
processor edit form so that the field renders the text the way it is
provided in the Es request, with unescaped special characters, if any
exist. For example, we want `\n` to render the same, not to render as a
new line (empty string) in the `replacement` field in the UI.

### Testing


**Deserialization:**
1. Open Dev tools and create an ingest pipeline:
```
PUT _ingest/pipeline/my_pipeline_1
{ 
  "description": "My test pipeline",
  "processors": [
    {
      "gsub": {
        "field": "message",
        "pattern": "\\\\n",
        "replacement": "\n"
      }
    }]
}
```
3. Run `GET _ingest/pipeline/my_pipeline_1` and observe the
`replacement` fields (should be displayed as a new line).
4. Go to Stack Management -> Ingest Pipelines and start editing
`my_pipeline_1`. The `replacement` field in the edit form should render
as `\n` instead of an empty string.


**Serialization:**
1. Go to Stack Management -> Ingest Pipelines and create a new pipeline
`my_pipeline_2` with a Gsub processor on the field `message`, pattern:
`\\\\n`, and replacement: `\n`. Save the pipeline
2. Go to Dev tools and get the pipeline: `GET
_ingest/pipeline/my_pipeline_2`. The `replacement` field in the response
should be displayed as a new line:
```
{
  "my_pipeline": {
    "description": "My test pipeline",
    "processors": [
      {
        "gsub": {
          "field": "message",
          "pattern": """\\n""",
          "replacement": """
"""
        }
      }
    ]
  }
}
```
3. Edit the pipeline in the UI and change the `replacement` field to
`\\n`.
4. Now the response in Dev tools should display `\n` (the new line
character was escaped).

You can also test with other escape characters. For example, creating a
pipeline with a `my\ttab` replacement field in the UI should display `my
tab` in the response.
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@ElenaStoeva @fkanout
[Ingest Pipelines] Fix gsub processor's replacement field serializati…
06fb633 
…on (elastic#175832)

Fixes elastic#170904

## Summary

This PR adds serialization to the `replacement` field in the Gsub
processor edit form so that the field renders the text the way it is
provided in the Es request, with unescaped special characters, if any
exist. For example, we want `\n` to render the same, not to render as a
new line (empty string) in the `replacement` field in the UI.

### Testing


**Deserialization:**
1. Open Dev tools and create an ingest pipeline:
```
PUT _ingest/pipeline/my_pipeline_1
{ 
  "description": "My test pipeline",
  "processors": [
    {
      "gsub": {
        "field": "message",
        "pattern": "\\\\n",
        "replacement": "\n"
      }
    }]
}
```
3. Run `GET _ingest/pipeline/my_pipeline_1` and observe the
`replacement` fields (should be displayed as a new line).
4. Go to Stack Management -> Ingest Pipelines and start editing
`my_pipeline_1`. The `replacement` field in the edit form should render
as `\n` instead of an empty string.


**Serialization:**
1. Go to Stack Management -> Ingest Pipelines and create a new pipeline
`my_pipeline_2` with a Gsub processor on the field `message`, pattern:
`\\\\n`, and replacement: `\n`. Save the pipeline
2. Go to Dev tools and get the pipeline: `GET
_ingest/pipeline/my_pipeline_2`. The `replacement` field in the response
should be displayed as a new line:
```
{
  "my_pipeline": {
    "description": "My test pipeline",
    "processors": [
      {
        "gsub": {
          "field": "message",
          "pattern": """\\n""",
          "replacement": """
"""
        }
      }
    ]
  }
}
```
3. Edit the pipeline in the UI and change the `replacement` field to
`\\n`.
4. Now the response in Dev tools should display `\n` (the new line
character was escaped).

You can also test with other escape characters. For example, creating a
pipeline with a `my\ttab` replacement field in the UI should display `my
tab` in the response.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ElenaStoeva ElenaStoeva

Labels
bug Fixes for quality problems that affect the customer experience Feature:Ingest Node Pipelines Ingest node pipelines management Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Ingest Pipelines] Fix gsub processor's replacement field serialization ElenaStoeva/kibana
4 participants
@alisonelizabeth @elasticmachine @ElenaStoeva @fabs-elastic