Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Size limit of a given field is 32KB #625

Open
A-Gray-Cat opened this issue Sep 6, 2024 · 1 comment
Open

[BUG] Size limit of a given field is 32KB #625

A-Gray-Cat opened this issue Sep 6, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@A-Gray-Cat
Copy link

What is the bug?
* When creating a MV/CI, I encountered the following error:
* A potential workaround: https://discuss.elastic.co/t/error-document-contains-at-least-one-immense-term-in-field/66486
* {"Message":"Fail to run query. Cause: failure in bulk execution:\n[316]: index [flint_securitylake_default_last_7day_ct_yue_mv], id [8no9yJEBJ8hCpANQgrau], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Document contains at least one immense term in field=\"api.response.data\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 82, 101, 99, 111, 114, 100, 115, 34, 58, 91, 123, 34, 70, 105, 110, 100, 105, 110, 103, 73, 100, 101, 110, 116, 105, 102, 105, 101]...', original message: bytes can be at most 32766 in length; got 39932]]; nested: OpenSearchException[OpenSearch exception [type=max_bytes_length_exceeded_exception, reason=max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 39932]];]"}

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Create a MV against a table that has a single field which can go beyond 32KB max limit.

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

  • OS: [e.g. iOS]
  • Version 2.13
  • Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

@A-Gray-Cat A-Gray-Cat added bug Something isn't working untriaged labels Sep 6, 2024
@dai-chen
Copy link
Collaborator

There seems an ignore_above field option to avoid this. But currently Flint doesn't set it automatically or provide a way for users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants