Elastic Search Index going over field limit

[ashabbir]

It might be happening from Headers.
we can convert headers into a string field and use it as index would resolve this issue

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node-1][10.132.22.96:9300][indices:data/write/update[s]]"}],"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [ache-data] has been exceeded"}

but please make sure it doesnt change the kafka format

[aecio]

I think you mean to not change the ELASTIC format in KAFKA, and if so, that is not possible. I can't change one without changing the other since they are serialized using the same code.

[ashabbir]

ops ya Do not change Elastic Format please

[aecio]

We will need to fix this and change the format before the next version release. For now, changing Elasticsearch setting to increase the index.mapping.total_fields.limit might work as a temporary workaround:

curl -XPUT 'http://localhost:9200/ache-data/_settings' -d '
{
  "index.mapping.total_fields.limit": 5000
}'