Improve the "number of words per attribute" limit

[curquiza]

Currently, the number of words per attribute is 1000. (We know internally this is not exactly a number of words but a number of positions)
We know some users are annoyed with this limit, see: meilisearch/meilisearch#1604

Checked with @Kerollmops and @ManyTheFish: this limit could be increased to 65536.
The implementation is considered as easy for the core team.

[gmourier]

Thanks @curqui. Do we have any idea of the performance impact this could have on indexing?

Having an option to increase the current default value of 1000 could be interesting if it is considered to have an impact on performance.

[Kerollmops]

It can, indeed, have an effect on indexing performances as more words and positions are processed. More words and positions can have an impact on the size of the database as more positions produce more pair proximities and stuff.

I like the idea to add a new setting like max-indexed-positions 🍏

[gmourier]

Closing the discussion because it has been added in v0.24.0.

This limit is not configurable but reaches 65535 and not 1000 anymore.