Korean language support

[intoxicated]

I'm thinking to switch from Algolia, but only if Meilisearch can support Korean language. I'd appreciate if you can consider to add :)

[ManyTheFish]

Hello @intoxicated, for now, Korean Language has no specialized segmentation or normalization in Meilisearch.
That means that the words will be split by whitespaces and we don't make any modifications to Korean characters.
That's been said, during the Hacktoberfest a contributor added a specialized segmenter for the Korean Language so we expect the Korean words to be better split in future versions.
Therefore, if you want to suggest any modifications or share some examples of what you expect in terms of search results, I would be happy to find a solution to enhance it.

[intoxicated]

For instance, nori tokenizer is one of standard tokenizer for korean language in ES. https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-nori-tokenizer.html whitespace tokenizing is definitely not enough for production use imo.

[ManyTheFish]

Hello @intoxicated,
Next week we will release a new version of Meilisearch including a better segmentation of the Korean Language.
You can already try the last beta version v1.0.0-rc.3 now.

Don't hesitate to give us feedback about this change and point out the remaining weaknesses of Meilisearch Korean support in order to help us enhance it in future versions.

Thanks!

[intoxicated]

as version 1.0.0, I reported on #623 that tokenizer behavior is inaccurate how it parses korean sentences. There are mainly two issues,

1. highlighting is inaccurate (containing not exact match but additional characters)
2. exact match doesn't work (i.e. two documents have searchable attribute with content 감독 and search result shows two items when searching with 감 but only one result is hit when searching with 감독

[intoxicated]

This is nori tokenizer example for input "개발자 츄르 입니다"

GET _analyze
{
  "tokenizer": "nori_tokenizer",
  "text" : [
      "개발자 츄르 입니다"
    ]
}
# 실행 결과 
{
  "tokens" : [
    {
      "token" : "개발",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "자",
      "start_offset" : 2,
      "end_offset" : 3,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "츄르",
      "start_offset" : 4,
      "end_offset" : 7,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "이",
      "start_offset" : 8,
      "end_offset" : 11,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "ᄇ니다",
      "start_offset" : 8,
      "end_offset" : 11,
      "type" : "word",
      "position" : 4
    }
  ]
}

[intoxicated]

and not sure this is only for dashboard issue but some contents are modified on the site

I searched for "한국" and content has "한국 축구 국가 대표팀 .." but it appears as

"'한국 축구' 축구 국가 대표팀 ..." where word "축구" is appeared twice with highlight result, but it supposed to be
"'한국' 축구 국가 대표팀 ..."

[ManyTheFish]

Hello @intoxicated,
thank you for your feedback, let's try to make a prototype pimping the behavior of the tokenizer, I'll not have the time to do it this week but maybe next week.

For the highlighting bug, I see that you are using v1.0.0, please use a version higher or equal than v1.0.2, this should fix this specific case.

[intoxicated]

got it, i will test with latest version for highlight issue

[intoxicated]

ok I just briefly tested with v1.0.2 and it seems much better than previous version I used

[intoxicated]

Today I found this issue. All three records in the pictures contain "타자" (meaning typing here) and but when searching "타" or "타자" only shows one result (which record is happened to be prefix with "타자") @ManyTheFish cc