Character normalization for accented characters

[eugene-yang]

Characters with accented characters should be normalized for better matching. Or at least should be an option for user to select.

Here is an example from CLEF.

{"id": "xxx", "lang": "eng", "query": "escap rold\u00e1n escap flight spanish civil guard - director lui rold\u00e1n", "text": "escape rold\u00e1n escape flight spanish civil guard-director luis rold\u00e1n", "report": null} 

[cash]

Do you mean normalize diacritics or remove them? The current code normalizes them: https://github.com/hltcoe/patapsco/blob/master/patapsco/util/normalize.py#L240

[dlawrie]

I think we want to remove diacritics.

On Thu, Jun 23, 2022 at 11:45 AM Cash Costello ***@***.***> wrote: Do you mean normalize diacritics or remove them? The current code normalizes them: https://github.com/hltcoe/patapsco/blob/master/patapsco/util/normalize.py#L240 — Reply to this email directly, view it on GitHub <#46 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJNDOVRQSK25MZVE2W2F3DVQSIDJANCNFSM5ZU2D5TA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

--

…

_________________________________________________ Dawn J. Lawrie Ph.D. Senior Research Scientist Human Language Technology Center of Excellence Johns Hopkins University 810 Wyman Park Drive Baltimore, MD 21211 ***@***.*** https://hltcoe.jhu.edu/faculty/dawn-lawrie/