Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace normalization module with frontend-side normalization via libICU #1892

Closed
lonvia opened this issue Jul 27, 2020 · 2 comments
Closed

Comments

@lonvia
Copy link
Member

lonvia commented Jul 27, 2020

The normalization module is one of the parts of Nominatim that make installation and deployment very difficult. It is also outdated and missing a lot scripts. We should get rid of the normalization module and do it in the frontend code instead. There we can easily make use of libICU or similar and will be flexible to add customized normalization modules.

There are two parts in the code where normalization is applied:

  • Import: currently the normalization is done as part of the placex update trigger. To move the normalization out, we need to do the normalization in the indexing code of the nominatim.py module. So for each place we need to get the possible names out of the database, normalize them and reinsert them for further processing in the trigger code.

  • Query: the normalization is the first stage of forward query processing (see make_standard_name()) in lib/Geocode.php.

The tricky part here is that the two parts are written in two different languages. Luckily both have decent support for libICU. Still we either first switch to Python for everything or do the normalization twice for a while.

@otbutz
Copy link
Contributor

otbutz commented Jul 28, 2020

Dealing with normalization in plpython is tricky: giggls/mapnik-german-l10n#53

@lonvia
Copy link
Member Author

lonvia commented May 7, 2021

Done in #2312.

@lonvia lonvia closed this as completed May 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants