buxfix(i18n): read utf8 resource if needed. #190

[khajavi]

No description provided.

[csokol]

I think this code is quite big and complex, I think we should extract it to a separate class to be able to unit test it easier, what do you think @khajavi?

[csokol]

@khajavi thanks for the contribution!

I suppose you've tested this modifications in your machine, but is hard to accept such a complex PR without unit tests.

I've just finished reviewing it and I think you should extract some classes to be able to unit test it easier. What do you think? Maybe some simple tests with properties files and different encodings?

What do you think @xdarklight @JamesSullivan @leocwolter?

[xdarklight]

The standard solution for this issue is to use unicode characters (\uXXXX).
You can use a JDK tool to convert your UTF-8 encoded Persian translation file into ISO-8859-1 with unicode characters with native2ascii (comes with a JDK of your choice): native2ascii -encoding UTF-8 text_utf8.properties text.properties

The downside of this "standard solution" is that the localization file is not editable with a standard text editor anymore (because all non-ASCII characters are encoded), but there are many tools out there - (Attesoro for example - which assist you with editing properties files: they present you the human readable (Persian) text while saving it as ISO-8859-1 with unicode characters.
This works not only for translations but also for the configuration/route properties (with your solution we would have to adjust route and configuration parsing code as well).

I'm not sure what we want to do in Mamute as both solutions have their pros and cons.
But in any case Mamute will get Persian translation thanks to @khajavi 👍

[JamesSullivan]

I think @xdarklight has summed it up quite nicely. If forced to choose I would go with the standard solution, using Unicode characters (\uXXXX), to keep things as simple as possible.

[khajavi]

Writing Unicode translations in ISO-8859-1 standard is cumbersome. I promote to use utf8 bundles for languages that need Unicode strings.

@xdarklight
could you please explain this issue?

This works not only for translations but also for the configuration/route properties (with your solution we would have to adjust route and configuration parsing code as well).

what is the problem with routing and configuration properties?

[xdarklight]

@khajavi see Environment implementation - it's not using MamuteLocalization to load the configuration properties.
I haven't checked the routes implementation but I'm guessing that it's similar there.
You would have to patch everything that loads property files to allow UTF-8 encoded files everywhere.

An alternative to using Attesoro is editing the translations with UTF-8, then using JDK's native2ascii to convert it to ISO-8859-1.
Once you have to update the translations you can use native2ascii -reverse to convert it back to UTF-8, edit it and finally convert it to ascii again.

There's also plenty of web-based translation tools available which can be used instead of manually editing the .properties files. Just to name a few:

• [http://zanata.org/]
• [https://weblate.org/en/]
• [https://www.transifex.com/]

Using these would be a decision which has to be made by the Mamute maintainers.
However, the result would be an ISO-8859-1 properties file - exactly what you get when using Attesoro (or any other tool).

@khajavi please don't get me wrong - your solution definitely has it's upsides!
But to make it work consistently in the whole application you have to customize many other places in the code.

[khajavi]

Hmm, seems you are right. I need some time to fix this issue.

On Thu, Feb 4, 2016 at 10:56 PM, Martin Blumenstingl <
[email protected]> wrote:

@khajavi https://github.com/khajavi see Environment implementation
http:///caelum/vraptor-environment/blob/master/src/main/java/br/com/caelum/vraptor/environment/DefaultEnvironment.java

• it's not using MamuteLocalization to load the configuration properties.
  I haven't checked the routes implementation but I'm guessing that it's
  similar there.
  You would have to patch everything that loads property files to allow
  UTF-8 encoded files everywhere.

An alternative to using Attesoro is editing the translations with UTF-8,
then using JDK's native2ascii to convert it to ISO-8859-1.
Once you have to update the translations you can use native2ascii -reverse
to convert it back to UTF-8, edit it and finally convert it to ascii again.

There's also plenty of web-based translation tools available which can be
used instead of manually editing the .properties files. Just to name a few:

• [http://zanata.org/]
• [https://weblate.org/en/]
• [https://www.transifex.com/]

Using these would be a decision which has to be made by the Mamute
maintainers.
However, the result would be an ISO-8859-1 properties file - exactly what
you get when using Attesoro (or any other tool).

@khajavi https://github.com/khajavi please don't get me wrong - your
solution definitely has it's upsides!
But to make it work consistently in the whole application you have to
customize many other places in the code.

—
Reply to this email directly or view it on GitHub
#212 (comment).

Milād Khājavi
http://blog.khajavi.ir
Having the source means you can do it yourself.
I tried to change the world, but I couldn’t find the source code.

[xdarklight]

@khajavi that's exactly what I would suggest you avoid: changing lots of code everywhere.

I just thought about it again and we might be able to mix both approaches:
You main point is that you want to be able to edit the .properties files in any text-editor you want.
On the other hand we should use the Java standard for reading the properties files in the application.
One solution to mix both could be the native2ascii-maven-plugin: this will automatically take care of the character conversion and it's transparent for the application. I'm not sure though how this would work in a development environment (where Mamute is started for example via eclipse) - but maybe you can figure this out :).

[felipeweb]

@khajavi let us know when you finish to make the changes

[khajavi]

Ok, sure.
I'm busy in these days. But I will work on this issue as soon as possible.

On Sat, Feb 13, 2016 at 6:40 PM, Felipe Oliveira [email protected]
wrote:

@khajavi https://github.com/khajavi let us know when you finish to make
the changes

—
Reply to this email directly or view it on GitHub
#212 (comment).