[Localization] : Special chars are not being migrated correctly

[jcastro-dotcms]

Parent Issue

• Register the new Locale Variable Upgrade Task as a DataTask #27978

Problem Statement

Some special characters in legacy Language Variables are not being migrated correctly.

Steps to Reproduce

In a local development environment:

• Start up dotCMS with the Full Starter.
• Go to the location of the Spanish language properties file in the /assets/ folder: assets/messages/cms_language_es.properties
• Delete all of its contents, and add a single entry:

something.here=Presentación de Productos

• Run the Upgrade Task that migrates legacy Language Variables via Postman, or use CURL:

curl --location 'http://localhost:8080/api/v1/upgradetask' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "upgradeTaskClass": "com.dotmarketing.startup.runonce.Task240306MigrateLegacyLanguageVariables"
}'

• Go to the Schema > Language Variables portlet, and filter for entries in Spanish.
• You'll see that the special character -- in this case, the ó -- was not migrated correctly:

Acceptance Criteria

Special characters must be migrated without problem.

dotCMS Version

As of dotCMS 24.08.

Proposed Objective

Core Features

Proposed Priority

Priority 2 - Important

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

• https://dotcms.freshdesk.com/a/tickets/28558

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

If necessary, you can go ahead and manually fix the imported legacy Language Variables.

Sub-Tasks & Estimates

No response

[jdotcms]

Hi @jcastro-dotcms
I have ran the steps and wasnt able to reproduce it

[jdotcms]

I have also created this test:

@Test
    public void test_encoding_iso_and_utf8() throws URISyntaxException, IOException {
        // Test code here
        final ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        final URL initFileURL = classLoader.getResource("messages/cms_language_es.properties");
        final List<String> lines = Files.readAllLines(Path.of(initFileURL.toURI()), StandardCharsets.ISO_8859_1);
        Assert.assertNotNull(lines);
        Assert.assertTrue(lines.size() > 0);
        final String line = lines.get(0);
        Assert.assertEquals("something.here=Presentación de Productos", line);
    }

And it is passing ok

[github-actions]

PRs:

• #30694 switching to UTF to see if test runs ok #30951
• #30694 switching from ISO to UTF-8 #30953