Project splitting problems

[andrey-helldar]

In version 10, we split the <locale>.json file into Laravel projects.

This is done because:

1. A 708-key file is difficult to maintain;
2. Due to the large file size, it is difficult to understand which keys are outdated;
3. It is not clear to which project this or that translation key belongs.

What problems did we face:

When splitting the file, the script removed the keys from the <locale>.json files that they "took" for themselves. Due to this, we were faced with the fact that the source file does not contain the required set of keys for the Laravel project itself (Laravel-Lang/lang#1761).

BUT file separation also carries part of the problem: now we can have the same translation key in several files.

At the moment I have restored the key translation as of version 9.1.2.

But how are we going to support the project further?

@Laravel-Lang/laravel-lang, any ideas?

[caouecs]

For me, in source directory ( the English version ) we have a json by package, but in locale directory, we have just one json file by language ( with all distinct tags from English version ).

Then, the cut is done by installer, not by this project

Pros:

• no duplicate tags
• just one file to translate

Cons:

• need installer to have tags for my package(s)

[andrey-helldar]

So. I propose to proceed as follows:

1. I am restoring files from version 9.1.2 (latest stable 😀);
2. From the localization folders I delete the files for splitting into packages;
3. In the laravel/framework project, I search for all used translation keys and save them to the source/en.json file;
4. In the source folder, I change the split files, saving only the keys for projects (Spark, Cashier, etc.) in them;
5. Push changes in Fix: Some translations have been lost in the new version (10.0) lang#1762;
6. We publish the changes as a bug fix, for example as 10.0.3.

As a result, we get something like this structure: https://gist.github.com/andrey-helldar/52e479ca2851dca5daab1eabe1581b2e

[andrey-helldar]

@caouecs, @luisprmat, what do you think?

[andrey-helldar]

Okay. I will wait for an answer from @luisprmat and only then will I start.

The main work will be to update the scripts :)

[ottenso]

Is everyone waiting for the others now? @caouecs have you checked the structure?

[andrey-helldar]

@ottenso, yes, I took into account my mistake and did not start development before everyone expressed their opinion.

There is a discussion in the next block: https://github.com/Laravel-Lang/lang/discussions/1764#discussioncomment-796908

[luisprmat]

• By this you mean just the structure? because i see that there exists missing keys.
• By projects it means that there will also be folders for breeze, fortify, jetstream, cashier, spark and nova?

For me there is no problem, go ahead and I will be attentive to review!

[caouecs]

I will check this week-end

[caouecs]

I'm agree

[andrey-helldar]

Excellent. Then I will start working on the changes today or tomorrow.

@luisprmat, @ottenso, see too.

[sotten]

it was right not to start development before everyone had expressed their opinion. I just had the impression that everyone was waiting for someone else. I just wanted to make sure that no endless loop of waiting is present ;)
Andrey you are doing a great job. I will keep an eye on your pull request. Maybe you can link me once there is a search function to find everything V6-V8 laravel/framework translation.

[andrey-helldar]

@sotten, Dankeschön 🤗

[dinhquochan]

It's mean, the locale directory we restore likes 9.x:

/locales/vi/
   - auth.php
   - *.php
   - vi.json

Like this?

[andrey-helldar]

Yes. The structure and principle of the source directory will change.

See this one.

[andrey-helldar]

@luisprmat, @caouecs, @sotten, guys. Need help.

To release a fix faster while I'm writing code, can you compile a list of translations from Laravel versions 6, 7, and 8?

We will combine all the values from these versions and add them to our project.

We need an json array with values like as:

[
  "Foo",
  "Bar"
]

[andrey-helldar]

We need exactly the English values, from which I can take translations for all localizations from version 9.

[sotten]

@andrey-helldar

If it's just about laravel/laravel and its composer dependencies, I think that should be the words for laravel/laravel V8.5.*
I have not considered laravel/ui, laravel/jetstream, laravel/breeze etc, they should be outsourced as a package.
Let me know if I am on the right path, then I can search the other versions as well.

[
    "All rights reserved.",
    "Forbidden",
    "Go Home",
    "Go to page :page",
    "Hello!",
    "Login",
    "Logout",
    "Not Found",
    "of",
    "Oh no",
    "Page Expired",
    "Pagination Navigation",
    "Regards",
    "Register",
    "results",
    "Server Error",
    "Service Unavailable",
    "Showing",
    "to",
    "Toggle navigation",
    "Too Many Requests",
    "Unauthorized",
    "Whoops!"
]

I have written a small script. Not complete, but maybe it can help someone:

#!/bin/bash
git clone https://github.com/laravel/laravel.git ./laravel/laravel
cd ./laravel/laravel
git tag -l --sort=version:refname v[6-8]* > ../../laravel_versions.txt
cd ../../
filename='laravel_versions.txt'
n=1
while read line; do
git clone --depth 1 --branch $line https://github.com/laravel/laravel.git ./laravel/laravel_$line
cd ./laravel/laravel_$line
composer install --ignore-platform-reqs --no-interaction --no-dev
cd ../../
n=$((n+1))
done < $filename

mkdir ./found
mkdir ./found_withPath
mkdir ./unique
mkdir ./unique_withPath
n=1
while read line; do
cd ./laravel/laravel_$line
grep -oh -r --include=*.{php,stub} -E "__\([^)]+\)|@lang\([^)]+\)|trans\([^)]+\)|trans_choice\([^)]+\)" . --exclude-dir={\*nesbot,symfony,Translation\*} > ../../found/$line.txt
grep -r --include=*.{php,stub} -E "__\([^)]+\)|@lang\([^)]+\)|trans\([^)]+\)|trans_choice\([^)]+\)" . --exclude-dir={\*nesbot,symfony,Translation\*} > ../../found_withPath/$line.txt
cd ../../
sort ./found/$line.txt | uniq > ./unique/$line.txt
sort ./found_withPath/$line.txt | uniq > ./unique_withPath/$line.txt
n=$((n+1))
done < $filename

n=1
while read line; do
cat ./unique/$line.txt >> merge.txt
cat ./unique_withPath/$line.txt >> merge_withPath.txt
n=$((n+1))
done < $filename
sort merge.txt | uniq > unique_merge.txt
sort merge_withPath.txt | uniq > unique_merge_withPath.txt

[andrey-helldar]

Yes, only needed for the laravel/laravel project (in fact, the laravel/framework project contains values).

Another problem is that in some parts of the framework, translations are passed through variables. I alone will process each version for a long time, that's why I asked for help with this.

[sotten]

My json array above I have checked with V6-V8. The translations passed through variables are more difficult to analyze. Maybe find the calls with a static code analysis? Does anyone have experience with tools for this?

[luisprmat]

The following array contains the translatable strings I found within laravel/framework, including those passed by variables and other ways to call the translator.

I cannot assure you that it is exhaustive but it is okay to continue. If we discover another way to find translations, we will add them later through PR.

This list does NOT include keys used exclusively in other packages (it doesn't include packages like breeze, fortify, jetstream, cashier, spark, nova - only laravel/framework ), if they appear in other packages it is because they are repeated there and are also used in Laravel base files.

[
    "All rights reserved.",
    "Forbidden",
    "Go Home",
    "Go to page :page",
    "Hello!",
    "If you did not create an account, no further action is required.",
    "If you did not request a password reset, no further action is required.",
    "If you’re having trouble clicking the \":actionText\" button, copy and paste the URL below\ninto your web browser:",
    "Login",
    "Logout",
    "Not Found",
    "of",
    "Oh no",
    "Page Expired",
    "Pagination Navigation",
    "Please click the button below to verify your email address.",
    "Regards",
    "Register",
    "Reset Password",
    "Reset Password Notification",
    "results",
    "Server Error",
    "Service Unavailable",
    "Showing",
    "The :attribute must contain at least one letter.",
    "The :attribute must contain at least one number.",
    "The :attribute must contain at least one symbol.",
    "The :attribute must contain at least one uppercase and one lowercase letter.",
    "The given :attribute has appeared in a data leak. Please choose a different :attribute.",
    "This action is unauthorized.",
    "This password reset link will expire in :count minutes.",
    "to",
    "Toggle navigation",
    "Too Many Requests",
    "Unauthorized",
    "Verify Email Address",
    "Whoops!",
    "You are receiving this email because we received a password reset request for your account."
]

[keljtanoski]

Hi team, sorry for not being involved i am working on two projects for my employee with a deadline, so i will try to focus in helping more once i deploy these two projects to production.

[andrey-helldar]

Hey! Nothing wrong. We have already released)