diff --git a/src/assets/images/localization/string_changes_firefox_change.png b/src/assets/images/localization/string_changes_firefox_change.png index 16ff55bd..9c44998d 100644 Binary files a/src/assets/images/localization/string_changes_firefox_change.png and b/src/assets/images/localization/string_changes_firefox_change.png differ diff --git a/src/assets/images/localization/string_changes_firefox_l10nchange.png b/src/assets/images/localization/string_changes_firefox_l10nchange.png index 5add7fb5..60878f2f 100644 Binary files a/src/assets/images/localization/string_changes_firefox_l10nchange.png and b/src/assets/images/localization/string_changes_firefox_l10nchange.png differ diff --git a/src/assets/images/localization/string_changes_firefox_start.png b/src/assets/images/localization/string_changes_firefox_start.png index d4d608de..0fd02fcb 100644 Binary files a/src/assets/images/localization/string_changes_firefox_start.png and b/src/assets/images/localization/string_changes_firefox_start.png differ diff --git a/src/localization/making_string_changes.md b/src/localization/making_string_changes.md index 2f98ef30..68aebd1c 100644 --- a/src/localization/making_string_changes.md +++ b/src/localization/making_string_changes.md @@ -88,8 +88,8 @@ In this scenario, the only way to ensure that obsolete translations are ignored ### Firefox Things become a lot more complicated for Firefox: -* Each localization is stored in a separate Mercurial repository. -* The `en-US` locale is stored in the code repository (e.g. `mozilla-central`). That locale is used by the build system to build the US English version of Firefox, but it’s not used as source by the localization toolchain. In fact, in order to ship all versions of Firefox (Nightly, Beta, Developer Edition, Release, ESR) from a single l10n repository, we rely on a special [cross-channel repository](https://firefox-source-docs.mozilla.org/l10n/crosschannel/index.html) called `gecko-strings`, which includes all strings for all shipping versions of Firefox. +* The `en-US` locale is stored in the code repository (e.g. `mozilla-central`). That locale is used by the build system to build the US English version of Firefox, but it’s not used as source by the localization toolchain. In fact, in order to ship all versions of Firefox (Nightly, Beta, Developer Edition, Release, ESR) from a single l10n repository, we rely on a special unified [repository](https://https://github.com/mozilla-l10n/firefox-l10n-source) in GitHub called `firefox-l10n-source`, which includes all strings for all shipping versions of Firefox. +* Localizations are stored in a separate repository (`firefox-l10n`) from source strings. This is the scenario before the string change. Once again, although it already looks very busy, it’s a simplified high level view of the actual system. @@ -99,7 +99,7 @@ And this is what happens if a string is changed without new ID: ![Firefox: change without new ID](../assets/images/localization/string_changes_firefox_change.png) -* The string is changed in `mozilla-central`, and the en-US Nightly build will start using it. This change is reflected in `gecko-strings`, but doesn’t impact other builds (e.g. beta), because that’s not used by the build system. +* The string is changed in `mozilla-central`, and the en-US Nightly build will start using it. This change is reflected in `firefox-l10n-source`, but doesn’t impact other builds (e.g. beta), because that’s not used by the build system. * As for the generic product, Pontoon reads the updated change, but doesn’t notify localizers or invalidate existing translations. In this scenario, simply removing existing translations from VCS isn’t possible, because we still need them to ship in older builds. diff --git a/src/products/firefox_desktop/adding_nightly.md b/src/products/firefox_desktop/adding_nightly.md index 9f9c3577..5e69f1b6 100644 --- a/src/products/firefox_desktop/adding_nightly.md +++ b/src/products/firefox_desktop/adding_nightly.md @@ -16,12 +16,10 @@ Then file a new tracking bug using this [bug template](https://mzl.la/3vwVtub). Note that the tracking bug is filed under `Mozilla Localizations :: Other`, as creating the Bugzilla component is a step that will be performed when the locale is ready to move to Beta/Release. If the component for this locale is already available, the bug should to be moved directly there. -To make it easier to track dependencies, also add the bug used to create the l10n-central repository (see [Adding a new locale to Pontoon](adding_pontoon.md)) as a dependency (`Depends on:` field) of the tracking bug. - ## Verify content in l10n repository Before enabling the build, it’s a good idea to perform some basic checks: -* Check `toolkit/global/intl.properties` ([en-US version](https://hg.mozilla.org/mozilla-central/file/default/toolkit/locales/en-US/chrome/global/intl.properties)) for evident mistakes. +* Check `it/toolkit/chrome/global/intl.properties` ([it version](https://github.com/mozilla-l10n/firefox-l10n/blob/main/it/toolkit/chrome/global/intl.properties)) for evident mistakes. ## Add new locale to build configuration diff --git a/src/products/firefox_desktop/adding_pontoon.md b/src/products/firefox_desktop/adding_pontoon.md index 977ff6a4..2d095f5a 100644 --- a/src/products/firefox_desktop/adding_pontoon.md +++ b/src/products/firefox_desktop/adding_pontoon.md @@ -1,7 +1,5 @@ # Adding a new locale to Pontoon * If the locale is not [available](https://pontoon.mozilla.org/teams/) in Pontoon yet, [consult this document](../../tools/pontoon/adding_new_locale.md) for instructions on how to add it. -* File a bug to create a new Mercurial repository in [l10n-central](https://hg.mozilla.org/l10n-central/) using [this bug template](https://mzl.la/3VxIZgs) (replace `%LOCALE_CODE%` with the actual locale code). -* Push at least one change to the repository, otherwise Pontoon will not pick it up during sync. The safest change is to initialize the `toolkit/chrome/global/intl.properties` file with the correct values ([example](https://hg.mozilla.org/l10n-central/ppl/rev/b3fd0faf59b0b45b2cf30c01d85157beee2a0bd0)). - -Once the repository is available and contains at least one commit, the locale can be added to the Firefox project in Pontoon. +* Enable the locale for Firefox from the [project admin interface](https://pontoon.mozilla.org/admin/projects/firefox/) by moving your locale from the Available column to the Localizable column. +* Initialize the `toolkit/chrome/global/intl.properties` file with the correct values: ([example](https://github.com/mozilla-l10n/firefox-l10n/commit/a887bff12994779ea8e605f4abc1b240a287ccb9)). diff --git a/src/products/firefox_desktop/build_system.md b/src/products/firefox_desktop/build_system.md index 8da2b1a1..b072f24b 100644 --- a/src/products/firefox_desktop/build_system.md +++ b/src/products/firefox_desktop/build_system.md @@ -4,12 +4,12 @@ ## Overview -Thanks to [cross-channel](https://firefox-source-docs.mozilla.org/l10n/crosschannel/index.html), we ship all versions of Firefox from a [single localization repository](https://hg.mozilla.org/l10n-central/), but each channel uses a different snapshot in time of that repository. +Thanks to using a [single source for localization](https://firefox-source-docs.mozilla.org/l10n/singlel10nsource.html), we ship all versions of Firefox from a [single localization repository](https://github.com/mozilla-l10n/firefox-l10n). - A job, called `l10n-bumper`, runs on Taskcluster every hour, and stores information on the l10n changesets to use for each build in a file called `l10n-changesets.json`: -* For Nightly (`mozilla-central`), [l10n-changesets.json](https://hg.mozilla.org/mozilla-central/file/default/browser/locales/l10n-changesets.json) always uses the revision `default`, and it’s only updated when changing the locales available in the build. -* Beta builds (`mozilla-beta`) still use the tip of the l10n repository, but the actual hash is stored in [l10n-changesets.json](https://hg.mozilla.org/releases/mozilla-beta/file/default/browser/locales/l10n-changesets.json) instead of `default`. -* Before Release Candidate week, the `l10n-bumper` is [manually stopped](https://github.com/mozilla/build-relengdocs/blob/master/procedures/release-duty/merge-duty/merge_duty.rst#turn-off-beta-l10n-bumper-on-rc-day) on `mozilla-beta`. Since the configuration change lands only on `mozilla-beta`, the `l10n-bumper` will automatically restart at the beginning of the next cycle, when the standard configuration is uplifted from `mozilla-central`. +A job, called `l10n-bumper`, runs on Taskcluster every hour, and stores information on which changeset (defined by the revision SHA of `firefox-l10n`) to use for each build in a file called `l10n-changesets.json`: +* Nightly (`mozilla-central`) changeset: [l10n-changesets.json](https://hg.mozilla.org/mozilla-central/file/default/browser/locales/l10n-changesets.json). This always uses the latest revision of `main`. +* Beta (`mozilla-beta`) changeset: [l10n-changesets.json](https://hg.mozilla.org/releases/mozilla-beta/file/default/browser/locales/l10n-changesets.json). This also uses the the latest revision of `main`, but it’s only updated twice a week. +* Before Release Candidate week, the `mozilla-beta` tree is [closed](https://moz-releng-docs.readthedocs.io/en/latest/how-to/releaseduty/merge-duty/merge_duty.html#merge-beta-to-release). The `l10n-bumper` will restart at the beginning of the next cycle, when the `mozilla-beta` tree is [reopened](https://moz-releng-docs.readthedocs.io/en/latest/how-to/releaseduty/merge-duty/merge_duty.html#re-opening-the-tree-s). * When the Beta code is merged to Release, [l10n-changesets.json](https://hg.mozilla.org/releases/mozilla-release/file/default/browser/locales/l10n-changesets.json) moves together with the rest of the code to `mozilla-release`. That means that Release builds will use the same changesets as the last beta with the same version number, and any further change [requires code uplifts](#updating-release). ## Timeline and deadlines @@ -30,4 +30,4 @@ When the code moves from `mozilla-beta` to `mozilla-release`, `l10n-changesets.j In case of severe issues affecting one or more locales, it’s still possible to manually update the shipping changesets. A patch needs to be provided for `l10n-changesets.json` in `mozilla-release` branch and approved for uplift by Release Drivers (see for example [this bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1513259) and [associated patch](https://hg.mozilla.org/releases/mozilla-release/rev/308fd26a204e)). Note that a dot release is needed in order to ship the updated version to users. [This script](https://github.com/flodolo/scripts/blob/master/mozilla_l10n/validate_fx_changesets_uplift/fx_release_changesets.py) can be used to validate the updated JSON. -The same process applies to ESR versions, as long as the associated esr repository is included in the [current version of cross-channel](https://hg.mozilla.org/mozilla-unified/file/tip/python/l10n/mozxchannel/__init__.py). +The same process applies to ESR versions, as long as the associated esr repository is included among the [supported versions in firefox-l10n-source](https://github.com/mozilla-l10n/firefox-l10n-source/blob/main/.github/update-config.json#L2). diff --git a/src/products/firefox_desktop/firefox_l10n_faqs.md b/src/products/firefox_desktop/firefox_l10n_faqs.md index c7558d9b..07135f59 100644 --- a/src/products/firefox_desktop/firefox_l10n_faqs.md +++ b/src/products/firefox_desktop/firefox_l10n_faqs.md @@ -42,9 +42,9 @@ These approaches allow to ship one package and support multiple languages, but t ### I landed strings in mozilla-central, when are they going to be localized? -New string changes are exported twice a day from mozilla-central into a repository called `gecko-strings-quarantine`, a unified repository that includes strings for all shipping versions of Firefox (nightly, beta, release). This repository is used as a buffer to avoid exposing potential issues to all locales working on Firefox. +New string changes are extracted twice a day from `gecko-dev` (the git mirror of mozilla-central) into a repository called `firefox-l10n-source`, a unified repository that includes strings for all shipping versions of Firefox (nightly, beta, release). This repository is used as a buffer to avoid exposing potential issues to all locales working on Firefox. -Typically once or twice a week, the content of the quarantine repository is pushed to the official repository, called `gecko-strings`, used by [Pontoon](https://pontoon.mozilla.org/projects/firefox/) as source reference. At this point, strings can be localized by community. +Typically once or twice a week, the quarantined content (in the `update` branch) is pushed to the `main` branch which acts as the official repository, used by [Pontoon](https://pontoon.mozilla.org/projects/firefox/) as source reference. At this point, strings can be localized by the community. ### Can I uplift a patch to Beta or Release? @@ -60,7 +60,7 @@ If version X is in Nightly, i.e. currently developed in mozilla-central, there i The sooner you land content in mozilla-central, the higher will be the chances that the content will be localized in several languages before reaching release. -A webapp is [available here](https://fx-trains.herokuapp.com/release/) with all the deadlines, including string freeze start, for current and future versions of Firefox (use the `«` and `»` near the title to change version). +A webapp is [available here](https://whattrainisitnow.com/release/?version=release) with all the deadlines, including string freeze start, for current and future versions of Firefox (use the `«` and `»` near the title to change version). ## Development @@ -72,11 +72,11 @@ A document including plenty of best practices is [available here](../../localiza Strings need to be added to localization files in known locations within the mozilla-central tree. For all the technical details about these paths and the supported formats, see [this document](https://firefox-source-docs.mozilla.org/build/buildsystem/locales.html#exposing-strings). -Once strings land in mozilla-central, they will be exposed for localization in Pontoon within a few days. +Once strings land in mozilla-central, they will be exposed for localization in Pontoon typically within a week or sooner. ### Where can I find the localized strings? -All shipping versions of Firefox are built from a single Mercurial repository for each locale (`l10n-central`). Repositories are available [here](https://hg.mozilla.org/l10n-central/). +All shipping versions of Firefox are built from a single GitHub repository called [firefox-l10n](https://github.com/mozilla-l10n/firefox-l10n), with a subdirectory for each locale. ### Can I land content without exposing it for localization? @@ -94,14 +94,14 @@ No, the existing infrastructure only allows to expose strings to all locales. If It’s always possible to restore an old string that was removed from code, as long as the text remains the same, and the string is used exactly in the same context. -It might also be possible to uplift the patch if the string is still available in the [gecko-string repository](https://hg.mozilla.org/l10n/gecko-strings). `gecko-strings` is a repository with a [superset](https://firefox-source-docs.mozilla.org/l10n/crosschannel/index.html) of strings from all [supported versions](https://hg.mozilla.org/mozilla-unified/file/tip/python/l10n/mozxchannel/__init__.py#l31) of Firefox: nightly, beta, release, ESR. +It might also be possible to uplift the patch if the string is still available in the [firefox-l10n-source](https://github.com/mozilla-l10n/firefox-l10n-source) repository. `firefox-l10n` acts as a single localization source for multiple builds, with a [superset](https://firefox-source-docs.mozilla.org/l10n/singlel10nsource.html) of strings from all [supported versions](https://github.com/mozilla-l10n/firefox-l10n-source/blob/main/.github/update-config.json#L2) of Firefox: nightly, beta, release, ESR. Consider this example: * The original string landed in Firefox 85, and was removed later in Firefox 92. * The list of support versions in cross-channel includes: ESR91, 92, 93, 94. * The string is restored in Firefox 94. -The string will still be available in `gecko-strings` until ESR91 becomes unsupported, since the string was removed after that release (in Firefox 92). This patch could be uplifted without creating any issue to localization (it would be a *no-op*). +The string will still be available in `firefox-l10n-source` until ESR91 becomes unsupported, since the string was removed after that release (in Firefox 92). This patch could be uplifted without creating any issue to localization (it would be a *no-op*). If the string is not available anymore, this is effectively a new string, and must be [treated as such](#can-i-uplift-a-patch-to-beta-or-release). @@ -135,7 +135,7 @@ Given that localization is managed by community volunteers, there is no SLA or g ### How do I communicate with localizers about my feature or patch? -If there is some specific information that you want to convey to localizers, like testing instructions or particular issues to look out for, get in touch with the [L10N PM for Firefox](#who-can-i-contact-if-i-have-more-questions). They will help you identify the best channel and way to relay this information. +If there is some specific information that you want to convey to localizers, like testing instructions or particular issues to look out for, get in touch with the [L10N PM for Firefox Desktop](#who-can-i-contact-if-i-have-more-questions). They will help you identify the best channel and way to relay this information. ### I see pending suggestions in Pontoon, how can I get them approved? diff --git a/src/products/firefox_desktop/fluent_migrations.md b/src/products/firefox_desktop/fluent_migrations.md index 612efcfe..a7a86527 100644 --- a/src/products/firefox_desktop/fluent_migrations.md +++ b/src/products/firefox_desktop/fluent_migrations.md @@ -2,38 +2,36 @@ [Fluent migrations](https://firefox-source-docs.mozilla.org/l10n/migrations/index.html) are used to migrate existing translations from legacy formats (DTD, properties) to Fluent, or existing Fluent messages to new messages or different FTL files. -At the core of each migration there is a Python file, called **migration recipe**, that instructs the system on how to port existing translations to Fluent. Typically, a patch migrating content to Fluent will: +At the core of each migration there is a Python file, called a **migration recipe**, that instructs the system on how to port existing translations to Fluent. Typically, a patch migrating content to Fluent will: * Add new Fluent strings or files. * Remove obsolete strings. * Include a migration recipe. -Once such patch lands in `mozilla-central`, it will be converted as part of the cross-channel generation, and pushed to a quarantine repository (`gecko-strings-quarantine`). At this point, this new content is not visible yet in [Pontoon](https://pontoon.mozilla.org/). +Once one such patch lands in [gecko-dev](https://github.com/mozilla/gecko-dev), string changes will be pushed to the `update` branch of [firefox-l10n-source](https://github.com/mozilla-l10n/firefox-l10n-source). At this point, this new content is not visible yet in [Pontoon](https://pontoon.mozilla.org/). -The goal of migrations is to port existing translations to the new format without forcing community to retranslate them. For that to happen, migrations need to be performed before the new strings are pushed to the official `gecko-strings` repository, and become available in Pontoon. +The goal of a migration is to port existing translations to a new format or file without forcing the community localizers to retranslate them. For this to happen, migrations need to be performed before the new strings are pushed to the `main` branch of the `firefox-l10n-source` repository and become available in Pontoon. -This means that patches including migrations should only be pushed from the quarantine repository to `gecko-strings` following the steps described in this document. +This means that patches that include migrations should only be pushed from the `update` branch to `main` following the steps described in this document. ## Running Fluent migrations -This section of the document describes in detail which operations need to be performed in order to run migrations on over 150 Mercurial repositories for Firefox. +This section of the document describes in detail which operations need to be performed in order to run migrations. ### Prerequisites Several access rights are required: -* Write access to [l10n-central](https://hg.mozilla.org/l10n-central), i.e. SSH access with `active_scm_l10n` rights. It’s possible to check your *Access Groups* in [your profile on people.mozilla.org](https://people.mozilla.org/e#nav-access-groups). +* Admin access to `firefox-l10n-source` and `firefox-l10n` (as a member of the `l10n-firefox` team). * Admin access to [Pontoon](https://pontoon.mozilla.org/). * An account on Heroku with access to [Pontoon apps](https://dashboard.heroku.com/apps/mozilla-pontoon/resources). In order to run migrations: * Install the [fluent.migrate](https://pypi.org/project/fluent.migrate/) Python package. -* Clone all [l10n-central](https://hg.mozilla.org/l10n-central) repositories. -* Clone [gecko-strings-quarantine](https://hg.mozilla.org/l10n/gecko-strings-quarantine/) and [gecko-strings](https://hg.mozilla.org/l10n/gecko-strings). +* Clone the [firefox-l10n](https://github.com/mozilla-l10n/firefox-l10n) repository. +* Clone the [firefox-l10n-source](https://github.com/mozilla-l10n/firefox-l10n-source) repository. -A set of scripts to automate cloning and updating of the l10n-central repositories is [available here](https://github.com/flodolo/scripts/tree/master/mozilla_l10n/clone_hgmo). +### Ensure there are no blocking issues in the update branch of firefox-l10n-source -### Ensure there are no blocking issues in gecko-strings-quarantine - -As part of migrations, content needs to be pushed from the quarantine repository to `gecko-strings`. Since content can’t be pushed selectively, any pending issue needs to be solved before starting the process (eventually backing out problematic changes). For more information about the review process, see [this document](review.md). +As part of migrations, content needs to be pushed from the `update` branch to the `main` branch of `firefox-l10n-source`. Any pending issue needs to be solved before starting the process (eventually manually backing out problematic changes) to prevent problematic strings being exposed to localizers. For more information about the review process, see [this document](review.md). ### Stop sync in Pontoon @@ -41,7 +39,7 @@ Before starting the migration process, sync needs to be suspended for Firefox fr This is needed for several reasons: * It removes the chance of conflicts in the l10n repository between the running migration and Pontoon committing changes for other projects. -* The migrated strings need to be added in the same cycle as the new strings for the source language (`en-US`). If they’re added before, Pontoon will ignore them, creating a misalignment between internal database and l10n repositories. If they’re added after, these strings will be displayed as missing in Pontoon, and some locales might try to translate them while it’s not needed. +* The migrated strings need to be added in the same cycle as the new strings for the source language (`en-US`). If they’re added before, Pontoon will ignore them, creating a misalignment between the internal database and l10n repositories. If they’re added after, these strings will be displayed as missing in Pontoon, and some locales might try to translate them while it’s not needed. The last step here is to make sure that the current sync process has completed: * Check the [Sync Logs](https://pontoon.mozilla.org/sync/log/) page, click on the last one, and make sure that there are no projects with `---` in the `DURATION` field. @@ -50,23 +48,20 @@ The last step here is to make sure that the current sync process has completed: A series of tools and helpers to run migrations is available in [this repository](https://github.com/flodolo/fluent-migrations): * Clone the repository locally and follow the instructions available in the [README](https://github.com/flodolo/fluent-migrations/blob/master/README.md) to set the configuration file. -* Add the new migration recipes in the `recipes` folder. The script will look for any Python file starting with `bug_` in that folder, so it’s possible to run multiple migrations in one pass. More information about the folder’s structure are available in the repository’s README. +* Add the new migration recipes in the `recipes` folder. The script will look for any Python file starting with `bug_` in that folder, so it’s possible to run multiple migrations in one pass. More information about the folder’s structure is available in the repository’s README. At this point, while each migration has been tested as part of the review before landing, it’s always good to run the migration against one locale, e.g. `it`: -* Update the locale’s repository, check the results of `compare-locales`. Assuming l10n repositories are cloned in `~/migrations/locales`, and the quarantine repository in `~/migrations/gecko-strings-quarantine`, the command to run is `compare-locales ~/migrations/gecko-strings-quarantine/_configs/browser.toml ~/migrations/locales it`. Save the output in order to compare it with the results after the migration. +* Update the locale’s repository, check the results of `compare-locales`. Assuming the l10n repository is cloned in `~/migrations/firefox-l10n`, and the quarantine repository is in `~/migrations/firefox-l10n-source` with the `update` branch checked out, the command to run is `compare-locales ~/migrations/firefox-l10n-source/_configs/browser.toml ~/migrations/firefox-l10n it`. Save the output in order to compare it with the results after the migration. * Run the migration, without pushing, only for `it`: `./scripts/migrate it wet-run`. Then run compare-locales again, and check if the results are as expected: migrated strings should not appear as missing anymore, and there should be no errors. -* If everything looks as expected, re-run the script adding the push option `./scripts/migrate it wet-run push`. -### Push updates to gecko-strings +### Push updates to the main branch of firefox-l10n-source -Before [pushing updates](review.md#push-reviewed-strings-to-gecko-strings) to the official [gecko-strings repository](https://hg.mozilla.org/l10n/gecko-strings), make sure that: -* The new strings and migrations are available in the [quarantine repository](https://hg.mozilla.org/l10n/gecko-strings-quarantine). -* There are [no pending issues](#ensure-there-are-no-blocking-issues-in-gecko-strings-quarantine) that prevent the current content to be exposed in Pontoon. +Before running the migration of localization strings, [merge](review.md#merge-reviewed-strings-to-main-branch-of-firefox-l10n-source) the source (en-US) strings of the `update` branch in `firefox-l10n-source` to the `main` branch. Make sure that: +* The new strings and migrations are available in the `update` branch. +* All [pending issues](#ensure-there-are-no-blocking-issues-in-the-update-branch-of-firefox-l10n-source) are dealt with. * The migration has been tested locally and works as expected. -Once the new strings are pushed to `gecko-strings`, there’s no easy way to roll back, and the procedure needs to be completed before re-enabling sync in Pontoon. The alternative is forcing all locales to retranslate this content, and losing the entire benefit of migrations. - -Pontoon relies on a [different repository](https://hg.mozilla.org/users/m_owca.info/firefox-central/), that is generated every 20 minutes based on the content of `gecko-strings`. That’s why it’s useful to run this update before starting the actual migrations. +Once the new source strings are merged, the remaining procedures need to be completed before re-enabling sync in Pontoon, otherwise additional manual work on the repository to revert the commit will be required. The alternative is forcing all locales to retranslate this content, and losing the entire benefit of migrations. ### Run migrations @@ -75,15 +70,22 @@ To run the migration on all repositories, use `./scripts/migrate wet-run push` i * Run the migration and commit changes. * Push changes to the remote. -The duration of this step depends on the number of migrations and the speed of the Internet connection, but it should take about 20-30 minutes. +The duration of this step depends on the number of migrations and the speed of the Internet connection, but this should only take a few minutes to run. While migrations run, it’s important to look out for errors in the console, even if the standard output is quite noisy: * Python errors with a stack trace are unlikely at this stage of the development, since it means that the code encountered a scenario that it’s unable to manage. If something is broken in the `fluent.migrate` package, or there are issues with Python dependencies, the migration should stop at the very beginning. -* For Mercurial, one potential error scenario is a push that would create a new head in the remote repository (`abort: push creates new remote head`, followed by a changeset ID). The script automatically pulls from the remote repository before running migrations, and Pontoon’s sync is disabled, so that shouldn’t happen, unless a previous migration failed to push to remote and left unsynced local commits. In this case, the easiest solution is to clone the repository for this locale from scratch, and run the migration only for that specific locale. + +When the script completes, by default a new branch named `update-migration` will be created in the `firefox-l10n` repository unless run with the `current-branch` argument. Note: the name of the branch created can be adjusted in your local [config.dist](https://github.com/flodolo/fluent-migrations/blob/main/config/config.dist#L12) file. + +### Create and merge pull request against main in firefox-l10n + +Use the URL generated in your terminal output in the previous step to create a pull request against the `main` branch in GitHub for the migrated strings in [firefox-l10n](https://github.com/mozilla-l10n/firefox-l10n). (Or create a pull request manually.) + +Upon creation, review the diff against the `main` branch to ensure everything ran correctly, then if there’s nothing wrong approve the PR and merge it into `main`. ### Re-enable Sync in Pontoon -Once the actual migration is complete, the next step is to re-enable sync in Pontoon. Before starting, make sure that the [Pontoon repository](https://hg.mozilla.org/users/m_owca.info/firefox-central/) has a commit with the new strings. +Once the actual migration is complete, the next step is to re-enable sync in Pontoon. * Access the [admin panel for Firefox](https://pontoon.mozilla.org/admin/projects/firefox/), click the `SYNC` button at the bottom, then deselect `SYNC DISABLED` and click `SAVE PROJECT`. * Clicking the `SYNC` button will spawn a new sync process just for Firefox. Check the [Sync Logs](https://pontoon.mozilla.org/sync/log/) page to see when it’s finished. Depending on the amount of changes, this can require from 15 to over 30 minutes. It’s also possible to monitor the status in Papertrail, accessible from the [Resources](https://dashboard.heroku.com/apps/mozilla-pontoon/resources) page in Heroku, and filter the log using the string `app/worker`. diff --git a/src/products/firefox_desktop/review.md b/src/products/firefox_desktop/review.md index 6954d7f2..f7ca0b86 100644 --- a/src/products/firefox_desktop/review.md +++ b/src/products/firefox_desktop/review.md @@ -4,140 +4,98 @@ ## Review landed strings -Starting from Firefox 57, all versions of Firefox desktop ship by localizing a single repository containing the reference English strings, called [gecko-strings](https://hg.mozilla.org/l10n/gecko-strings). It is generated from strings landing in the code repository for each branch (e.g. mozilla-central and comm-central for Nightly, mozilla-beta and comm-beta for Beta, etc.), and it’s exposed to localization tools like Pontoon. +All versions of Firefox desktop ship by localizing a single repository containing the reference English strings, called [firefox-l10n-source](https://github.com/mozilla-l10n/firefox-l10n-source). It is generated from strings landing in the Firefox source code repository [gecko-dev](https://github.com/mozilla/gecko-dev) (a mirror of the Mercurial repository `mozilla-central` which will eventually be phased out). Automation in `firefox-l10n-source` extracts strings from each of the version specific branches in `gecko-dev` (e.g. master for Nightly, beta for Beta, release for Release, etc.). -There is a second repository, [gecko-strings-quarantine](https://hg.mozilla.org/l10n/gecko-strings-quarantine/), used as a buffer to avoid exposing poor strings to the larger audience of localizers. +There are two branches in `firefox-l10n-source`: `main` which contains all strings currently exposed for localization, and `update` which acts as a string quarantine, used as a buffer to avoid exposing poor strings to the larger audience of localizers. Strings are regularly merged from `update` into `main` after a review by the Localization Team; newly added and removed strings in `main` are synced with our localization tool Pontoon. -The localizations for all channels can be found in [l10n-central](https://hg.mozilla.org/l10n-central/), with a single repository for each locale. +Localized strings for all channels can be found in [firefox-l10n](https://github.com/mozilla-l10n/firefox-l10n), with a single subfolder for each locale. The review process consists of three parts: -* Review strings landing in `mozilla-central`. Currently `comm-central` doesn’t undergo a similar review process. -* Review strings landing in `gecko-strings-quarantine`. The quarantine repository is updated twice a day via automation in [Task Cluster](https://firefox-source-docs.mozilla.org/taskcluster/kinds.html#l10n-cross-channel). -* Push reviewed strings to `gecko-strings`, and start the localization process. +* Review strings landing in `mozilla-central`. +* Review strings extracted into the `update` branch of `firefox-l10n-source`. The [automation](https://github.com/mozilla-l10n/firefox-l10n-source/blob/main/.github/workflows/update.yml) runs twice a day in the repository with any changes added to a pull request against the `main` repository. +* Review and merge strings in the pull request to `main`, and start the localization process. + +Things to look out for during the review process: +* Unclear strings and missing localization comments: the best way to identify them is to translate the strings, only having the string and comment as context (not the entire file, or the bug). For example: is the word used as both a noun and a verb in English? Is the ID clear enough to give context (e.g. `buttonLabel`)? +* String changes without new IDs. +* Duplicated strings. +* [Localization issues](https://mozilla-l10n.github.io/documentation/localization/dev_best_practices.html), like misused plural forms, unclear comments, etc. ### Review strings landing in mozilla-central You can get the list of changesets touching localized strings in the last 2 days from [mozilla-central](https://hg.mozilla.org/mozilla-central/log?rev=keyword("locales/en-US")+and+pushdate("-2")). Adjust the `pushdate` part if you want to see more or less days. -There are some unrelated changesets, like en-US dictionary updates, but the majority of landings are relevant and need to be checked for localization issues. +Note: this review includes changes landing in `mozilla-central`, but those changes may not have landed yet in `gecko-dev`, and consequently in `firefox-l10n-source`. Once changes have completed the landing process, those strings will also be available to review in the next step - [Review strings in update branch of firefox-l10n-source](#review-strings-in-update-branch-of-firefox-l10n-source). -You need to open each changeset, and identify changed files that are relevant for localization (.properties, .dtd, .ini). +There are some unrelated changesets, like en-US dictionary updates, but the majority of landings are relevant and need to be checked for localization issues. One exception to this are strings that are purposely not meant to be exposed to localizers, typically called "preview" strings. While these strings are often saved to files within the `browser/locales-preview` directory, this includes any localization file not saved in a `locales/en-US` path. -Things to look out for: -* Unclear strings and missing localization comments: the best way to identify them is to translate the strings, only having the string and comment as context (not the entire file, or the bug). For example: is the word used both a noun and a verb in English? Is the ID clear enough to give context (e.g. `buttonLabel`)? -* String changes without new IDs. -* Duplicated strings. -* [Localization issues](https://mozilla-l10n.github.io/documentation/localization/dev_best_practices.html), like misused plural forms, unclear comments, etc. +You need to open each changeset, and identify changed files that are relevant for localization (.properties, .dtd, .ini). In case of issues, you have two options: * Ask sheriffs (via bug or [Matrix in #sheriffs](https://chat.mozilla.org/#/room/#sheriffs:mozilla.org)) to back out the patch. * Ask clarifications in the bug, and decide if it’s worth to stop exposing new strings until the issue is fixed. -### Review strings landing in gecko-strings-quarantine +### Review strings in update branch of firefox-l10n-source + +The next step is to check changes landing in the `update` branch of [firefox-l10n-source](https://github.com/mozilla-l10n/firefox-l10n-source/tree/update). Automation creates a [pull request]((https://github.com/mozilla-l10n/firefox-l10n-source/pulls) ) which includes any changed or added strings. Review the diff generated by the pull request. -The next step is to spot check changes landed in [gecko-strings-quarantine](https://hg.mozilla.org/l10n/gecko-strings-quarantine/shortlog). Here are some things to look out for: +Here are some things to look out for: * Check if a changeset is removing strings. This should happen only when a string landed in Nightly and was removed during the same cycle, or at the beginning of a release cycle when a group of strings becomes unused in all shipping versions. -**IMPORTANT:** Patches including Fluent migrations need to be kept in quarantine, and can only be pushed to `gecko-strings` following the process described in [this document](../firefox_desktop/fluent_migrations.md). +**IMPORTANT:** Patches including Fluent migrations need to be kept in the `update` branch, and can only be merged into `main` following the process described in [this document](../firefox_desktop/fluent_migrations.md). -### Run compare-locales against gecko-strings +### Run compare-locales against a localization repository -A good next step to check for issues is to run [compare-locales](https://pypi.org/project/compare-locales/) against the `gecko-strings` repository. +A good next step to check for issues is to run compare-locales against a localization repository frequently updated (Italian and French are good examples). -First of all make sure that your environment is [correctly set up](../../tools/mercurial/setting_mercurial_environment.md), and update your local `mozilla-unified` clone. +First of all make sure that you have both [firefox-l10n-source](https://github.com/mozilla-l10n/firefox-l10n-source/) and [firefox-l10n](https://github.com/mozilla-l10n/firefox-l10n) repositories cloned locally, and your local clones are updated. -`compare-locales` needs to be installed on your system. You can either install a specific release, or clone the [hg repository](https://hg.mozilla.org/l10n/compare-locales/) and install it via `pip install -e .`. You can check that `compare-locales` is running correctly by checking its version: +`compare-locales` needs to be installed on your system. You can either install a specific release, or install the latest release version via `pip install compare-locales`. You can check that `compare-locales` is running correctly by checking its version: ``` $ compare-locales --version -compare-locales 8.1 +compare-locales 9.0.4 ``` Let’s assume that: -* [gecko-strings-quarantine](https://hg.mozilla.org/l10n/gecko-strings-quarantine) is cloned in `~/l10n/gecko-strings-quarantine`. -* [mozilla-unified](https://hg.mozilla.org/mozilla-unified) is cloned in `~/src/mozilla-unified`, and you checked out the version corresponding to the converted changeset. - -Then run - -``` -$ compare-locales --unified --full ~/src/mozilla-unified/browser/locales/l10n.toml ~/l10n gecko-strings-quarantine -``` - -When running these, you should see no errors or warnings. When running them against the central revisions, you should see no missing or changed strings, while having obsolete strings is expected. When running against beta or release revisions, expect to have changed strings, but again, no missing strings. - -Note: when running compare-locales against a non-existing locale code, use the `--full` command line argument to get all strings in submodules. In particular for gecko-strings, you need that, otherwise you only get the strings in the `browser` directory. - -### Run compare-locales against a localization repository +* `firefox-l10n-source` is cloned in `~/firefox-l10n-source`. +* `firefox-l10n` is cloned in `~/firefox-l10n`. -A good next step to check for issues is to run compare-locales against a localization repository frequently updated (Italian and French are good examples). -* [l10n-central/it](https://hg.mozilla.org/l10n-central/it) is cloned in `~/l10n/it`. - -To run compare-locales against `mozilla-unified` and Italian you can run: - -``` -$ compare-locales --unified ~/src/mozilla-unified/browser/locales/l10n.toml ~/l10n it -``` - -To run compare-locales against `gecko-strings-quarantine` and Italian you can run: +To run compare-locales against the string quarantine of `firefox-l10n-source` and Italian, first checkout the `update` branch of `firefox-l10n-source`, then run this command: ``` -$ compare-locales --unified ~/l10n/gecko-strings-quarantine/_configs/browser.toml ~/l10n it +$ compare-locales ~/firefox-l10n-source/_configs/browser.toml ~/firefox-l10n it ``` -Both are really long commands, so it’s convenient to create Bash aliases in `~/.bash_profile` for them, e.g. +Because the command is quite long, it’s convenient to create a Bash aliases in `~/.bash_profile` for it, e.g. ``` -cmp_moz="compare-locales --unified ~/src/mozilla-unified/browser/locales/l10n.toml ~/l10n it" -cmp_mozx="compare-locales --unified ~/l10n/gecko-strings-quarantine/_configs/browser.toml ~/l10n it" +cmp_moz="compare-locales ~/firefox-l10n-source/_configs/browser.toml ~/firefox-l10n it" ``` -Let’s start with the output of `compare-locales` against `gecko-strings-quarantine`: most of the time, it should only report missing strings. There will be obsolete strings only if a string was removed, which is a rare event in cross-channel. +Reviewing the output of `compare-locales` against the `update` branch of `firefox-l10n-source`: most of the time, it should only report missing strings. There will be obsolete strings only if a string was removed, which is a rare event. For example, this is the output for a fully localized locale. ``` -$ compare-locales --unified ~/l10n/gecko-strings-quarantine/_configs/browser.toml ~/l10n it +$ compare-locales ~/firefox-l10n-source/_configs/browser.toml ~/firefox-l10n it it: -changed: 9914 -changed_w: 52351 -keys: 1383 -unchanged: 883 -unchanged_w: 1085 -91% of entries changed +changed 11768 +changed_w 69481 +unchanged 899 +unchanged_w 1094 +keys 426 +92% of entries changed ``` Check the results for duplicated strings and errors. For example, if a new error shows up for a missing variable, it’s likely that a string changed without a new ID and introduced new variables. -The output of `compare-locales` against `mozilla-unified` is going to contain a lot of noise, since it includes all strings that are obsolete for `mozilla-central`, but are still needed for other branches. If you’re interested in only seeing missing strings, i.e. strings that need to be added to the l10n repository, you can `grep` the results by piping the output to `egrep '^\s*\+'`. - -``` -$ compare-locales --unified ~/src/mozilla-unified/browser/locales/l10n.toml ~/l10n it | egrep '^\s*\+' -``` - -### Push reviewed strings to gecko-strings +You could also run `compare-locales` against the `gecko-dev` (git) or `mozilla-unified` (hg) repository. However, the output is going to contain a lot of noise, since it includes all strings that are obsolete for `mozilla-central`, but are still needed for other branches. If you’re interested in only seeing missing strings, i.e. strings that need to be added to the l10n repository, you can `grep` the results by piping the output to `egrep '^\s*\+'`. -If there are no issues in `gecko-strings-quarantine` and no pending [Fluent migrations](../firefox_desktop/fluent_migrations.md), the next step is to push changes to `gecko-strings` and expose content to tools. - -Before pushing, [this script](https://github.com/flodolo/scripts/blob/main/mozilla_l10n/gecko_strings/gecko-quarantine-diff.py) can be used to compare the tip of the two repositories (it requires a local clone of `gecko-strings-quarantine`). - -One time setup: after you cloned `gecko-strings-quarantine` on your system, you need to edit its `.hg/hgrc` file, and add `gecko-strings` as path. While you only need `https` for pulling the quarantine repository, you need `ssh` in order to push updates to `gecko-strings`. - -The content of `~/l10n/gecko-strings-quarantine/.hg/hgrc` should be similar to this: - -``` -[paths] -default = https://hg.mozilla.org/l10n/gecko-strings-quarantine -gecko-strings = ssh://hg.mozilla.org/l10n/gecko-strings ``` - -To push the current `default` to `gecko-strings`, from the `gecko-strings-quarantine` folder simply run: - -``` -$ hg push -r default gecko-strings +$ compare-locales ~/mozilla-unified/browser/locales/l10n.toml ~/l10n it | egrep '^\s*\+' ``` -Instead of `default`, you can also push a specific changeset, e.g. +### Merge reviewed strings to main branch of firefox-l10n-source -``` -$ hg push -r 4c05bc050007 gecko-strings -``` +If there are no issues in the `update` branch of `firefox-l10n-source` and no pending [Fluent migrations](../firefox_desktop/fluent_migrations.md), the next step is to merge the `update` branch into the `main` repository by approving and merging the [pull request](https://github.com/mozilla-l10n/firefox-l10n-source/pulls) in GitHub. Once merged this will expose content to Pontoon. diff --git a/src/tools/mercurial/creating_mercurial_patch.md b/src/tools/mercurial/creating_mercurial_patch.md index 8e6577a4..14f8dfb6 100644 --- a/src/tools/mercurial/creating_mercurial_patch.md +++ b/src/tools/mercurial/creating_mercurial_patch.md @@ -198,116 +198,3 @@ More information about this workflow are available in the following pages: Once the patch has been reviewed, you have two options: * If you have L3 access to the repository, you can use [Lando](https://moz-conduit.readthedocs.io/en/latest/lando-user.html) to land your commit directly. If your reviewer has it, you can ask them to land. - -## Creating a patch using Queues - -### Creating a patch - -Consider the following example: setting up productization for Urdu (ur) on Firefox desktop (l10n repository). The first step is to create a `region.properties` file. Move into the repository folder and check its status: - -```BASH -$ cd ~/mozilla/mercurial/l10n/ur/l10n-central - -$ hg status -? browser/chrome/browser-region/region.properties -``` - -File is brand new and needs to be added: - -```BASH -$ hg add browser - -$ hg status -A browser/chrome/browser-region/region.properties -``` - -You need to assign a name to this patch, it’s easy to use a reference to the bug number: for example, if the bug number is 123456, the file could be called `bug123456.patch` (note the added extension `.patch`). - -```BASH -$ hg qnew bug123456.patch -``` - -At this point you will be asked to provide a commit message for your patch (in `nano` if you followed the instructions to set up the environment): write your commit message, then press `CTRL+O` to save the file, `enter` to confirm the proposed filename, and finally `CTRL+X` to exit the editor. - -The commit message should be the same as the bug, for example `Bug 123456 - Set up searchplugins for "ur" and Firefox desktop`. - -You are ready to *pop* the patch out of the queue. Since it’s likely that there are no other patches, you can pop them all with `-a`. - -```BASH -$ hg qpop -a -popping bug123456.patch -patch queue now empty -``` - -The patch is stored inside the `.hg/patches` folder in the root of the repository (in the suggested setup, the full path would be `~/mozilla/mercurial/l10n-central/.hg/patches`). You can copy the file through the command line or the file explorer. For example, on macOS you can open the folder in Finder by typing: - -```BASH -$ open ~/mozilla/mercurial/l10n/ur/l10n-central/.hg/patches -``` - -Or you can copy the file on the Desktop with - -```BASH -$ cp ~/mozilla/mercurial/l10n/ur/l10n-central/.hg/patches/bug123456.patch ~/Desktop -``` - -Now you need to attach the file to Bugzilla and set an appropriate reviewer for it. - -### Updating an existing patch - -Let’s assume that the review found some issues with the patch and you need to update it. The fastest way is to import the .patch file without committing, update the files as needed, and create a new patch using the same process explained above. - -Assuming the file is called `bug123456.patch` and it’s in your desktop, you can move in the repository folder and import the file like this: - -```BASH -$ cd ~/mozilla/mercurial/l10n/ur/l10n-central - -$ hg import --no-commit ~/Desktop/bug123456.patch -``` - -Don’t forget the `--no-commit` part. If you do, the patch will be added to your repository and you’ll need to clone the original repository again. - - -**Note:** You can drag and drop the patch file on the terminal on macOS to get its full path instead of typing it. In other words, type `hg import --no-commit ` (leave an empty space at the end), then drag the icon of the patch on the Terminal’s window: its full path will appear automatically. - - -At this point you’re ready to modify the files, and create a new patch. The only difference is that you will need to use a different filename, for example `bug123456v1.patch`. - -### Landing the patch - -Your patch got a `r+`, so you need to update the commit message to reference the review, import the patch and push it to the remote server. - -**Note:** If the patch is for a repository where you don’t have write access, you don’t need to follow these instructions, only to set the `checkin-needed` flag in Bugzilla. - -Open the .patch file in your editor, find the line with the commit message, and add `r=NICKNAME` to the commit message. For example, the last line in - -``` -# HG changeset patch -# User SOMENAME -# Parent b45d23bc53df9227aa262acb2a5c6b0ab903b76e -Bug 123456 - Set up searchplugins for "ur" and Firefox for Android -``` - -Should become like this, assuming flod is the reviewer. - -``` -Bug 123456 - Set up searchplugins for "ur" and Firefox for Android, r=flod -``` - -Then you need to import the patch, this time without the `no-commit` parameter. - -```BASH -$ cd ~/mozilla/mercurial/l10n/ur/l10n-central - -$ hg import ~/Desktop/bug123456.patch -``` - -The patch has been imported and committed. If you get an error while applying the patch, check if the editor you’re using hasn’t modified other lines of the patch, for example removing trailing whitespaces. In that case, using `nano` to edit the patch is probably your faster option. - -Now you’re ready to push to the remote repository - -```BASH -$ hg push -``` - -The reply from the server will contain the URL to your changeset: copy and paste that URL in the bug. If the bug only contains a patch to region.properties, you can close the bug after landing; if it also contains a patch stored in MozReview, the bug will be automatically closed when the patch is merged to mozilla-central. diff --git a/src/tools/mercurial/setting_mercurial_environment.md b/src/tools/mercurial/setting_mercurial_environment.md index ef147dab..73216234 100644 --- a/src/tools/mercurial/setting_mercurial_environment.md +++ b/src/tools/mercurial/setting_mercurial_environment.md @@ -35,18 +35,6 @@ It’s divided into sections, with section names between square parentheses. For username = something@example.com ``` -## Set up SSH access for l10n repositories - -In order to be able to commit directly to l10n repositories, you need SSH access via your LDAP account (both obtained through a bug, where you also provide your SSH key). - -Assuming your SSH key is stored in `~/.ssh/id_rsa`, your `.ssh/config` file should have a line that looks like this - -``` -Host hg.mozilla.org -User YOUR_LDAP_EMAIL_ADDRESS -IdentityFile ~/.ssh/id_rsa -``` - ## Set up MozPhab for Phabricator Detailed instructions and explanations are available in [the official documentation](https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#setting-up-mozphab).