Document not found (404)
+This URL is invalid, sorry. Please use the navigation bar or search to continue.
+ +diff --git a/book/.nojekyll b/book/.nojekyll new file mode 100644 index 0000000000..f17311098f --- /dev/null +++ b/book/.nojekyll @@ -0,0 +1 @@ +This file makes sure that Github Pages doesn't process mdBook's output. diff --git a/book/404.html b/book/404.html new file mode 100644 index 0000000000..926e14de31 --- /dev/null +++ b/book/404.html @@ -0,0 +1,226 @@ + + +
+ + +This URL is invalid, sorry. Please use the navigation bar or search to continue.
+ +The documentation in this repository pertains to the application-services library, primarily the sync and storage components, firefox account client and the nimbus-sdk experimentation client.
+The markdown is converted to static HTML using mdbook. To add a new document, you need to add it to the SUMMARY.md file which produces the sidebar table of contents.
+The mdbook
crate is required in order to build the documentation:
cargo install mdbook mdbook-mermaid mdbook-open-on-gh
+
+The repository documents are be built with:
+./tools/build.docs.sh
+
+The built documentation is saved in build/docs/book
.
We want to record architectural decisions made in this project. +Which format and structure should these records follow?
+Chosen option: "MADR 2.1.2", because
+Technical Story: #4101
+We no longer want to depend on SQLCipher and want to use SQLite directly for build complexity and concerns over the long term future of the rust bindings. The encryption approach taken by SQLCipher means that in practice, the entire database is decrypted at startup, even if the logins functionality is not interacted with, defeating some of the benefits of using an encrypted database.
+The per-field encryption in autofill, which we are planning to replicate in logins, separates the storage and encryption logic by limiting the storage layer to the management of encrypted data. Applying this approach in logins will break the existing validation and deduping code so we need a way to implement per-field encryption while supporting the validation and de-duping behavior.
+Chosen Option: "Reduce the API functions that require the encryption key and pass the key to the remaining functions" because it will not require a security review as similar to the approach we have established in the codebase.
+Description
+Currently the below logins API functions would require the per-field encryption key:
+Note:
+get_all
, get_by_base_domain
, and get_by_id
functions will require the encryption key because they call the validate and fixup logic, not because we want to return logins with decrypted data.Propsed changes:
+add
and update
functions into a new add_or_update
function
+time_password_changed
fieldimport_multiple
function
+potential_dupes_ignoring_username
and check_valid_with_no_dupes
from the API
+add_or_update
functiondecrypt_and_fixup_login
function that both decrypts a login and performs the validate and fixup logic
+get_all
, get_by_base_domain
, and get_by_id
API functions to perform the fixup logicMaking the above changes will reduce the API functions requiring the encryption key to the following:
+add_or_update
decrypt_and_fixup_login
import_multiple
Pros
+Cons
+add_or_update
and import_multiple
Description
+Unlike the first option, the publicly exposed login API would only handle decrypted login records and all encryption is internal (which works because we always have the key). Any attempt to use the API will fail as the login records are not encrypted or decrypted if the key is not available.
+Proposed changes:
+add
and update
functions into add_or_update
potential_dupes_ignoring_username
and check_valid_with_no_dupes
from the APIPros
+Cons
+Some of our users have corrupt SQLite databases and this makes the related +component unusable. The best way to deal with corrupt databases is to simply +delete the database and start fresh (#2628). However, we only want to do this +for persistent errors, not transient errors like programming logic errors, disk +full, etc. This ADR deals with 2 related questions:
+Decision B follows from the choice of A. Since we're being conservative in +identifying errors, we can delete the database file with relative confidence.
+"Check for errors for all database operations and compare against known +corruption error types" also seems like a reasonable solution that we may +pursue in the future, but we decided to wait for now. Checking for errors +during opening time is the simpler solution to implement and should fix the +issue in many cases. The plan is to implement that first, then monitor +sentry/telemetry to decide what to do next.
+This option would be similar to 1, but instead of deleting the file we would +move it to a backup location. When we started up, we could look for backup +files and try to import lost data.
+wipe()
on the database, we could also
+delete any backup data.Our iOS consumers currently obtain application-services as a pre-compiled .framework
bundle
+distributed via Carthage. The current setup is not
+compatible with building on new M1 Apple Silicon machines and has a number of other problems.
+As part of a broader effort to modernize the build process of iOS applications at Mozilla,
+we have been asked to re-evaluate how application-services components are dsitributed for iOS.
See Problems with the current setup for more details.
+MozillaAppServices
Carthage build to a similar
+all-in-one Swift Package, distributed as a binary artifact.MozillaAppServices
Carthage build into a separate
+Swift Package target for each component, with a shared dependency on pre-compiled
+Rust code as a binary artiact.Chosen option: (D) Distribute multiple source-based Swift Packages, with pre-compiled Rust code.
+This option will provide the best long-term consumer experience for iOS developers, and +has the potential to simplify maintenance for application-services developers after an +initial investment of effort.
+import MozillaAppServices
with independent imports of each component.We will maintain the existing Carthage build infrastructure in the application-services repo and continue publishing a pre-built Carthage framework, +to support firefox-ios until they migrate to Swift Packages.
+We will add an additional iOS build task in the application-services repo, that builds just the Rust code as a .xcframework
bundle.
+An initial prototype shows that this can be achieved using a relatively straightforward shell script, rather than requiring a second Xcode project.
+It will be published as a .zip
artifact on each release in the same way as the current Carthage framework.
+The Rust code will be built as a static library, so that the linking process of the consuming application can pull in
+just the subset of the Rust code that is needed for the components it consumes.
We will initially include only Nimbus and its dependencies in the .xcframework
bundle,
+but will eventually expand it to include all Rust components (including Glean, which will continue
+to be included in the application-services
repo as a git submodule)
We will create a new repository rust-components-swift
to serve as the root of the new Swift Package distribution.
+It will import the application-services
repository as a git submodule. This will let us iterate quickly on the
+Swift packaging setup without impacting existing consumers.
We will initially include only Nimbus and its dependencies in this new repository, and the Nimbus swift code
+it will depend on Glean via the external glean-swift
package. In the future we will publish all application-services
+components that have a Swift interface through this repository, as well as Glean and any future Rust components.
+(That's why the repository is being given a deliberately generic name).
The rust-components-swift
repo will contain a Package.swift
file that defines:
.xcframework
bundle of Rust code.We will add automation to the rust-components-swift
repo so that it automatically tracks
+releases made in the application-services
repo and creates a corresponding git tag for
+the Swift package.
At some future date when all consumers have migrated to using Swift packages, we will remove +the Carthage build setup from the application-services repo.
+At some future date, we will consider whether to move the Package.swift
definition in to the application-services
repo,
+or whether it's better to keep it separate. (Attempting to move it into the application-services
will involve non-trivial
+changes to the release process, because the checksum of the released .xcframework
bundle needs to be included in
+the release taged version of the Package.swift
file.)
In this option, we would make no changes to our iOS build and publishing process.
+This option isn't really tractable for us, but it's included for completeness.
+In this option, we would try to change our iOS build and publising process as little +as possible, but use Carthage's recent support for building platform-independent +XCFrameworks in order +to support consumers running on M1 Apple Silicon.
+Overall, current circumstances feel like a good opportunity to invest a little more +time in order to set ourselves up for better long-term maintainability +and happier consumers. The main benefit of this option (it's quicker!) is less attractive +under those circumstances.
+In this option, we would compile the Rust code and Swift code for all our components into
+a single .xcframework
bundle, and then distribute that as a
+binary artifact via Swift Package. This is similar to the approach
+currently taken by Glean (ref Bug 1711447)
+except that they only have a single component.
Overall, this option would be a marked improvement on the status quo, but leaves out some potential +improvements. For not that much more work, we can make some of the "Neutral" and "Bad" points +here into "Good" points.
+In this option, we would compile just the Rust code for all our components into a single
+.xcframework
bundle and distribute that as a binary artifact via Swift Package.
+We would then declare a separate Swift source target for the Swift wrapper of each component,
+each depending on the compiled Rust code but appearing as a separate item in the Swift package
+definition.
The only downside to this option appears to be the amount of work involved, but an initial +prototype has given us some confidence that the change is tractable and that it may lead +to a system that is easier to maintain over time. It is thus our preferred option.
+It doesn't build for M1 Apple Silicon machines, because it's not possible to support
+both arm64 device builds and arm64 simulator builds in a single binary .framework
.
Carthage is dispreferred by our current iOS consumers.
+We don't have much experience with the setup on the current Application Services team, +and many of its details are under-documented. Changing the build setup requires Xcode +and some baseline knowledge of how to use it.
+All components are built as a single Swift module, meaning they can see each other's +internal symbols and may accidentally conflict when naming things. For example we can't +currently have two components that define a structure of the same name.
+Consumers can only use the pre-built binary artifacts if they are using the same
+version of Xcode as was used during the application-services build. We are not able
+to use Swift's BUILD_LIBRARY_FOR_DISTRIBUTION
flag to overcome this, because some
+of our dependencies do not support this flag (specifically, the Swift protobuf lib).
Technical Story: https://mozilla-hub.atlassian.net/browse/SDK-323
+As an experimenter, I would like to run experiments early on a user's first run of the application. However, the experiment data is only available on the second run. We would like to have that experiment data available before the user's first run. +For more information: https://docs.google.com/document/d/1Qw36_7G6XyHvJZdM-Hxh4nqYZyCsYajG0L5mO33Yd5M/edit
+initial_experiments.json
that defines the experiments that will be applied early on the first runNone of the options were feasible, so for now we are sticking with option (A) Do Nothing until there are experiments planned that are expected to run on early startup on the first run, then we will revaluate our options.
+The (B) Bundle Experiment data with app on release option was rejected mainly due to difficulty in disabling experiments and pausing enrollments. This can create a negative user experience as it prevents us from disabling any problematic experiments. Additionally, it ties experiment creation with application release cycles.
+The (C) Retrieve Experiment data on first run, and deal with delay option was rejected due to the fact it changes the Nimbus SDK will no longer be idempotent,and the possibility of introducing undesirable UI.
+Status: proposed
+Discussion: https://github.com/mozilla/application-services/pull/5302
+Deciders:
+Date: 2022-12-16
+Mozilla’s mobile browsers have a requirement to access the remote settings service, but currently lack any libraries or tools which are suitable without some work. +A concrete use case is the management of search engine configurations, which are stored in Remote Settings for Firefox Desktop, but shipped as individual files on our mobile browsers, requiring application releases for all changes.
+A constraint on any proposed solutions is that this work will be performed by Mozilla's mobile team, who have limited experience with Rust, and that it is required to be completed in Q1 2023.
+This document identifies the requirements, then identifies tools which already exist and are close to being suitable, then identifies all available options we can take, and outlines our decision.
+The requirements are for a library which is able to access Mozilla’s Remote Settings service and return the results to our mobile browsers. +This list of requirements is not exhaustive, but instead focuses on the requirements which will drive our decision making process. +As such, it identifies the non-requirements first.
+The following items all may have some degree of desirability, but they are not hard requirements for the initial version
+The requirements we do have for the initial version are:
+We have identified the following libraries which may be suitable for this project.
+There is a version of the remote settings client in desktop, written in Javascript. +It has been used and been relatively stable since at least 2018, so can be considered very capable, but the roadblock to it being suitable for use by our mobile browsers is that it is written in Javascript, so while it might be possible to expose it to Android via geckoview, there’s no reasonable path to have it made available to iOS.
+There is an existing remote settings client on github. +This client is written in Rust and has evolved over a number of years. +The most recent changes were made to support being used in Merino, which was re-written in Python, so there are no known consumers of this library left.
+The main attributes of this library relevant to this discussion are:
+The nimbus-sdk is a component in the application-services repository written in Rust. It has client code which talks to the remote-settings server and while this has only actually been used with the "Nimbus" collection there's no reason to believe it can't be used in the more general case. +The main attributes of this library relevant to this discussion are:
+The requirements of this client are such that writing new libraries in Kotlin and Swift is currently a realistic option. +However, we are rejecting this option because we don’t want to duplicate the effort required to write and maintain two libraries - inevitably, the features and capabilities will diverge. +Future requirements such as supporting content signature verification would lead to significant duplication.
+Writing a new library from scratch in Rust and exposing it via UniFFI so it can be used by both platforms is also a possibility. +However, we are rejecting this option because existing Rust libraries already exist, so we would be better served by modifying or forking one of the existing libraries.
+Modifying or forking the existing client is an attractive option. +It would require a number of changes - the async capabilities would probably need to be removed (using a Rust event loop in our mobile browsers is something we are trying to avoid until we better understand the implications given these browsers already have an event loop and their own threading model).
+The persistence model used by this library is something that is not a requirement for the new library, which isn’t itself a problem, but it probably would preclude being able to use this library by Nimbus - so the end result is that we would effectively have two remote-settings clients written in Rust and used by our browsers.
+Some API changes would probably be required to make it suitable for use by UniFFI would also be necessary, but these would be tractable.
+We would need to update nimbus to use this client, which would almost certainly require moving this client into the application-services repository to avoid the following issues:
+Splitting the existing client out from Nimbus in a way that allows Nimbus to continue to use it, while also making it available for stand-alone use is also an attractive option.
+In particular, the feature set of that client overlaps with the requirements of the new library - no local persistence is necessary and no signature verification is required. It is already used by a component which is exposed via UniFFI.
+Note that this option does not preclude both Nimbus and this new crate from moving to the existing remote settings client at some point in the future. +A key benefit of this decision is that it keeps nimbus and the new crate using the same client, so updating both to use a different client in the future will always remain an option.
+We have chosen Option 3 because it allows us to reuse the new client in Nimbus, as well as on iOS and on Android with minimal initial development effort. +If the new library ends up growing requirements that are already in the existing remote settings client, we remain able to copy that functionality from that library into this.
+This section is non-normative - ie, is not strictly part of the ADR, but exists +for context.
+This is a very high-level view of the tasks required here.
+Create a new top-level component in the application-services repository, identify the exact API we wish to expose for this new library, describe this API using UniFFI, then implement the API with “stubs” (eg, using rust todo!()
or similar). This is depicted as RemoteSettings
in the diagram.
Identify which parts of Nimbus should be factored out into a shared component (depicted as rs-client
in the diagram below) and move that functionality to the new shared component. Of note:
Identify which of the nimbus tests should move to the new client and move them.
+Update Nimbus to take a dependency on the new component and use it, including tests.
+Flesh out the API of the new top-level component using the new shared component (ie, replace the todo!()
macros with real code.)
Identify any impact on the Nimbus android/swift code - in particular, any shared configuration and initialization code identified above in the application-services repo.
+Implement the Android and iOS code in the application-services repo desired to make this an ergonomic library for the mobile platforms.
+Update the mobile code for the UniFFI changes made to Nimbus, if any.
+Implement the mobile code which consumes the new library, including tests.
+Profit?
+This diagram attempts to depict this final layout. Note:
+rs-client
and RemoteSettings
are both new components, everything else already exists. Please do not consider these names as suggestions! Names are hard, I'm sure we can do better.Cargo.toml
)support
, but please ignore that anomaly.flowchart RL + subgraph app-services-support[Shared components in application-services/components/support] + rs-client + other-support-components + end + subgraph app-services-components[Top-level application-services components, in application-services/components] + Nimbus + RemoteSettings + Viaduct + end + subgraph mobile [Code in the mobile repositories] + Fenix + Firefox-iOS + end + + Nimbus -- nimbus.udl --> mobile + RemoteSettings -- remote_settings.udl --> mobile + + rs-client -.-> Nimbus + other-support-components -.-> Nimbus + rs-client -.-> RemoteSettings + other-support-components -.-> RemoteSettings + Viaduct -.-> rs-client + other-support-components -.-> rs-client ++
This section is non-normative - ie, is not strictly part of the ADR, but exists +for context.
+Content Signatures have been explicitly called out as a non-requirement. Because this capability was a sticking point in the desktop version of the remote settings client, and because significant effort was spent on it, it's worth expanding on this here.
+Because https will be enforced for all network requests, the consumers of this library can have a high degree of confidence that:
+Content signatures offer an additional capability of checking the content of a remote settings response matches the signature generated with a secret key owned by Mozilla, independenty of the https certificates used for the request itself.
+This capability was added to the desktop version primarily to protect the integrity of the data at rest. +Because the Desktop client cached the responses on disk, there was a risk that this data could be tampered with - so it was effectively impossible to guarantee that the data finally presented to the application is what was initially sent.
+The main threat-model that required this capability was 3rd party applications installed on the same system where Firefox was installed. +Because of the security model enforced by Desktop operating systems (most notably Windows), there was evidence that these 3rd-party applications would locate and modify the cache of remote-settings responses and modify them in a way that benefited them and caused revenue harm to Mozilla - the most obvious example is changing the search provider settings.
+The reason we are declaring this capability a non-requirement in the initial version is two-fold:
+We have also declared caching of responses a non-requirement, meaning there's no data at rest managed by this library which is vulnerable to this kind of attack.
+The mobile operating systems have far stronger application isolation - in the general case, a 3rd party mobile application is prevented from touching any of the files used by other applications.
+Obviously though, things may change in the future - for example, we might add response caching, so we must be sure to reevaluate this requirement as other requirements change.
+ +A significant part of the project is migrating users’ history from the old database to a new one. To measure risk, we ran a dry-run migration. A dry-run migration runs a background thread in the user’s application and attempts to migrate to a fake database. The dry-run was implemented purely to collect telemetry on the migration to evaluate risk. The results can be found in the following Looker dashboard. Below is a list of observations.
+The following is a list of observations from the experiment:
+Given the observations from the dry-run experiment, the rest of the document examines an approach to answer the question: How can we increase the rate of which migrations end, and simultaneously keep the user’s database size at a reasonable size?
+The user implication of keeping the rate of ended migrations high is that users keep their history, and can interact normally with the URL bar to search history, searching history in the history panel and navigating to sites they visited in the past.
+The user implication of keeping a reasonable database size is that the database is less likely to lock on long queries. Meaning we reduce performance issues when users use the search bar, the history panel and when navigating to sites.
+Finally, it’s important to note that power users and daily active users will be more likely to have large histories and thus:
+We must not lose users’ recent history.
+User’s experience with History must not regress, and ideally should improve.
+Chosen option: Introduce a visit number-based limit for the migration. This option was chosen because given our decision drivers:
+The biggest negative consequence is that Users with more visits than the limit, will lose visits.
+This section describes a suggested limit for the visits. Although it’s backed up with telemetry, the specific number is up for discussion. It’s also important to note that statistical significance was not a part of the analysis. Migration has run for over 16,000 users and although that may not be a statistically significant representation of our population, it’s good enough input to make an educated suggestion.
+We will look at https://mozilla.cloud.looker.com/looks/1078 which demonstrates a distribution of our users based on the number of history visits. Note that the chart is based on our release population.
+We will look at https://mozilla.cloud.looker.com/looks/1081. The chart demonstrates the rate at which migrations end by the number of visits. We bucket users in buckets of 10,000 visits.
+Based on the above, we’re suggesting a limit of 10,000 visits because
+places.db
to 75MiBThis log lists the architectural decisions for MADR.
+ +For new ADRs, please use template.md as basis. +More information on MADR is available at https://adr.github.io/madr/. +General information about architectural decision records is available at https://adr.github.io/.
+ +Use UniFFI, which can produce Kotlin +bindings for your Rust code from an interface definition file.
+If UniFFI doesn't currently meet your needs, please open an issue to discuss how the tool can +be improved.
+As a last resort, you can make hand-written bindings from Rust to Kotlin,
+essentially manually performing the steps that UniFFI tries to automate
+for you: flatten your Rust API into a bunch of pub extern "C"
functions,
+then use JNA to call them
+from Kotlin. The details of how to do that are well beyond the scope of
+this document.
Published packages should be named org.mozilla.appservices.$NAME
where $NAME
+is the name of your component, such as logins
. The Java namespace in which
+your package defines its classes etc should be mozilla.appservices.$NAME.*
.
Add it to .buildconfig-android.yml
in the root of this repository.
+This will cause it to be automatically included as part of our release
+publishing pipeline.
Assuming that you're building the Rust code as part of the application-services
+build and release process, your pub extern "C"
API should always be available
+from a file named libmegazord.so
.
There are a number of them. The issue boils down to the fact that you need to be +completely certain that a JVM is associated with a given thread in order to call +java code on it. The difficulty is that the JVM can GC its threads and will not +let rust know about it.
+JNA can work around this for us to some extent, at the cost of some complexity.
+The approach it takes is essentially to spawn a thread for each callback
+invocation. If you are certain you’re going to do a lot of callbacks and they
+all originate on the same thread, you can have them all run on a single thread
+by using the CallbackThreadInitializer
.
With the help of JNA's workarounds, calling back from Rust into Kotlin isn’t too bad +so long as you ensure that Kotlin cannot GC the callback while rust code holds onto it +(perhaps by stashing it in a global variable), and so long as you can either accept the overhead of extra threads being instantiated on each call or are willing to manage +the threads explicitly.
+Note that the situation would be somewhat better if we used JNI directly (and +not JNA), but this would cause us to need to generate different Rust FFI code for +Android than for iOS.
+Ultimately, in any case where there is an alternative to using a callback, you +should probably pursue that alternative.
+For example if you're using callbacks to implement async I/O, it's likely better to +move to doing a blocking call, and have the calling code dispatch it on a background +thread. It’s very easy to run such things on a background thread in Kotlin, is in line +with the Android documentation on JNI usage, and in our experience is vastly simpler +and less painful than using callbacks.
+(Of course, not every case is solvable like this).
+We get a couple things from using JNA that we wouldn't with JNI.
+We are able to use the same Rust FFI code on all platforms. If we used JNI we'd +need to generate an Android-specific Rust FFI crate that used the JNI APIs, and +a separate Rust FFI crate for exposing to Swift.
+JNA provides a mapping of threads to callbacks for us, making callbacks over +the FFI possible. That said, in practice this is still error prone, and easy +to misuse/cause memory safety bugs, but it's required for cases like logging, +among others, and so it is a nontrivial piece of complexity we'd have to +reimplement.
+However, it comes with the following downsides:
+bool
as a typedef for a 32-bit int
).ffi_support
docs, but a
+major one is when to use Pointer
vs String
(getting this wrong will
+often work, but may corrupt memory).We aim to avoid triggering these bugs by auto-generating the JNA bindings +rather than writing them by hand.
+packagingOptions { doNotStrip "**/*.so" }
line from the
+build.gradle file of the component you want to debug.unsafe { std::ptr::write_volatile(0 as *const _, 1u8) }
usually is
+what I do).breakpoint set --name foo
, or breakpoint set --file foo.rs --line 123
.
+I don't know how to bring up this prompt reliably, so I often do step 1 to
+get it to appear, delete the crashing code, and then set the
+breakpoint using the CLI. This is admittedly suboptimal.This document provides an overview of the build-and-publish pipeline used to make our work +in this repo available to consuming applications. It's intended both to document the pipeline +for development and maintenance purposes, and to serve as a basic analysis of the integrity +protections that it offers (so you'll notice there are notes and open questions in place where +we haven't fully hashed out all those details).
+The key points:
+#releaseduty-mobile
Slack channel.
+Our main point of contact is @mihai.For Android consumers these are the steps by which Application Services code becomes available, +and the integrity-protection mechanisms that apply at each step:
+main
via pull request.
+main
without review.main
.
+For iOS consumers the corresponding steps are:
+main
via pull request, as above.main
, as above.For consuming in mozilla-central, see how to vendor components into mozilla-central +
+This is a diagram of the pipeline as it exists (and is planned) for the Nimbus SDK, one of the +libraries in Application Services: +(Source: https://miro.com/app/board/o9J_lWx3jhY=/)
+ +There's an appsvc-moz github account owned by one of the application-services team (currently markh, but we should consider rotating ownership). +Given only 1 2fa device can be connected to a github account, multiple owners doesn't seem practical. +In most cases, whenever a github account needs to own a secret for any CI, it will be owned by this account.
+CircleCI config requires a github token (owned by @appsvc-moz). This is a "personal access token" +obtained via github's Settings -> Developer Settings -> Personal Access Tokens -> Classic Token. This token:
+Once you have generated the token, it must be added to https://app.circleci.com/settings/project/github/mozilla/application-services/environment-variables as the environment variable GITHUB_TOKEN
When working on Application Services, it's important to set up your environment for building the Rust code and the Android or iOS code needed by the application.
+Building for the first time is more complicated than a typical Rust project. +To build for an end-to-end experience that enables you to test changes in +client applications like Firefox for Android (Fenix) and Firefox iOS, there are a number of build +systems required for all the dependencies. The initial setup is likely to take +a number of hours to complete.
+Complete this section before moving to the android/iOS build instructions.
+ $ git clone https://github.com/mozilla/application-services # (or use the ssh link)
+ $ cd application-services
+ $ git submodule update --init --recursive
+
+Install Rust: install via rustup
+Install your system dependencies:
+Install the system dependencies required for building NSS
+apt install gyp
(required for NSS)apt install ninja-build
apt install python3
apt install zlib1g-dev
apt install perl
apt install patch
Install the system dependencies required for SQLcipher
+apt install tclsh
(required for SQLcipher)xcode-select --install
brew install ninja python
which python3
maps to the freshly installed homebrew python.
+source
the profile before continuing:
+alias python3=$(brew --prefix)/bin/python3
+
+python
maps to the same Python version. You may have to
+create a symlink:
+PYPATH=$(which python3); ln -s $PYPATH `dirname $PYPATH`/python
+
+wget https://bootstrap.pypa.io/ez_setup.py -O - | python3 -
+git clone https://chromium.googlesource.com/external/gyp.git ~/tools/gyp
+cd ~/tools/gyp
+python3 setup.py install
+
+~/tools/gyp
to your path:
+export PATH="~/tools/gyp:$PATH"
+
+export PATH="$PATH:$(brew --prefix)/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin"
+
+Install windows build tools
+++Why Windows Subsystem for Linux (WSL)?
+It's currently tricky to get some of these builds working on Windows, primarily due to our use of SQLcipher. By using WSL it is possible to get builds working, but still have them published to your "native" local maven cache so it's available for use by a "native" Android Studio.
+
sudo apt install unzip
sudo apt install python3
Note: must be python 3.6 or latersudo apt install build-essential
sudo apt-get install zlib1g-dev
sudo apt install tcl-dev
Check dependencies and environment variables by running: ./libs/verify-desktop-environment.sh
++Note that this script might instruct you to set some environment variables, set those by adding them to your +
+.zshrc
or.bashrc
so they are set by default on your terminal. If it does so instruct you, you must +run the command again after setting them so the libraries are built.
cargo test
Once you have successfully run ./libs/verify-desktop-environment.sh
and cargo test
you can move to the Building for Fenix and Building for iOS sections below to setup your local environment for testing with our client applications.
The following instructions assume that you are building application-services
for Fenix, and want to take advantage of the
+Fenix Auto-publication workflow for android-components and application-services.
JAVA_HOME
to point to the JDK 17 installation directory.ANDROID_SDK_ROOT
and ANDROID_HOME
to the Android Studio sdk location and add it to your rc file (either .zshrc
or .bashrc
depending on the shell you use for your terminal).Configure menu > System Settings > Android SDK > SDK Tools > NDK > Show Package Details > NDK (Side by side)
+./libs/verify-android-environment.sh
Note: For non-Ubuntu linux versions, it may be necessary to execute $ANDROID_HOME/tools/bin/sdkmanager "build-tools;26.0.2" "platform-tools" "platforms;android-26" "tools"
. See also this gist for additional information.
Configure maven to use the native windows maven repository - then, when doing ./gradlew install
from WSL, it ends up in the Windows maven repo. This means we can do a number of things with Android Studio in "native" windows and have then work correctly with stuff we built in WSL.
sudo apt install maven
~/.m2
folder~/.m2
create a file called settings.xml
{username}
with your username: <settings>
+ <localRepository>/mnt/c/Users/{username}/.m2/repository</localRepository>
+ </settings>
+
+gem install xcpretty
./libs/verify-ios-environment.sh
to check your setup and environment
+variables../megazords/ios-rust/build-xcframework.sh
to build all the binaries needed to consume a-s in iOSOnce the script passes, you should be able to run the Xcode project.
+++Note: The built Xcode project is located at
+megazords/ios-rust/MozillaTestServices.xcodeproj
.
++Note: This is mainly for testing the rust components, the artifact generated in the above steps should be all you need for building application with application-services
+
Detailed steps to build Firefox iOS against a local application services can be found this document
+ +Anyone is welcome to help with the Application Services project. Feel free to get in touch with other community members on Matrix or through issues on GitHub.
+Participation in this project is governed by the +Mozilla Community Participation Guidelines.
+You can file issues on GitHub. Please try to include as much information as you can and under what conditions +you saw the issue.
+Build instructions are available in the building
page. Please let us know if you encounter any pain-points setting up your environment.
Below are a few different queries you can use to find appropriate issues to work on. Feel free to reach out if you need any additional clarification before picking up an issue.
+good-first-issue
good-second-issue
label.Patches should be submitted as pull requests (PRs).
+++When submitting PRs, We expect external contributors to push patches to a fork of
+application-services
. For more information about submitting PRs from forks, read GitHub's guide.
Before submitting a PR:
+./automation/tests.py changes
. The script will calculate which components were changed and run test suites, linters and formatters against those components. Because the script runs a limited set of tests, the script should execute in a fairly reasonable amount of time.
+swiftformat --swiftversion 5
on the modified code.When submitting a PR:
+main
branch.This project is production Mozilla code and subject to our engineering practices and quality standards. Every patch must be peer reviewed by a member of the Application Services team.
+ +This repository uses third-party code from a variety of sources, so we need to be mindful +of how these dependencies will affect our consumers. Considerations include:
+We're still evolving our policies in this area, but these are the +guidelines we've developed so far.
+Unlike Firefox,
+we do not vendor third-party source code directly into the repository. Instead we rely on
+Cargo.lock
and its hash validation to ensure that each build uses an identical copy
+of all third-party crates. These are the measures we use for ongoing maintence of our
+existing dependencies:
Cargo.lock
into the repository.--locked
flag to cargo build
, as an additional
+assurance that the existing Cargo.lock
will be respected.main
branchAdding a new dependency, whether we like it or not, is a big deal - that dependency and everything +it brings with it will become part of Firefox-branded products that we ship to end users. +We try to balance this responsibility against the many benefits of using existing code, as follows:
+unsafe
, or of code that is unusually resource-intensive to build.Updating to new versions of existing dependencies is a normal part of software development +and is not accompanied by any particular ceremony.
+We currently depend only on the following Kotlin dependencies:
+ +We currently depend on the following developer dependencies in the Kotlin codebase, +but they do not get included in built distribution files:
+No additional Kotlin dependencies should be added to the project unless absolutely necessary.
+We currently do not depend on any Swift dependencies. And no Swift dependencies should be added to the project unless absolutely necessary.
+We currently depend on local builds of the following system dependencies:
+ +No additional system dependencies should be added to the project unless absolutely necessary.
+ +On a high level, Firefox Sync has three main components:
+Additionally, the token server assists in providing metadata to Firefox, so that it knows which sync server to communicate with. +
+Since we have multiple Firefox apps (Desktop, iOS, Android, Focus, etc) Firefox sync can sync across platforms. Allowing users +to access their up-to-date data across apps and devices. +
+Before our Rust Components came to life, each application had its own implementation of the sync and FxA client protocols. +This lead to duplicate logic across platforms. This was problematic since any modification to the sync or FxA client business logic +would need to be modified in all implementations and the likelihood of errors was high. +
+Currently, we are in the process of migrating many of the sync implementation to use our Rust Component strategy. +Fenix primarily uses our Rust Components and iOS has some integrated as well. Additionally, Firefox Desktop also uses +one Rust component (Web Extension Storage).
+The Rust components not only unify the different implementations of sync, they also provide a convenient local storage for the apps. +In other words, the apps can use the components for storage, with or without syncing to the server. +
+The following table has the status of each of our sync Rust Components +| Application\Component | Bookmarks | History | Tabs | Passwords | Autofill | Web Extension Storage | FxA Client | +|-----------------------|-----------|---------|------|-----------|----------|-----------------------|------------| +| Fenix | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | ✔️ | +| Firefox iOS | ✔️ | | ✔️ | ✔️ | | | ✔️ | +| Firefox Desktop | | | | | | ✔️ | | +| Focus | | | | | | | |
+In an aspirational future, all the applications would use the same implementation for Sync. +However, it's unlikely that we would migrate everything to use the Rust components since some implementations +may not be prioritized, this is especially true for desktop which already has stable implementations. +That said, we can get close to this future and minimize duplicate logic and the likelihood of errors. +
+You can edit the diagrams in the following lucid chart (Note: Currently only Mozilla Employees can edit those diagrams): https://lucid.app/lucidchart/invitations/accept/inv_ab72e218-3ad9-4604-a7cd-7e0b0c259aa2
+Once they are edited, you can re-import them here by replacing the old diagrams in the docs/diagrams
directory on GitHub. As long as the
+names are the same, you shouldn't need to edit those docs!
The data below has been added as a tool for future pragma analysis work and is expected to be useful so long as our pragma usage remains stable or this doc is kept up-to-date. This should help us understand our current pragma usage and where we may be able to make improvements.
+Pragma | Value | Component | Notes |
---|---|---|---|
cache_size | -6144 | places | |
foreign_keys | ON | autofill, places, tabs, webext-storage | |
journal_mode | WAL | autofill, places, tabs, webext-storage | |
page_size | 32768 | places | |
secure_delete | true | logins | |
temp_store | 2 | autofill, logins, places, tabs, webext_storage | Setting temp_store to 2 (MEMORY) is necessary to avoid SQLITE_IOERR_GETTEMPPATH errors on Android (see here for details) |
wal_autocheckpoint | 62 | places | |
wal_checkpoint | PASSIVE | places | Used in the sync finished step in history and bookmarks syncing and in the places run_maintenance function |
Each Rust component published by Application Services is conceptually a stand-alone library, but for
+distribution we compile all the rust code for all components together into a single .so
file. This
+has a number of advantages:
This process is affectionately known as "megazording" and the resulting artifact as a megazord library.
+On Android, the situation is quite complex due to the way packages and dependencies are managed. +We need to distribute each component as a separate Android ARchive (AAR) that can be managed as a dependency +via gradle, we need to provide a way for the application to avoid shipping rust code for components that it +isn't using, and we need to do it in a way that maintanins the advantages listed above.
+This document describes our current approach to meeting all those requirements on Android. Other platforms +such as iOS are not considered.
+We publish a separate AAR for each component (e.g. fxaclient, places, logins) which contains
+just the Kotlin wrappers that expose the relevant functionality to Android. Each of these AARs depends on a separate
+shared "megazord" AAR in which all the rust code has been compiled together into a single .so
file.
+The application's dependency graph thus looks like this:
This generates a kind of strange inversion of dependencies in our build pipeline:
+It's a little odd, but it has the benefit that we can use gradle's dependency-replacement features to easily +manage the rust code that is shipping in each application.
+By default, an application that uses any appservices component will include the compiled rust code +for all appservices components.
+To reduce its overall code size, the application can use gradle's module replacement +rules +to replace the "full-megazord" AAR with a custom-built megazord AAR containing only the components it requires. +Such an AAR can be built in the same way as the "full-megazord", and simply avoid depending on the rust +crates for components that are not required.
+To help ensure this replacement is done safely at runtime, the mozilla.appservices.support.native
package
+provides helper functions for loading the correct megazord .so
file. The Kotlin wrapper for each component
+should load its shared library by calling mozilla.appservices.support.native.loadIndirect
, specifying both
+the name of the component and the expected version number of the shared library.
The full-megazord AAR contains compiled rust code that targets various Android platforms, and is not
+suitable for running on a Desktop development machine. In order to support integration with unittest
+suites such as robolectric, each megazord has a corresponding Java ARchive (JAR) distribution named e.g.
+full-megazord-forUnitTests.jar
. This contains the rust code compiled for various Desktop architectures,
+and consumers can add it to their classpath when running tests on a Desktop machine.
This setup mostly works, but has a handful of rough edges.
+The build.gradle
for each component needs to declare an explicit dependency on project(":full-megazord")
,
+otherwise the resulting AAR will not be able to locate the compiled rust code at runtime. It also needs to
+declare a dependency between its build task and that of the full-megazord, for reasons. Typically this looks something
+like:
tasks["generate${productFlavor}${buildType}Assets"].dependsOn(project(':full-megazord').tasks["cargoBuild"])
+
+In order for unit tests to work correctly, the build.gradle
for each component needs to add the rustJniLibs
+directory of the full-megazord project to its srcDirs
, otherwise the unittests will not be able to find and load
+the compiled rust code. Typically this looks something like:
test.resources.srcDirs += "${project(':full-megazord').buildDir}/rustJniLibs/desktop"
+
+The above also means that unittests will not work correctly when doing local composite builds, +because it's unreasonable to expect the main project (e.g. Fenix) to include the above in its build scripts.
+ +Some application-services components collect telemetry using the Glean SDK.
+Products that send telemetry via Glean must request a data-review following +the Firefox Data Collection process +before integrating any of the components listed below.
+ +Like almost all Rust projects, the entire point of the application-services +components is that they be used by external projects. If these components +use Rust features available in only the very latest Rust version, this will +cause problems for projects which aren't always able to be on that latest +version.
+Given application-services is currently developed and maintained by Mozilla +staff, it should be no surprise that an important consideration is +mozilla-central (aka, the main Firefox repository).
+It should also come as no surprise that the Rust policy for mozilla-central +is somewhat flexible. There is an official Rust Update Policy Document + +but everything in the future is documented as "estimated".
+Ultimately though, that page defines 2 Rust versions - "Uses" and "Requires", +and our policy revolves around these.
+To discover the current, actual "Uses" version, there is a Meta bug on Bugzilla that keeps +track of the latest versions as they are upgraded.
+To discover the current, actual "Requires" version, see searchfox
+Our official Rust version policy is:
+All components will ship using, have all tests passing, and have clippy emit +no warnings, with the same version mozilla-central currently "uses".
+All components must be capable of building (although not necessarily with +all tests passing nor without clippy errors or other warnings) with the same +version mozilla-central currently "requires".
+This policy only applies to the "major" and "minor" versions - a different +patch level is still considered compliant with this policy.
+All CI for this project will try and pin itself to this same version. At
+time of writing, this means that our circle CI integration
+ and
+rust-toolchain configuration
+will specify the versions (and where possible, the CI configuration file will
+avoid duplicating the information in rust-toolchain
)
We should maintain CI to ensure we still build with the "Requires" version.
+As versions inside mozilla-central change, we will bump these versions +accordingly. While newer versions of Rust can be expected to work correctly +with our existing code, it's likely that clippy will complain in various ways +with the new version. Thus, a PR to bump the minimum version is likely to also +require a PR to make changes which keep clippy happy.
+In the interests of avoiding redundant information which will inevitably +become stale, the circleci and rust-toolchain configuration links above +should be considered the canonical source of truth for the currently supported +official Rust version.
+ +++ + +This is a high level description of the decision highlighted in the ADR that introduced Swift Packages as a strategy to ship our Rust components. That document includes that tradeoffs and why we chose this approach.
+
The strategy includes two main parts:
+Cargo.toml
as a static libraryrust-components-swift
repository which has a Package.swift
that includes the xcframework
and acts as the swift package the consumers importapplication-services
In application-services
, in the megazords/ios-rust
directory, we have the following:
module.modulemap
: The module map tells the Swift compiler how to use C APIs.
+1. MozillaRustComponents.h
: The header is used by the module map as a shortcut to specify all the available header files
+1. Info.plist
: The plist
file specifies metadata about the resulting xcframework. For example, architectures and subdirectories.build-xcframework.sh
script that stitches things together into a full xcframework bundle:
+xcframework
format is not well documented; briefly:
+.zip
file.Info.plist
. The Info.plist
describes what lives in which directory..framework
directory for that architecture.++It's a little unusual that we're building the xcframework by hand, rather than defining it as the build output of an Xcode project. It turns out to be simpler for our purposes, but does risk diverging from the expected format if Apple changes the details of xcframeworks in future Xcode releases.
+
rust-components-swift
repositoryThe repository is a Swift Package for distributing releases of Mozilla's various Rust-based application components. It provides the Swift source code packaged in a format understood by the Swift package manager, and depends on a pre-compiled binary release of the underlying Rust code published from application-services
The rust-components-swift
repo mainly includes the following:
Package.swift
: Defines all the targets
and products
the package exposes.
+Package.swift
also includes where the package gets the xcframework
that application-services
buildsmake_tag.sh
: A script that does the following:
+++ +Consumers would then import the
+rust-components-swift
swift package, by indicating the url of the package on github (i.e https://github.com/mozilla/rust-components-swift) and selecting a version using the git tag.
We've identified the need for a "sync manager" (although are yet to identify a +good name for it!) This manager will be responsible for managing "global sync +state" and coordinating each engine.
+At a very high level, the sync manager is responsible for all syncing. So far, +so obvious. However, given our architecture, it's possible to identify a +key architectural split.
+The embedding application will be responsible for most high-level operations. +For example, the app itself will choose how often regular syncs should +happen, what environmental concerns apply (eg, should I only sync on WiFi?), +letting the user choose exactly what to sync, and so forth.
+A lower-level component will be responsible for the direct interaction with +the engines and with the various servers needed to perform a sync. It will +also have the ultimate responsibility to not cause harm to the service (for +example, it will be likely to enforce some kind of rate limiting or ensuring +that service requests for backoff are enforced)
+Because all application-services engines are written in Rust, it's tempting to +suggest that this lower-level component also be written in Rust and everything +"just works", but there are a couple of complications here:
+For iOS, we hope to integrate with older engines which aren't written in +Rust, even if iOS does move to the new Sync Manager.
+For Desktop, we hope to start by reusing the existing "sync manager" +implemented by Desktop, and start moving individual engines across.
+There may be some cross-crate issues even for the Rust implemented engines. +Or more specifically, we'd like to avoid assuming any particular linkage or +packaging of Rust implemented engines.
+Even with these complications, we expect there to be a number of high-level +components, each written in a platform specific language (eg, Kotlin or Swift) +and a single lower-level component to be implemented in Rust and delivered +as part of the application-services library - but that's not a free-pass.
+Why "a number of high-level components"? Because that is the thing which +understands the requirements of the embedding application. For example, Android +may end up with a single high-level component in the android-components repo +and shared between all Android components. Alternatively, the Android teams +may decide the sync manager isn't generic enough to share, so each app will +have their own. iOS will probably end up with its own and you could imagine +a future where Desktop does too - but they should all be able to share the +low level component.
+The primary responsibilities of the "high level" portion of the sync manager are:
+Manage all FxA interaction. The low-level component will have a way to +communicate auth related problems, but it is the high-level component +which takes concrete FxA action.
+Expose all UI for the user to choose what to sync and coordinate this with +the low-level component. Note that because these choices can be made on any +connected device, these choices must be communicated in both directions.
+Implement timers or any other mechanism to fully implement the "sync +scheduler", including any policy decisions such as only syncing on WiFi, +etc.
+Implement a UI so the user can "sync now".
+Collect telemetry from the low-level component, probably augment it, then +submit it to the telemetry pipeline.
+The primary responsibilities of the "low level" portion of the sync manager are:
+Manage the meta/global
, crypto/keys
and info/collections
resources,
+and interact with each engine as necessary based on the content of these
+resources.
Manage interaction with the token server.
+Enforce constraints necessary to ensure the entire ecosystem is not +subject to undue load. For example, this component should ignore attempts to +sync continuously, or to sync when the services have requested backoff.
+Manage the "clients" collection - we probably can't ignore this any longer, +especially for bookmarks (as desktop will send a wipe command on bookmark +restore, and things will "be bad" if we don't see that command).
+Define a minimal "service state" so certain things can be coordinated with +the high-level component. Examples of these states are "everything seems ok", +"the service requested we backoff for some period", "an authentication error +occurred", and possibly others.
+Perform, or coordinate, the actual sync of the rust implemented engines - +from the containing app's POV, there's a single "sync now" entry-point (in +practice there might be a couple, but conceptually there's a single way to +sync). Note that as below, how non-rust implemented engines are managed is +TBD.
+Manage the collection of (but not the submission of) telemetry from the +various engines.
+Expose APIs and otherwise coordinate with the high-level component.
+Stuff we aren't quite sure where it fits include:
+The above has been carefully written to try and avoid implementation details - +the intent is that it's an overview of the architecture without any specific +implementation decisions.
+These next sections start getting specific, so implementation choices need to +be made, and thus will possibly be more contentious.
+In other words, get your spray-cans ready because there's a bikeshed being built!
+However, let's start small and make some general observations.
+Some apps only care about a subset of the engines - lockbox is one such app +and only cares about a single collection/engine. It might be the case that +lockbox uses a generic application-services library with many engines +available, even though it only wants logins. Thus, the embedding application +is the only thing which knows which engines should be considered to "exist". +It may be that the app layer passes an engine to the sync manager, or the +sync manager knows via some magic how to obtain these handles.
+Some apps will use a combination of Rust components and "legacy" +engines. For example, iOS is moving some of the engines to using Rust +components, while other engines will be ported after delivery of the +sync manager, if they are ported at all. We also plan +to introduce some rust engines into desktop without integrating the +"sync manager"
+The rust components themselves are designed to be consumed as individual +components - the "logins" component doesn't know anything about the +"bookmarks" component.
+There are a couple of gotchyas in the current implementations too - there's an +issue when certain engines don't yet appear in meta/global - see bug 1479929 +for all the details.
+The tl;dr of the above is that each rust component should be capable of +working with different sync managers. That said though, let's not over-engineer +this and pretend we can design a single, canonical thing that will not need +changing as we consider desktop and iOS.
+There's loads of state here. The app itself has some state. The high-level +Sync Manager component will have state, the low-level component will have state, +and each engine has state. Some of this state will need to be persisted (either +on the device or on the storage servers) and some of this state can be considered +ephemeral and lives only as long as the app.
+A key challenge will be defining this state in a coherent way with clear +boundaries between them, in an effort to allow decoupling of the various bits +so Desktop and iOS can fit into this world.
+This state management should also provide the correct degree of isolation for +the various components. For example, each engine should only know about state +which directly impacts how it works. For example, the keys used to encrypt +a collection should only be exposed to that specific engine, and there's no +need for one engine to know what info/collections returns for other engines, +nor whether the device is currently connected to WiFi.
+A thorn here is for persisted state - it would be ideal if the low-level +component could avoid needing to persist any state, so it can avoid any +kind of storage abstraction. We have a couple of ways of managing this:
+The state which needs to be persisted is quite small, so we could delegate +state storage to the high-level component in an opaque way, as this +high-level component almost certainly already has storage requirements, such +as storing the "choose what to sync" preferences.
+The low-level component could add its own storage abstraction. This would +isolate the high-level component from this storage requirement, but would +add complexity to the sync manager - for example, it would need to be passed +a directory where it should create a file or database.
+We'll probably go with the former.
+Let's try and move into actionable decisions for the implementation. We expect +the implementation of the low-level component to happen first, followed very +closely by the implementation of the high-level component for Android. So we +focus first on these.
+The clients engine includes some meta-data about each client. We've decided +we can't replace the clients engine with the FxA device record and we can't +simply drop this engine entirely.
+Of particular interest is "commands" - these involve communicating with the +engine regarding commands targetting it, and accepting commands to be send to +other devices. Note that outgoing commands are likely to not originate from a sync, +but instead from other actions, such as "restore bookmarks".
+However, because the only current requirement for commands is to wipe the +store, and because you could anticipate "wipe" also being used when remotely +disconnecting a device (eg, when a device is lost or stolen), our lives would +probably be made much simpler by initially supporting only per-engine wipe +commands.
+Note that there has been some discussion about not implementing the client +engine and replacing "commands" with some other mechanism. However, we have +decided to not do that because the implementation isn't considered too +difficult, and because desktop will probably require a number of changes to +remove it (eg, "synced tabs" will not work correctly without a client record +with the same guid as the clients engine.)
+Note however that unlike desktop, we will use the FxA device ID as the client +ID. Because FxA device IDs are more ephemeral than sync IDs, it will be +necessary for engines using this ID to track the most-recent ID they synced +with so the old record can be deleted when a change is detected.
+For the purposes of the sync manager, we define:
+An engine is the unit exposed to the user - an "engine" can be enabled +or disabled. There is a single set of canonical "engines" used across the +entire sync ecosystem - ie, desktop and mobile devices all need to agree +about what engines exist and what the identifier for an engine is.
+An Api is the unit exposed to the application layer for general application +functionality. Application services has 'places' and 'logins' Apis and is +the API used by the application to store and fetch items. Each 'Api' may +have one or more 'stores' (although the application layer will generally not +interact directly with a store)
+A store is the code which actually syncs. This is largely an implementation +detail. There may be multiple stores per engine (eg, the "history" engine +may have "history" and "forms" stores) and a single 'Api' may expose multiple +stores (eg, the "places Api" will expose history and bookmarks stores)
+A collection is a unit of storage on a server. It's even more of an +implementation detail than a store. For example, you might imagine a future +where the "history" store uses multiple "collections" to help with containers.
+In practice, this means that the high-level component should only need to care +about an engine (for exposing a choice of what to sync to the user) and an +api (for interacting with the data managed by that api). The low-level +component will manage the mapping of engines to stores.
+This document isn't going to outline the history of how "declined" is used, nor +talk about how this might change in the future. For the purposes of the sync +manager, we have the following hard requirements:
+The low-level component needs to know what the currently declined set of
+engines is for the purposes of re-populating meta/global
.
The low-level component needs to know when the declined set needs to change +based on user input (ie, when the user opts in to or out of a particular +engine on this device)
+The high-level component needs to be aware that the set of declined engines +may change on every sync (ie, when the user opts in to or out of a particular +engine on another device)
+A complication is that due to networks being unreliable, there's an inherent +conflict between "what is the current state?" and "what state changes are +requested?". For example, if the user changes the state of an engine while +there's no network, then exits the app, how do we ensure the user's new state +is updated next time the app starts? What if the user has since made a +different state request on a different device? Is the state as last-known on +this device considered canonical?
+To clarify, consider:
+User on this device declines logins. This device now believes logins is +disabled but history is enabled, but is unable to write this to the server +due to no network.
+The user declines history on a different device, but doesn't change logins. +This device does manage to write the new list to the server.
+This device restarts and the network is up. It believes history is enabled +but logins is not - however, the state on the server is the exact opposite.
+How does this device react?
+(On the plus side, this is an extreme edge-case which none of our existing +implementations handle "correctly" - which is easy to say, because there's +no real definition for "correctly")
+Regardless, the low-level component will not pretend to hide this complexity +(ie, it will ignore it!). The low-level component will allow the app to ask +for state changes as part of a sync, and will return what's on the server at +the end of every sync. The app is then free to build whatever experience +it desires around this.
+The low-level component needs to have the ability to disconnect all engines +from Sync. Engines which are declined should also be reset.
+Because we will need wipe() functionality to implement the clients engine, +and because Lockbox wants to wipe on disconnect, we will provide disconnect +and wipe functionality.
+Breaking the above down into actionable tasks which can be some somewhat +concurrently, we will deliver:
+A straw-man for the API we will expose to the high-level components. This +probably isn't too big, but we should do this as thoroughly as we can. In +particular, ensure we have covered:
+Declined management - how the app changes the declined list and how it learns +of changes from other devices.
+How telemetry gets handed from the low-level to the high-level.
+The "state" - in particular, how the high-level component understands the +auth state is wrong, and whether the service is in a degraded mode (eg, +server requested backoff)
+How the high-level component might specify "special" syncs, such as "just +one engine" or "this is a pre-sleep, quick-as-possible sync", etc
+There's a straw-man proposal for this at the end of the document.
+We should build a utility (or 2) which can stand in for the high-level +component, for testing and demonstration purposes.
+This is something like places-utils.rs and the little utility Grisha has +been using. This utility should act like a real client (ie, it should have +an FxA device record, etc) and it should use the low-level component in +exactly the same we we expect real products to use it.
+Because it is just a consumer of the low-level component, it will force us to +confront some key issues, such as how to get references to engines stored in +multiple crates, how to present a unified "state" for things like auth errors, +etc.
+The initial work for the clients engine can probably be done without too +much regard for how things are tied together - for example, much work could +be done without caring how we get a reference to engines across crates.
+Implementing things needed to we can expose the correct state to the high-level +manager for things like auth errors, backoff semantics, etc
+There will be lots of loose ends to clean up - things like telemetry, etc.
+We have identified that iOS will, at least in the short term, want the +sync manager to be implemented in Swift. This will be responsible for +syncing both the Swift and Rust implemented engines.
+At some point in the future, Desktop may do the same - we will have both +Rust and JS implemented engines which need to be coordinated. We ignore this +requirement for now.
+This approach still has a fairly easy time coordinating with the Rust +implemented engines - the FFI will need to expose the relevant sync +entry-points to be called by Swift, but the Swift code can hard-code the +Rust engines it has and make explicit calls to these entry-points.
+This Swift code will need to create the structures identified below, but this +shouldn't be too much of a burden as it already has the information necessary +to do so (ie, it already has info/collections etc)
+TODO: dig into the Swift code and make sure this is sane.
+While we use rust struct definitions here, it's important to keep in mind that +as mentioned above, we'll need to support the manager being written in +something other than rust, and to support engines written in other than rust.
+The structures below are a straw-man, but hopefully capture all the information +that needs to be passed around.
++ +#![allow(unused)] + +fn main() { +// We want to define a list of "engine IDs" - ie, canonical strings which +// refer to what the user perceives as an "enigine" - but as above, these +// *do not* correspond 1:1 with either "stores" or "collections" (eg, "history" +// refers to 2 stores, and in a future world, might involve 3 collections). +enum Engine { + History, // The "History" and "Forms" stores. + Bookmarks, // The "Bookmark" store. + Passwords, +} + +impl Engine { + fn as_str(&self) -> &'static str { + match self { + History => "history", + // etc + } +} + +// A struct which reflects engine declined states. +struct EngineState { + engine: Engine, + enabled: bool, +} + +// A straw-man for the reasons why a sync is happening. +enum SyncReason { + Scheduled, + User, + PreSleep, + Startup, +} + +// A straw man for the general status. +enum ServiceStatus { + Ok, + // Some general network issue. + NetworkError, + // Some apparent issue with the servers. + ServiceError, + // Some external FxA action needs to be taken. + AuthenticationError, + // We declined to do anything for backoff or rate-limiting reasons. + BackedOff, + // Something else - you need to check the logs for more details. + OtherError, +} + +// Info we need from FxA to sync. This is roughly our Sync15StorageClientInit +// structure with the FxA device ID. +struct AccountInfo { + key_id: String, + access_token: String, + tokenserver_url: Url, + device_id: String, +} + +// Instead of massive param and result lists, we use structures. +// This structure is passed to each and every sync. +struct SyncParams { + // The engines to Sync. None means "sync all" + engines: Option<Vec<Engine>>, + // Why this sync is being done. + reason: SyncReason, + + // Any state changes which should be done as part of this sync. + engine_state_changes: Vec<EngineState>, + + // An opaque state "blob". This should be persisted by the app so it is + // reused next sync. + persisted_state: Option<String>, +} + +struct SyncResult { + // The general health. + service_status: ServiceStatus, + + // The result for each engine. + engine_results: HashMap<Engine, Result<()>>, + + // The list of declined engines, or None if we failed to get that far. + declined_engines: Option<Vec<Engine>>, + + // When we are allowed to sync again. If > now() then there's some kind + // of back-off. Note that it's not strictly necessary for the app to + // enforce this (ie, it can keep asking us to sync, but we might decline). + // But we might not too - eg, we might try a user-initiated sync. + next_sync_allowed_at: Timestamp, + + // New opaque state which should be persisted by the embedding app and supplied + // the next time Sync is called. + persisted_state: String, + + // Telemetry. Nailing this down is tbd. + telemetry: Option<JSONValue>, +} + +struct SyncManager {} + +impl SyncManager { + // Initialize the sync manager with the set of Engines known by this + // application without regard to the enabled/declined states. + // XXX - still TBD is how we will pass "stores" around - it may be that + // this function ends up taking an `impl Store` + fn init(&self, engines: Vec<&str>) -> Result<()>; + + fn sync(&self, params: SyncParams) -> Result<SyncResult>; + + // Interrupt any current syncs. Note that this can be called from a different + // thread. + fn interrupt() -> Result<()>; + + // Disconnect this device from sync. This may "reset" the stores, but will + // not wipe local data. + fn disconnect(&self) -> Result<()>; + + // Wipe all local data for all local stores. This can be done after + // disconnecting. + // There's no exposed way to wipe the remote store - while it's possible + // stores will want to do this, there's no need to expose this to the user. + fn wipe(&self) -> Result<()>; +} +}
This document provides a high-level overview of how syncing works. Note: each component has its own quirks and will handle sync slightly differently than the general process described here.
+sync15
and support/sync15-traits
handle the general syncing logic and define the SyncEngine
traitlogins
, places
, autofill
, etc). These implement SyncEngine
.sync_manager
manages the overall syncing process.SyncManager.sync()
to start the sync process.SyncManager
creates SyncEngine
instances to sync the individual components. Each SyncEngine
corresponds to a collection
on the sync server.SyncManager
is responsible for performing the high-level parts of the sync process:
sync()
function to start the sync, passing
+in a SyncParams
object in, which describes what should be synced.SyncManager
performs all network operations on behalf of the individual engines. It's also responsible for tracking the general authentication state (primarily by inspecting the responses from these network requests) and fetching tokens from the token server.SyncManager
checks if we are currently in a backoff period and should wait before contacting the server again.SyncManager
instantiates a SyncEngine
for each enabled component. We currently use 2 different methods for this:
+SyncManager
to hold a weakref to a Store
use that to create the SyncEngine
(tabs and places). The SyncEngine
uses the Store
for database access, see the TabsStore
for an example.SyncEngine
, hiding the details of how that engine gets created (autofill/logins). These components also define a Store
instance for the SyncEngine
to use, but it's all transparent to the SyncManager
. (See autofill::get_registered_sync_engine()
and autofill::db::store::Store
)SyncManager
passes the local encryption key to their SyncEngine
sync_multiple()
function from the sync15
crate, sending it the SyncEngine
instances. sync_multiple()
then calls the sync()
function for each individual SyncEngine
SyncEngine
is defined in the support/sync15-traits
crate and defines the interface for syncing a component.SyncEngine
instance is created for each syncSyncEngine.apply_incoming()
does the main work. It is responsible for processing incoming records from the server in order to update the local records and calculating which local records should be synced back.apply_incoming
patternSyncEngine
instances are free to implement apply_incoming()
any way they want, but the most components follow a general pattern.
guid
as its primary key. A record will share the same guid
for the local/mirror/staging table.apply_incoming
stagesOption
s for the local/mirror/staging records.sync15
code so that it can upload them to the server.The local table has an integer column syncChangeCounter which is incremented every time the embedding app makes a change to a local record (eg, updating a field). Thus, any local record with a non-zero change counter will need to be updated on the server (with either the local record being used, or after it being merged if the record also changed remotely). At the start of the sync, when we are determining what action to take, we take a copy of the change counter, typically in a temp staging table. After we have uploaded the record to the server, we decrement the counter by whatever it was when the sync started. This means that if a record is changed in between staging the record and uploading it, the change counter will not drop to zero, and so it will correctly be seen as locally modified on the next sync
+ +We'd like to keep cargo test
, cargo build
, cargo check
, ... reasonably
+fast, and we'd really like to keep them fast if you pass -p
for a specific
+project. Unfortunately, there are a few ways this can become unexpectedly slow.
+The easiest of these problems for us to combat at the moment is the unfortunate
+placement of dev-dependencies in our build graph.
If you perform a cargo test -p foo
, all dev-dependencies of foo
must be
+compiled before foo
's tests can start. This includes dependencies only used
+non-test targets, such as examples or benchmarks.
In an ideal world, cargo could run your tests as soon as it finished with the +dependencies it needs for those tests, instead of waiting for your benchmark +suite, or the arg-parser your examples use, or etc.
+Unfortunately, all cargo knows is that these are dev-dependencies
, and not
+which targets actually use them.
Additionally, unqualified invocations of cargo (that is, without -p
) might
+have an even worse time if we aren't careful. If I run, cargo test
, cargo
+knows every crate in the workspace needs to be built with all dev
+dependencies, if places
depends on fxa-client
, all of fxa-clients
+dev-dependencies must be compiled, ready, and linked in at least to the lib
+target before we can even think about starting on places
.
We have not been careful about what shape the dependency graph ends up as when example code is +taken into consideration (as it is by cargo during certain builds), and as a +result, we have this problem. Which isn't really a problem we +want to fix: Example code can and should depend on several different components, +and use them together in interesting ways.
+So, because we don't want to change what our examples do, or make +major architectural changes of the non-test code for something like this, we +need to do something else.
+To fix this, we manually insert "cuts" into the dependency graph to help cargo +out. That is, we pull some of these build targets (e.g. examples, benchmarks, +tests if they cause a substantial compile overhead) into their own dedicated +crates so that:
+Some rules of thumb for when / when not to do this:
+All rust examples should be put in examples/*
.
All rust benchmarks should be put in testing/separated/*
. See the section
+below on how to set your benchmark up to avoid redundant compiles.
Rust tests which brings in heavyweight dependencies should be evaluated on an +ad-hoc basis. If you're concerned, measure how long compilation takes +with/without, and consider how many crates depend on the crate where the test +lives (e.g. a slow test in support/foo might be far worse than one in a leaf +crate), etc...
+To be clear, this is way more important for benchmarks (which always compile as +release and have a costly link phase).
+Say you have a directory structure like the following:
+mycrate
+ ├── src
+ │ └── lib.rs
+ | ...
+ ├── benches
+ │ ├── bench0.rs
+ | ├── bench1.rs
+ │ └── bench2.rs
+ ├── tests
+ │ ├── test0.rs
+ | ├── test1.rs
+ │ └── test2.rs
+ └── ...
+
+When you run your integration tests or benchmarks, each of test0
, test1
,
+test2
or bench0
, bench1
, bench2
is compiled as it's own crate that runs
+the tests in question and exits.
That means 3 benchmark executables are built on release settings, and 3 +integration test executables.
+If you've ever tried to add a piece of shared utility code into your integration
+tests, only to have cargo (falsely) complain that it is dead code: this is why.
+Even if test0.rs
and test2.rs
both use the utility function, unless
+every test crate uses every shared utility, the crate that doesn't will
+complain.
(Aside: This turns out to be an unintentional secondary benefit of this approach
+-- easier shared code among tests, without having to put a
+#![allow(dead_code)]
in your utils.rs. We haven't hit that very much here,
+since we tend to stick to unit tests, but it came up in mentat several times,
+and is a frequent complaint people have)
Anyway, the solution here is simple: Create a new crate. If you were working in
+components/mycrate
and you want to add some integration tests or benchmarks,
+you should do cargo new --lib testing/separated/mycrate-test
(or
+.../mycrate-bench
).
Delete .../mycrate-test/src/lib.rs
. Yep, really, we're making a crate that
+only has integration tests/benchmarks (See the "FAQ0" section at the bottom of
+the file if you're getting incredulous).
Now, add a src/tests.rs
or a src/benches.rs
. This file should contain mod foo;
declarations for each submodule containing tests/benchmarks, if any.
For benches, this is also where you set up the benchmark harness (refer to +benchmark library docs for how).
+Now, for a test, add: into your Cargo.toml
+[[test]]
+name = "mycrate-test"
+path = "src/tests.rs"
+
+and for a benchmark, add:
+[[test]]
+name = "mycrate-benches"
+path = "src/benches.rs"
+harness = false
+
+Because we aren't using src/lib.rs
, this is what declares which file is the
+root of the test/benchmark crate. Because there's only one target (unlike with
+tests/*
/ benches/*
under default settings), this will compile more quickly.
Additionally, src/tests.rs
and src/benches.rs
will behave like a normal
+crate, the only difference being that they don't produce a lib, and that they're
+triggered by cargo test
/cargo run
respectively.
src/*
instead of disabling autotests
/autobenches
Instead of putting tests/benchmarks inside src
, we could just delete the src
+dir outright, and place everything in tests
/benches
.
Then, to get the same one-rebuild-per-file behavior that we'll get in src
, we
+need to add autotests = false
or autobenches = false
to our Cargo.toml,
+adding a root tests/tests.rs
(or benches/benches.rs
) containing mod
decls
+for all submodules, and finally by referencing that "root" in the Cargo.toml
+[[tests]]
/ [[benches]]
list, exactly the same way we did for using src/*
.
This would work, and on the surface, using tests/*.rs
and benches/*.rs
seems
+more consistent, so it seems weird to use src/*.rs
for these files.
My reasoning is as follows: Almost universally, tests/*.rs
, examples/*.rs
,
+benches/*.rs
, etc. are automatic. If you add a test into the tests folder, it
+will run without anything else.
If we're going to set up one-build-per-{test,bench}suite as I described, this
+fundamentally cannot be true. In this paradigm, if you add a test file named
+blah.rs
, you must add a mod blah
it to the parent module.
It seems both confusing and error-prone to use tests/*
, but have it behave
+that way, however this is absolutely the normal behavior for files in src/*.rs
+-- When you add a file, you then need to add it to it's parent module, and this
+is something Rust programmers are pretty used to.
(In fact, we even replicated this behavior (for no reason) in the places
+integration tests, and added the mod
declarations to a "controlling" parent
+module -- It seems weird to be in an environment where this isn't required)
So, that's why. This way, we make it way less likely that you add a test file
+to some directory, and have it get ignored because you didn't realize that in
+this one folder, you need to add a mod mytest
into a neighboring tests.rs.
Each component in the Application Services repository has three parts (the Rust code, +the Kotlin wrapper, and the Swift wrapper) so there are quite a few moving +parts involved in adding a new component. This is a rapid-fire list of all +the things you'll need to do if adding a new component from scratch.
+Your component should live under ./components
in this repo.
+Use cargo new --lib ./components/<your_crate_name>
to create a new library crate,
+and please try to avoid using hyphens in the crate name.
See the Guide to Building a Rust Component for general +advice on designing and structuring the actual Rust code, and follow the +Dependency Management Guidelines if your crate +introduces any new dependencies.
+Use UniFFI to define how your crate's
+API will get exposed to foreign-language bindings. By convention, put the interface
+definition file at ./components/<your_crate_name>/<your_crate_name>.udl
. Use
+the builtin-bindgen
feature of UniFFI to simplify the build process, by
+putting the following in your Cargo.toml
:
[build-dependencies]
+uniffi_build = { version = "<latest version here>", features=["builtin-bindgen"] }
+
+Include your new crate in the application-services
workspace, by adding
+it to the members
and default-members
lists in the Cargo.toml
at
+the root of the repository.
In order to be published to consumers, your crate must be included in the +"megazord" crate for each target platform:
+./megazords/full/Cargo.toml
and
+add a pub use <your_crate_name>
to ./megazords/full/src/lib.rs
../megazords/ios-rust/rust/Cargo.toml
and
+add a pub use <your_crate_name>
to ./megazords/ios-rust/src/lib.rs
.Run cargo check -p <your_crate_name>
in the repository root to confirm that
+things are configured properly. This will also have the side-effect of updating
+Cargo.lock
to contain your new crate and its dependencies.
Make a ./components/<your_crate_name>/android
subdirectory to contain
+Kotlin- and Android-specific code. This directory will contain a gradle
+project for building your Kotlin bindings.
Copy the build.gradle
file from ./components/crashtest/android/
into
+your own component's directory, and edit it to replace the references to
+crashtest.udl
with your own component's .udl
file.
Create a file ./components/<your_crate_name>/uniffi.toml
with the
+following contents:
[bindings.kotlin]
+package_name = "mozilla.appservices.<your_crate_name>"
+cdylib_name = "megazord"
+
+Create a file ./components/<your_crate_name>/android/src/main/AndroidManifest.xml
+with the following contents:
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+ package="org.mozilla.appservices.<your_crate_name>" />
+
+In the root of the repository, edit .buildconfig-android.yml
to add
+your component's metadata. This will cause it to be included in the
+gradle workspace and in our build and publish pipeline. Check whether
+it builds correctly by running:
./gradlew <your_crate_name>:assembleDebug
You can include hand-written Kotlin code alongside the automatically +generated bindings, by placing `.kt`` files in a directory named:
+./android/src/test/java/mozilla/appservices/<your_crate_name>/
You can write Kotlin-level tests that consume your component's API, +by placing `.kt`` files in a directory named:
+./android/src/test/java/mozilla/appservices/<your_crate_name>/
.So you would end up with a directory structure something like this:
+components/<your_crate_name>/
+Cargo.toml
uniffi.toml
src/
+android/
+build.gradle
src/
+main/
+AndroidManifest.xml
java/mozilla/appservices/<your_crate_name>/
+test/java/mozilla/appservices/<your_crate_name>/
+Run your component's Kotlin tests with ./gradlew <your_crate_name>:test
+to confirm that this is all working correctly.
Make a ./components/<your_crate_name>/ios
subdirectory to contain
+Swift- and iOS-specific code. The UniFFI-generated swift bindings will
+be written to a subdirectory named Generated
.
You can include hand-written Swift code alongside the automatically
+generated bindings, by placing .swift
files in a directory named:
+./ios/<your_crate_name>/
.
So you would end up with a directory structure something like this:
+components/<your_crate_name>/
+Cargo.toml
uniffi.toml
src/
+ios/
+<your_crate_name>/
+Generated/
+++For more information on our how we ship components using the Swift Package Manager, check the ADR that introduced the Swift Package Manager
+
You will need to do the following steps to include the component in the megazord:
+Update its uniffi.toml
to include the following settings:
[bindings.swift]
+ffi_module_name = "MozillaRustComponents"
+ffi_module_filename = "<crate_name>FFI"
+
+Add the component as a dependency to the Cargo.toml
in megazords/ios-rust/
Add a pub use
declaration for the component in megazords/ios-rust/src/lib.rs
Add logic to the megazords/ios-rust/build-xcframework.sh
to copy or generate its header file into the build
Add an #import
for its header file to megazords/ios-rust/MozillaRustComponents.h
Add your component into the iOS "megazord" through the Xcode project, which can only really by done using the Xcode application, which can only really be done if you're on a Mac.
+Open megazords/ios-rust/MozillaTestServices/MozillaTestServices.xcodeproj
in Xcode.
In the Project navigator, add a new Group for your new component, pointing to
+the ./ios/
directory you created above. Add the following entries to the Group:
.udl
file for you component, from ../src/<your_crate_name>.udl
..swift
files for your component++Make sure that the "Copy items if needed" option is unchecked, and that +nothing is checked in the "Add to targets" list.
+
The result should look something like this:
+ +Click on the top-level "MozillaTestServices" project in the navigator, then go to "Build Phases".
+++Double-check that
+<your_crate_name>.udl
does not appear in the "Copy Bundle Resources" section.
Add <your_crate_name>.udl
to the list of "Compile Sources". This will trigger an Xcode Build Rule that generates
+the Swift bindings automatically. Also include any hand-written .swift
files in this list.
Finally, in the Project navigator, add a sub-group named "Generated", pointing to the ./Generated/
subdirectory, and
+containing entries for the files generated by UniFFI:
+* <your_crate_name>.swift
+* <your_crate_name>FFI.h
+Make sure that "Copy items if needed" is unchecked, and that nothing is checked in "Add to targets".
++Double-check that
+<your_crate_name>.swift
does not appear in the "Compile Sources" section.
The result should look something like this:
+ +Build the project in Xcode to check whether that all worked correctly.
+To add Swift tests for your component API, create them in a file under
+megazords/ios-rust/MozillaTestServicesTests/
. Use this syntax to import
+your component's bindings from the compiled megazord:
@testable import MozillaTestServices
+
+In Xcode, navigate to the MozillaTestServicesTests
Group and add your
+new test file as an entry. Select the corresponding target, click on
+"Build Phases", and add your test file to the list of "Compile Sources".
+The result should look something like this:
Use the Xcode Test Navigator to run your tests and check whether +they're passing.
+rust-components-swift
The Swift source code and generated UniFFI bindings are distributed to consumers (eg: Firefox iOS) through rust-components-swift
.
A nightly taskcluster job prepares the rust-component-swift
packages from the source code in the application-services repository. To distribute your component with rust-component-swift
, add the following to the taskcluster script in taskcluster/scripts/build-and-test-swift.py
:
<your_crate_name>.udl
file to BINDINGS_UDL_PATHS
+FOCUS_UDL_PATHS
if your component is also targeting Firefox FocusSOURCE_TO_COPY
+FOCUS_SOURCE_TO_COPY
if your component is also targeting Firefox FocusYour component should now automatically get included in the next rust-component-swift
nightly release.