Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data review: FxA error tracking metrics #5872

Closed
bendk opened this issue Oct 16, 2023 · 2 comments
Closed

Data review: FxA error tracking metrics #5872

bendk opened this issue Oct 16, 2023 · 2 comments
Assignees
@eliserichards

Comments

@bendk
Copy link
Contributor

bendk commented Oct 16, 2023 •
edited
Loading

  1. What questions will you answer with this data?

How often do operations with our fxa-client fail and in what ways?

  1. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

These metrics will allow us to measure changes in the quality of the fxa-client. For example, the first use of these metrics will be to monitor the effect of the moving the FxA code from firefox-android into application-services:

  1. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We also plan to use manual testing. However these metrics will be better for detecting low-frequency bugs.

  1. Can current instrumentation answer these questions?

No.

  1. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.
Measurement Description Data collection category Tracking bug
FxA operation count (counter) Category 2 "Interaction data" #5871
FxA error count (labeled counter, with these categories: network, authorization, fxa_other, unexpected) Category 2 "Interaction data" #5871
  1. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

This collection is documented in the Glean Dictionary at https://dictionary.telemetry.mozilla.org/

  1. How long will this data be collected? Choose one of the following:

I want to permanently monitor this data. (Ben Dean-Kawamura)

  1. What populations will you measure?

All release channels, countries, and locales.

  1. If this data collection is default on, what is the opt-out mechanism for users?

Standard Firefox telemetry controls

  1. Please provide a general description of how you will analyze this data.
  • When deploying changes to FxA, we will monitor changes in the telemetry.
  • As part of triage, we will check to see that the error rates are roughly stable.
  1. Where do you intend to share the results of your analysis?

The sync team

  1. Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

No

┆Issue is synchronized with this Jira Task
┆Epic: Moving fxa-client functionality out of android-components
@eliserichards
Copy link

  1. What questions will you answer with this data?

How often do operations with our fxa-client fail and in what ways?

  1. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

These metrics will allow us to measure changes in the quality of the fxa-client. For example, the first use of these metrics will be to monitor the effect of the moving the FxA code from firefox-android into application-services:

* [New fxa client [firefox-android: bendk/new-fxa-client] #5851](https://github.com/mozilla/application-services/pull/5851)

* [Bug 1856729 - Switch to new app-services FxaClient mozilla-mobile/firefox-android#3918](https://github.com/mozilla-mobile/firefox-android/pull/3918)
  1. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We also plan to use manual testing. However these metrics will be better for detecting low-frequency bugs.

  1. Can current instrumentation answer these questions?

No.

  1. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Description Data collection category Tracking bug
FxA operation count (counter) Category 2 "Interaction data" #5871
FxA error count (labeled counter, with these categories: network, authorization, fxa_other, unexpected) Category 2 "Interaction data" #5871

  1. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

This collection is documented in the Glean Dictionary at https://dictionary.telemetry.mozilla.org/

  1. How long will this data be collected? Choose one of the following:

I want to permanently monitor this data. (Ben Dean-Kawamura)

  1. What populations will you measure?

All release channels, countries, and locales.

  1. If this data collection is default on, what is the opt-out mechanism for users?

Standard Firefox telemetry controls

  1. Please provide a general description of how you will analyze this data.
* When deploying changes to FxA, we will monitor changes in the telemetry.

* As part of triage, we will check to see that the error rates are roughly stable.
  1. Where do you intend to share the results of your analysis?

The sync team

  1. Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

No

┆Issue is synchronized with this Jira Task ┆Epic: Moving fxa-client functionality out of android-components

Data Review Form (to be filled by Data Stewards)

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
  1. Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.
  • Yes, through the general "Send Usage Data" preference in the applications' settings.
  1. If the request is for permanent data collection, is there someone who will monitor the data over time?
  • Yes, this data will be permanently collected. Ben Dean-Kawamura ([email protected]) will be monitoring.
  1. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
  • Category 2 - interaction data.
  1. Is the data collection request for default-on or default-off?
  • Default on.
  1. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
  • No.
  1. Is the data collection covered by the existing Firefox privacy notice? If unsure: escalate to legal if:
  • Yes.
  1. Does the data collection use a third-party collection tool? If yes, escalate to legal.
  • No.

Result

data-review+

@bendk
Copy link
Contributor Author

bendk commented Oct 18, 2023

Thanks for the review! I just put the metrics PR on the merge queue.

@bendk bendk closed this as completed Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eliserichards eliserichards

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bendk @eliserichards