Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cli: README: move to public-docs #1180

Merged
merged 1 commit into from
Jul 19, 2024
Merged

cli: README: move to public-docs #1180

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
271 changes: 1 addition & 270 deletions cli/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,273 +12,4 @@ If you run into problems with the `aws-lc-rs` dependency, you can try building w

Or you can download a binary for your OS and host architecture from our [releases](https://github.com/divviup/divviup-api/releases).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep this line (it's the second clause after the "either" above).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


### Divvi Up quickstart

We'll walk through a simple exercise in which we will:

- Run Divvi Up and a pair of aggregators locally in Docker
- Shard some measurements into reports and upload them, simulating DAP clients
- Collect the results of an aggregation

To get started, you'll need:

- A Unix like shell
- A `divviup` binary (see instructions above)
- A `compose.yaml` (you can use the one in this repository or the one you find alongside the `divviup` binaries on our [releases](https://github.com/divviup/divviup-api/releases) page)
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).

First, get Divvi Up running locally. In your shell, navigate to the directory containing `compose.yaml` and run:

```sh
docker compose up --wait
```

This will:

- deploy two Janus aggregators to act as leader and helper in your tasks
- deploy the Divvi Up control plane to coordinate the aggregators
- pair the aggregators with the control plane to make them available for use in task definitions
- create an account named "demo"

Set an environment variable to direct `divviup` to use your local control plane and to use an empty
API token:

```sh
export DIVVIUP_API_URL=http://localhost:8080
export DIVVIUP_TOKEN=""
```

Next, obtain the ID of the demo account:

```sh
divviup account list
```

You should get a single account back with the name "demo", like this:

```json
[
{
"id": "8ea1fc65-a669-48dd-96ac-c920fcb0ae90",
"name": "demo",
"created_at": "2024-06-14T22:48:00.683659Z",
"updated_at": "2024-06-14T22:48:00.683659Z",
"admin": false
}
]
```

Now, set an environment variable for the account ID:

```sh
# Substitute the account ID that was generated by your Divvi Up instance
export DIVVIUP_ACCOUNT_ID=8ea1fc65-a669-48dd-96ac-c920fcb0ae90
```

List the aggregators and identify the ID of both a leader and a helper.

```sh
divviup aggregator list
```

The output will contain JSON objects like:

```json
{
"id": "3650870b-56e6-4eac-8944-b7ca36569b33",
"role": "Either",
"name": "Divvi Up staging-dap-09-1",
"dap_url": "https://staging-dap-09-1.api.example.com/",
"api_url": "https://staging-dap-09-1.api.example.com/aggregator-api",
"is_first_party": true,
"vdafs": [
"Prio3Count",
"Prio3Sum",
"Prio3Histogram",
"Prio3SumVec"
],
"query_types": [
"TimeInterval",
"FixedSize"
],
"protocol": "DAP-09",
"features": [
"TokenHash",
"TimeBucketedFixedSize",
"UploadMetrics"
]
},
{
"id": "96301951-c848-4a57-b4f5-32812e4db1be",
"account_id": null,
"created_at": "2024-03-21T22:47:15.467139Z",
"updated_at": "2024-04-18T17:33:30.439465Z",
"deleted_at": null,
"role": "Either",
"name": "Divvi Up staging-dap-09-2",
"dap_url": "https://staging-dap-09-2.api.example.com/",
"api_url": "https://staging-dap-09-2.api.example.com/aggregator-api",
"is_first_party": false,
"vdafs": [
"Prio3Count",
"Prio3Sum",
"Prio3Histogram",
"Prio3SumVec"
],
"query_types": [
"TimeInterval",
"FixedSize"
],
"protocol": "DAP-09",
"features": [
"TokenHash",
"UploadMetrics",
"TimeBucketedFixedSize"
]
}
```

Set the two IDs into environment variables. NOTE: These IDs will vary based on your configuration.

```sh
# Substitute the leader and helper aggregator IDs that were generated by your Divvi Up instance
export LEADER_ID=3650870b-56e6-4eac-8944-b7ca36569b33
export HELPER_ID=96301951-c848-4a57-b4f5-32812e4db1be
```

Next, generate a collector-credential for the task. The collector credential will be used by the collector to export the aggregated statistics.

```sh
divviup collector-credential generate --save
```

The output will be like:

```sh
{
"id": "0a0f8ea8-b603-4416-b138-b7f217153bb7",
"hpke_config": {
"id": 144,
"kem_id": "X25519HkdfSha256",
"kdf_id": "HkdfSha256",
"aead_id": "Aes128Gcm",
"public_key": "V9IpdJxS91MHPiNTjwDk9DFS-5M_neVrPxlmvolmTTo"
},
"created_at": "2024-05-03T15:23:56.624726Z",
"deleted_at": null,
"updated_at": "2024-05-03T15:23:56.624727Z",
"name": "collector-credential-144",
"token_hash": "VItYJdAyWYIvooe8GzkGnVTvaMkvWc9G-eiwxudfWww",
"token": "A6JDAYPiYDXmNyh-OpYXGw"
}

Saved new collector credential to /your/current/directory/collector-credential-144.json. Keep this file safe!
```

**Note**: This credential isn't sensitive because it's only useful in your local instance of Divvi Up. But collector credentials generated against the production service should be carefully managed. Consider using a password manager to protect them.

Make a note of the path where the credential was saved, and save the collector credential ID to an environment variable.

```sh
# Substitute the collector credential path and ID that were generated by your Divvi Up instance
export COLLECTOR_CREDENTIAL_PATH=/your/current/directory/collector-credential-144.json
export COLLECTOR_CREDENTIAL_ID=0a0f8ea8-b603-4416-b138-b7f217153bb7
```

Create the the histogram task. In this case the task is a set of values from 0 to 10 for use in collecting a net-promoter score for a survey.

```sh
divviup task create --name net-promoter-score \
--leader-aggregator-id $LEADER_ID --helper-aggregator-id $HELPER_ID \
--collector-credential-id $COLLECTOR_CREDENTIAL_ID \
--vdaf histogram --categorical-buckets 0,1,2,3,4,5,6,7,8,9,10 \
--min-batch-size 100 --max-batch-size 200 --time-precision 60sec
```

The output will contain a JSON object:

```json
{
"id": "Siwa4QTEnQXMfRPyhir8AzS4EBqfTebmEzKfvajDgYk",
"account_id": "a9c571ba-5f3d-4814-8d8b-c5bb0f5030b7",
"name": "net-promoter-score",
"vdaf": {
"type": "histogram",
"buckets": [
"0",
"1",
"2",
"3",
"4",
"5",
"6",
"7",
"8",
"9",
"10"
],
"chunk_length": 4
},
"min_batch_size": 100,
"max_batch_size": null,
"created_at": "2024-05-03T15:27:56.229891Z",
"updated_at": "2024-05-03T15:27:56.229891Z",
"time_precision_seconds": 60,
"report_count": 0,
"aggregate_collection_count": 0,
"expiration": "2025-05-03T15:27:55.88511Z",
"leader_aggregator_id": "3650870b-56e6-4eac-8944-b7ca36569b33",
"helper_aggregator_id": "96301951-c848-4a57-b4f5-32812e4db1be",
"collector_credential_id": "0a0f8ea8-b603-4416-b138-b7f217153bb7",
"report_counter_interval_collected": 0,
"report_counter_decode_failure": 0,
"report_counter_decrypt_failure": 0,
"report_counter_expired": 0,
"report_counter_outdated_key": 0,
"report_counter_success": 0,
"report_counter_too_early": 0,
"report_counter_task_expired": 0
}
```

Save the ID of the task into an environment variable.

```sh
# Substitute the task ID that was generated by your Divvi Up instance
export TASK_ID=Siwa4QTEnQXMfRPyhir8AzS4EBqfTebmEzKfvajDgYk
```

Upload a random set of 150 metrics.

```sh
for i in {1..150}; do
measurement=$(( $RANDOM % 10 ))
divviup dap-client upload --task-id $TASK_ID --measurement $measurement;
done
```

Wait a little while to let the aggregators run the aggregation jobs. Then, get collection results:

```sh
divviup dap-client collect \
--task-id $TASK_ID \
--collector-credential-file $COLLECTOR_CREDENTIAL_PATH \
--current-batch
```

You will get a result like:

```sh
Number of reports: 113
Interval start: 2024-06-05 21:31:00 UTC
Interval end: 2024-06-05 21:34:00 UTC
Interval length: 180s
Aggregation result: [14, 10, 10, 13, 13, 8, 16, 13, 9, 7, 0]
collection: Collection { partial_batch_selector: PartialBatchSelector { batch_identifier: BatchId(wPLBlC6iHWp_YBBAYP_ig5nal0FOz1QlLSaC42U7sm0) }, report_count: 113, interval: (2024-06-05T21:31:00Z, TimeDelta { secs: 180, nanos: 0 }), aggregate_result: [14, 10, 10, 13, 13, 8, 16, 13, 9, 7, 0] }
```

If you get an error like "The batch implied by the query is invalid", then the aggregators are still working on processing your uploaded reports. Given them a few more minutes.

If you get fewer reports in the collection than you uploaded, that's because you sent the collection request too soon and not all the reports were prepared yet. Those reports will be available in a later collection, after enough additional reports are uploaded to meet the minimum batch size.

Congratulations! You've just used secure multi-party computation to do a privacy-preserving aggregation over secret shares of measurements. Now you can try experimenting with different tasks. Try creating different histograms with different numbers of buckets, or see if you can use `Prio3SumVec` to compute sums over bit vectors.
A [complete tutorial for the divviup tool](https://docs.divviup.org/command-line-tutorial/) is available.
Loading