Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: align aggregation namings from commp learnings #62

Merged
merged 9 commits into from
Jun 22, 2023
Merged

fix: align aggregation namings from commp learnings #62

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
97df283
fix: align aggregation namings from commp and commd
vasco-santos Jun 7, 2023
dcef229
fix: swap proof with link
vasco-santos Jun 7, 2023
99cbfa4
chore: address first review comments
vasco-santos Jun 9, 2023
060181a
chore: iterate by moving away from segment naming
vasco-santos Jun 14, 2023
e8e8c14
chore: drop src and use single schema definition
vasco-santos Jun 15, 2023
903894e
fix: lint issue
vasco-santos Jun 15, 2023
df468f1
chore: drop car cid from invocation block
vasco-santos Jun 20, 2023
9bc01c0
chore: apply suggestions from code review
vasco-santos Jun 22, 2023
c54e73d
fix: remove unused type
vasco-santos Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 59 additions & 76 deletions w3-aggregation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,31 +117,7 @@ In this document, we will be looking at `spade-proxy.web3.storage` as an impleme

### `aggregate/offer`

A Storefront principal can invoke a capabilty to offer an aggregate that is ready to be included in Filecoin deal(s).

```iplsch
type AggregateOffer struct {
with StorefrontDID
nb AggregateOfferDetail
}

type AggregateOfferDetail struct {
offer OfferCBOR
commitmentProof Proof
}

type OfferCBOR any
type Proof any
Gozala marked this conversation as resolved.
Show resolved Hide resolved

type struct OfferDetails {
size Int
link Link
src [URL]
commitmentProof Proof
}

type Offer [OfferDetails]
```
A Storefront principal can invoke a capabilty to offer an aggregate that is ready to be included in Filecoin deal(s). See [schema](#aggregateoffer-schema).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give that filecoin spec refers to these actors as Aggregators it may be better idea to use the same term as them as opposed to Storefront, which I suggested before I was familiar with the preexisting terminology.


> `did:web:web3.storage` invokes capability from `did:web:spade.storage`

Expand All @@ -153,8 +129,11 @@ type Offer [OfferDetails]
"with": "did:web:web3.storage",
"can": "aggregate/offer",
"nb": {
"offer": { "/": "bafy...many-cars" }, /* dag-cbor CID */
"commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */
"offer": { "/": "bafy...many-cars" }, /* dag-cbor CID with offer content */
"piece": {
"link": { "/": "commitment...aggregate-proof" },
"size": 10102020203013342343
} /* commitment proof for aggregate */
}
Comment on lines 131 to 137
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given all the fields that we've dropped I wonder if we should just flatten things up and have this more like

Suggested change
"nb": {
"offer": { "/": "bafy...many-cars" }, /* dag-cbor CID */
"commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */
"offer": { "/": "bafy...many-cars" }, /* dag-cbor CID with offer content */
"piece": {
"link": { "/": "commitment...aggregate-proof" },
"size": 10102020203013342343
} /* commitment proof for aggregate */
}
"nb": {
"offer": { "/": "bafy...offer" }, /* embedded with invocation */
}

Where bafy...offer is CBOR of structure like:

{
   "link": { "/": "offer...commp" }
   "size": 1010101
   "pieces": [
       {
           "link": { "/": "car...commp" },
           "size": 1010
       },
       // ....
   ]
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hesitant to voice this opinion, yet I wonder if we should omit the size which I believe could be determinstically derived from the pieces. Unless I'm mistaken existing libraries require to specify size (on aggregates specifically) because the remaining space could be filled with 0s. However it creates space for an error where you may have two different aggregates for exact same pieces.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem here is that this is not directly compatible with UCAN LOG. When we want to create metrics or to run consumers based on invocations/receipts of these invocations we would need to do extra ops to read block from CAR file. We could make the block as you suggest, which would probably be the exact content we send to Spade in the end, but I would still keep link and size in the nb field so that we can easily track things from the log without a layer of indirection

}],
"prf": [],
Expand All @@ -165,21 +144,19 @@ type Offer [OfferDetails]
Invoking `aggregate/offer` capability submits an aggregate to a broker service for inclusion in one or more Filecoin deals. The `nb.offer` field represents a "Ferry" aggregate offer that is ready for a Filecoin deal. Its value is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. This block MUST be included in the CAR file that transports the invocation. Its format is:

```json
/* offers block as OfferDetails type, encoded as DAG-JSON (for readability) */
/* offers block as PieceInfo type, encoded as DAG-JSON (for readability) */
[
{
"link": { "/": "bag...file0" }, /* CAR CID */
"link": { "/": "commitment...car0" }, /* COMMP CID */
"size": 110101,
"commitmentProof": { "/": "commitment...car0" }, /* COMMP CID */
"src": ["https://w3s.link/ipfs/bag...file0"]
},
{
/* ... */
}
]
```

Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `link` field has the CAR CID, while the `commitmentProof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `size` field MUST be set to the byte size of the CAR file. The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. Note that `src` field is optional and can be provided in a different part of the flow such as when deal is signed or through a previously agreed API.
Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. It includes an array of Filecoin `piece` info required by Storage Providers. Out of band, Storefront will provide to Storage Providers a `src` HTTP URL to each CAR file in the offer.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I think "Out of band, ...." is somewhat misleading here. Perhaps document should say HTTP URLs will be issued during deal signing.

Aside: I have also been thinking that we may want to delegate UCAN that can be used for fetching aggregate pieces to the actor we sign with a deal with. That would prevent other actors from using that URL without sharing a private key or a UCAN delegation. It could also allow us to defer the pre-signed URL creation up until actor decides to fetch.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also wondering if aggreate/offer should be delegating capability to a "broker" to issue signed URLs for the aggregate pieces. That would allow broker to do it as needed and perhaps avoid back & forth. I realize we would still need to sign the deal with our key, but if delegated signatures were an option we could probably even get away from that.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have started a thread on this line of thought here https://filecoinproject.slack.com/archives/C05C7CUPEKX/p1687366781115129

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the discussion in the linked thread. I think we should reframe "spade-proxy" as an "agency" to which "aggregators" can submit (offer) aggregates along with all the UCAN delegations that would allow it to:

  1. Create HTTP read URLs for the aggregate pieces.
  2. Sign deals with "storage providers" on behalf of the aggregator.

That would I think simplify things for both "aggregator" and "agency" (spade-proxy) reducing coordination between them as "agency" would be able to take the aggregate and either get it into a filecoin.

@vasco-santos I think this also could remove for mapping CAR CID <-> Piece CID, instead pieces could come with associated UCANs, allow agency to create HTTP URLs for them. Aggregator would still need to map CARs to pieces to surface corresponding filecoin state, which I think makes sense, it's like a reference to filecoin state.


Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with the aggregate).

Expand All @@ -204,7 +181,7 @@ See [`offer/arrange`](#offerarrange) section to see the subsequent task.

### `aggregate/get`

A Storefront principal can invoke a capability to get state of a previously accepted offer.
A Storefront principal can query state of accepted aggregate by invoking `aggregate/get` capability. See [schema](#aggregateget-schema).

> `did:web:web3.storage` invokes capability from `did:web:spade.storage`

Expand All @@ -216,7 +193,7 @@ A Storefront principal can invoke a capability to get state of a previously acce
"with": "did:web:web3.storage",
"can": "aggregate/get",
"nb": {
"commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */
"subject": { "/": "commitment...aggregate-proof" } /* commitment proof */
}
}],
"prf": [],
Expand Down Expand Up @@ -257,7 +234,7 @@ Once this invocation is executed, a receipt is generated with the resulting aggr

### `offer/arrange`

When a broker receives an `aggregate/offer` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with join task to be performed asynchronously.
When a broker receives an `aggregate/offer` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with join task to be performed asynchronously. See [schema](#offerarrange-schema).

```json
{
Expand All @@ -267,7 +244,7 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip
"with": "did:web:spade.storage",
"can": "offer/arrange",
"nb": {
"commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */
"pieceLink": { "/": "commitment...aggregate-proof" } /* commitment proof */
}
}],
"prf": [],
Expand All @@ -282,7 +259,7 @@ Once this invocation is executed, a receipt is generated with the result of the
"ran": "bafy...arrange",
"out": {
"ok": {
"commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */
"pieceLink": { "/": "commitment...aggregate-proof" } /* commitment proof */
}
},
"fx": {
Expand All @@ -294,16 +271,16 @@ Once this invocation is executed, a receipt is generated with the result of the
}
```

If offered aggregate is invalid, details on failing commitmentProofs are also reported:
If offered aggregate is invalid, details on failing pieces are also reported:

```json
{
"ran": "bafy...invocation",
"out": {
"error": {
"commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */
"pieceLink": { "/": "commitment...aggregate-proof" }, /* commitment proof */
"cause": [{
"commitmentProof": { "/": "commitment...car0" },
"piece": { "/": "commitment...car0" },
"reason": "reasonCode",
}],
},
Expand All @@ -317,73 +294,79 @@ If offered aggregate is invalid, details on failing commitmentProofs are also re
}
```

### Schema
## Schema

```ipldsch
type Aggregate union {
| Link "queued"
| Link "status"
| Link "rejected"
} representation keyed
### Base types

```ipldsch
type AggregateCapability enum {
AggregateOffer "aggregate/offer"
AggregateGet "aggregate/get"
} representation inline {
discriminantKey "can"
}

type AggregateGet struct {
with StorefrontDID
nb SucceedAggregateRef
type OfferCapability union {
OfferArrange "offer/arrange"
} representation inline {
discriminantKey "can"
}

type SucceedAggregateRef struct {
commitmentProof Proof
type AggregateRef struct {
pieceLink Link
}

type AggregateRef struct {
commitmentProof Proof
type SubjectRef struct {
subject Link
}

type StorefrontDID string
type URL string

type BrokerDID string
```

### `aggregate/offer` schema

```ipldsch
type AggregateOffer struct {
with StorefrontDID
nb AggregateOfferDetail
}

type AggregateOfferDetail struct {
offer OfferCBOR
commitmentProof Proof
# Contains each individual piece within Aggregate piece
offer &Offer
# Piece as Aggregate of CARs with padding
piece PieceInfo
}

type OfferCBOR any
type Proof any
type Offer [PieceInfo]

type struct OfferDetails {
size Int
link Link
src? [URL]
commitmentProof Proof
# https://github.com/filecoin-project/go-state-types/blob/1e6cf0d47cdda75383ef036fc2725d1cf51dbde8/abi/piece.go#L47-L50
type PieceInfo {
# Size in nodes. For BLS12-381 (capacity 254 bits), must be >= 16. (16 * 8 = 128)
size Int
link Link
}
```

type Offer [OfferDetails]

type StorefrontDID string
type URL string
type OfferCBOR any
type Proof any
### `aggregate/get` schema

type OfferCapability union {
OfferArrange "offer/arrange"
} representation inline {
discriminantKey "can"
```ipldsch
type AggregateGet struct {
with StorefrontDID
nb SubjectRef
}
```

### `offer/arrange` schema

```ipldsch
type OfferArrange struct {
with BrokerDID nb AggregateRef
with BrokerDID
nb AggregateRef
}

type BrokerDID string
```

[`did:web`]: https://w3c-ccg.github.io/did-method-web/
Expand Down