feat: generate sharded DAG index on client and invoke w `index/add` #1451

alanshaw · 2024-05-14T15:58:10Z

This PR generates index data for blocks as CAR shards are constructed. Once all shards have been successfully sent, a sharded DAG index is encoded and stored (using blob/add) and then index/add is invoked with the CID of the index as a parameter.

alanshaw · 2024-05-14T15:59:34Z

packages/upload-client/src/car.js

@@ -7,7 +7,7 @@ import varint from 'varint'
 */

 /** Byte length of a CBOR encoded CAR header with zero roots. */
-const NO_ROOTS_HEADER_LENGTH = 17
+const NO_ROOTS_HEADER_LENGTH = 18


🤦 FML this was wrong. Either it was wrong the whole time or @ipld/car started encoding the header differently.

nit: would be great to just export it from @ipld/car

alanshaw · 2024-05-14T16:00:32Z

packages/upload-client/test/helpers/random.js

@@ -19,7 +19,7 @@ export async function randomBytes(size) {
    } else {
      crypto.getRandomValues(chunk)
    }
-    size -= bytes.length
+    size -= chunk.length


FML we were creating random data in tests where the last 65,536 bytes were random but the rest was ZEROs.

alanshaw · 2024-05-14T16:02:14Z

packages/upload-client/src/sharding.js

+          for (const slice of slices.values()) {
+            slice[0] += diff
+          }
+          controller.enqueue(await encodeCAR(blocks, slices, rootCID))


Putting a root CID in the last CAR shard is a massive PITA. I wish we didn't do that.

Wait, I thought we were putting it in the first shard.

Let's put it as a separate thing that is not in the set of shards. It would be easier to change that now.

Yes I would like all shards to be rootless and track the DAG root in upload/add and/or content claims.

gammazero

See comments. I am not sure about the placement of the DAG block. Maybe better to change the index to keep it separate.

Review of test.index.js was not done as I still need to understand how that code works.

gammazero · 2024-05-15T03:27:09Z

packages/upload-client/src/index.js

+
+          // add the CAR shard itself to the slices
+          meta.slices.set(meta.cid.multihash, [0, meta.size])
+          shardIndexes.push(meta.slices)


Check with ShardedDAGIndex, but I thought this was supposed to be the first slice.

They are sorted when encoded anyway so it doesn't matter where you put it?

gammazero · 2024-05-15T03:31:32Z

packages/upload-client/test/dag-index.test.js

+      { message: 'failed index/add invocation' }
+    )
+  })
+})


Will any of these tests fail if the DAG block is in the wrong place or missing? If no, then that would be good. Or, change the index to put the DAGblock in a different place than the others.

No, but there probably should be a test that asserts the additional slice that spans the whole blob is added...

vasco-santos · 2024-05-15T05:47:56Z

packages/upload-client/src/car.js

@@ -7,7 +7,7 @@ import varint from 'varint'
 */

 /** Byte length of a CBOR encoded CAR header with zero roots. */
-const NO_ROOTS_HEADER_LENGTH = 17
+const NO_ROOTS_HEADER_LENGTH = 18


nit: would be great to just export it from @ipld/car

packages/upload-client/src/index.js

…m:web3-storage/w3up into feat/generate-sharded-dag-index-in-client

vasco-santos

LGTM

some suggestions for follow ups in review, + filecoin/offer things I believe MUST be added

vasco-santos · 2024-05-15T09:39:21Z

packages/upload-client/src/index.js

@@ -183,5 +195,24 @@ async function uploadBlockStream(
  if (!root) throw new Error('missing root CID')

  await Upload.add(conf, root, shards, options)
+
+  const index = ShardedDAGIndex.create(root)


nit: should we have an option to not index? I mean, let's do this by default, but we wanted this to be optional, perhaps just an issue where user can tweak their indexing intentions would be good enough for now

vasco-santos · 2024-05-15T09:40:48Z

packages/upload-client/src/index.js

+  }
+
+  // Store the index in the space
+  const indexDigest = await Blob.add(conf, indexBytes.ok, options)


nit: this is great to store in the space, but we will need to look into deleting this on remove from space to not have its usage. Maybe we can just fill in issue now?

packages/upload-client/README.md

packages/upload-client/src/index.js

packages/w3up-client/README.md

packages/upload-client/src/index.js

Co-authored-by: Vasco Santos <[email protected]>

…m:web3-storage/w3up into feat/generate-sharded-dag-index-in-client

🤖 I have created a release *beep* *boop* --- ## [15.0.0](upload-api-v14.0.0...upload-api-v15.0.0) (2024-05-15) ### ⚠ BREAKING CHANGES * delegated capabilities required to use `uploadFile`, `uploadDirectory` and `uploadCAR` have changed. In order to use these methods your agent will now need to be delegated `blob/add`, `index/add`, `filecoin/offer` and `upload/add` capabilities. Note: no code changes are required. ### Features * generate sharded DAG index on client and invoke w `index/add` ([#1451](#1451)) ([a6d9026](a6d9026)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [19.0.0](access-v18.4.0...access-v19.0.0) (2024-05-15) ### ⚠ BREAKING CHANGES * delegated capabilities required to use `uploadFile`, `uploadDirectory` and `uploadCAR` have changed. In order to use these methods your agent will now need to be delegated `blob/add`, `index/add`, `filecoin/offer` and `upload/add` capabilities. Note: no code changes are required. ### Features * generate sharded DAG index on client and invoke w `index/add` ([#1451](#1451)) ([a6d9026](a6d9026)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [15.0.0](upload-client-v14.1.1...upload-client-v15.0.0) (2024-05-15) ### ⚠ BREAKING CHANGES * delegated capabilities required to use `uploadFile`, `uploadDirectory` and `uploadCAR` have changed. In order to use these methods your agent will now need to be delegated `blob/add`, `index/add`, `filecoin/offer` and `upload/add` capabilities. Note: no code changes are required. ### Features * generate sharded DAG index on client and invoke w `index/add` ([#1451](#1451)) ([a6d9026](a6d9026)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Alan Shaw <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [14.0.0](w3up-client-v13.1.1...w3up-client-v14.0.0) (2024-06-04) ### ⚠ BREAKING CHANGES * **upload-api:** integrate agent store for idempotence & invocation/receipt persistence ([#1444](#1444)) * delegated capabilities required to use `uploadFile`, `uploadDirectory` and `uploadCAR` have changed. In order to use these methods your agent will now need to be delegated `blob/add`, `index/add`, `filecoin/offer` and `upload/add` capabilities. Note: no code changes are required. ### Features * generate sharded DAG index on client and invoke w `index/add` ([#1451](#1451)) ([a6d9026](a6d9026)) * **upload-api:** integrate agent store for idempotence & invocation/receipt persistence ([#1444](#1444)) ([c9bf33e](c9bf33e)) ### Fixes * check for blob/accept receipts before blob/add is concluded ([#1459](#1459)) ([462518c](462518c)) * export blob client ([#1485](#1485)) ([7944077](7944077)) * rename blob and index client capabilities ([#1478](#1478)) ([17e3a31](17e3a31)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Vasco Santos <[email protected]>

alanshaw requested review from Gozala, joaosa, vasco-santos and gammazero May 14, 2024 15:58

alanshaw commented May 14, 2024

View reviewed changes

Alan Shaw added 2 commits May 14, 2024 17:13

feat: generate sharded DAG index on client and invoke w index/add

10e4dc3

fix: w3up-client dependency

6d691de

alanshaw force-pushed the feat/generate-sharded-dag-index-in-client branch from f8fc6d5 to 6d691de Compare May 14, 2024 16:13

Alan Shaw added 3 commits May 14, 2024 17:33

fix: use blob-index in upload-client

3844083

fix: more dependency fix

888ab77

chore: fml lint

46c924e

gammazero reviewed May 15, 2024

View reviewed changes

vasco-santos reviewed May 15, 2024

View reviewed changes

packages/upload-client/src/index.js Outdated Show resolved Hide resolved

Alan Shaw added 5 commits May 15, 2024 14:26

feat: integrate in w3up-client

8cc4017

Merge branch 'main' into feat/generate-sharded-dag-index-in-client

f78abb6

fix: exports

948959d

Merge branch 'feat/generate-sharded-dag-index-in-client' of github.co…

047cd3b

…m:web3-storage/w3up into feat/generate-sharded-dag-index-in-client

chore: appease linter

e272d97

alanshaw requested review from vasco-santos and gammazero May 15, 2024 13:51

vasco-santos approved these changes May 15, 2024

View reviewed changes

alanshaw commented May 15, 2024

View reviewed changes

packages/upload-client/src/index.js Outdated Show resolved Hide resolved

Alan Shaw and others added 5 commits May 15, 2024 15:41

refactor: apply suggestions from code review

7f8ec15

Co-authored-by: Vasco Santos <[email protected]>

fix: add filecoin/offer delegation to invocation config where needed

44dbaa4

refactor: apply suggestions from code review

5b9d184

Co-authored-by: Vasco Santos <[email protected]>

chore: appease linter

750124e

Merge branch 'feat/generate-sharded-dag-index-in-client' of github.co…

efb8dfa

…m:web3-storage/w3up into feat/generate-sharded-dag-index-in-client

alanshaw merged commit a6d9026 into main May 15, 2024
15 checks passed

alanshaw deleted the feat/generate-sharded-dag-index-in-client branch May 15, 2024 15:11

This was referenced May 15, 2024

chore(main): release upload-api 15.0.0 #1463

Merged

chore(main): release upload-client 15.0.0 #1465

Merged

chore(main): release w3up-client 14.0.0 #1464

Merged

chore(main): release access 19.0.0 #1462

Merged

This was referenced May 20, 2024

Support IPNI+Carpark indexing from the client storacha/project-tracking#24

Closed

[parent] ipni/offer protocol storacha/project-tracking#54

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: generate sharded DAG index on client and invoke w `index/add` #1451

feat: generate sharded DAG index on client and invoke w `index/add` #1451

alanshaw commented May 14, 2024

alanshaw May 14, 2024

vasco-santos May 15, 2024

alanshaw May 14, 2024

alanshaw May 14, 2024 •

edited

Loading

gammazero May 14, 2024

alanshaw May 15, 2024

gammazero left a comment

gammazero May 15, 2024

alanshaw May 15, 2024

gammazero May 15, 2024

alanshaw May 15, 2024

vasco-santos May 15, 2024

vasco-santos left a comment

vasco-santos May 15, 2024

alanshaw May 15, 2024

vasco-santos May 15, 2024

alanshaw May 15, 2024

feat: generate sharded DAG index on client and invoke w index/add #1451

feat: generate sharded DAG index on client and invoke w index/add #1451

Conversation

alanshaw commented May 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw May 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gammazero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: generate sharded DAG index on client and invoke w `index/add` #1451

feat: generate sharded DAG index on client and invoke w `index/add` #1451

alanshaw May 14, 2024 •

edited

Loading