feat: add IPNI EntryChunk encoding #18
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Provides an EntryChunk class to help encode batches of multihashes as IPNI EntryChunk IPLD blocks.
```js
import { EntryChunk } from '@web3-storage/ipni'
import { sha256 } from 'multiformats/hashes/sha2'
const hash = await sha256.encode(new Uint8Array())
const chunk = EntryChunk.fromMultihashes([hash])
const block = await chunk.export()
// the EntryChunk CID should be passed to an Advertisement as the `entries` Link.
console.log(`entries cid ${block.cid}`)
```
Encourages using dag-cbor as it's almost half the size as dag-json, and every entrychunk and advert has to be stored and replicated to 1 or more IPNI servers.
Borrows the idea of an `export()` that provides an encoded IPLD block from @Gozala in ucanto https://github.com/web3-storage/ucanto/blob/2200d43595b85a5e7b60c234987ff3ce91404401/packages/core/src/delegation.js#L248
Provides a cheap and accurate `calculateEncodedSize()` function to allow callers to determine when to split entries across multiple blocks.
Thanks to @rvagg for the very useful https://github.com/rvagg/cborg module that makes that possible and the inspriation in the calculateHeaderLength fn from js-car
see: https://github.com/ipld/js-car/blob/562c39266edda8422e471b7f83eadc8b7362ea0c/src/buffer-writer.js#L215
License: MIT
Signed-off-by: Oli Evans <[email protected]>
alanshaw
requested changes
Dec 14, 2023
License: MIT Signed-off-by: Oli Evans <[email protected]>
alanshaw
approved these changes
Dec 14, 2023
License: MIT Signed-off-by: Oli Evans <[email protected]>
License: MIT Signed-off-by: Oli Evans <[email protected]>
olizilla
pushed a commit
that referenced
this pull request
Dec 15, 2023
🤖 I have created a release *beep* *boop* --- ## [3.1.0](v3.0.0...v3.1.0) (2023-12-15) ### Features * add IPNI EntryChunk encoding ([#18](#18)) ([d4e44c3](d4e44c3)) ### Bug Fixes * test that CID parameters are compatible CID types ([#17](#17)) ([b0ca303](b0ca303)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Provides an EntryChunk class to help encode batches of multihashes as IPNI EntryChunk IPLD blocks.
Encourages using dag-cbor as it's almost half the size as dag-json, and every entrychunk and advert has to be stored and replicated to 1 or more IPNI servers.
Borrows the idea of an
export()that provides an encoded IPLD block from @Gozala in ucanto https://github.com/web3-storage/ucanto/blob/2200d43595b85a5e7b60c234987ff3ce91404401/packages/core/src/delegation.js#L248Provides a cheap and accurate
calculateEncodedSize()function to allow callers to determine when to split entries across multiple blocks.Thanks to @rvagg for the very useful https://github.com/rvagg/cborg module that makes that possible and the inspriation in the calculateHeaderLength fn from js-car
see: https://github.com/ipld/js-car/blob/562c39266edda8422e471b7f83eadc8b7362ea0c/src/buffer-writer.js#L215
Fixes #2
WIP on storacha/specs#85
License: MIT