remove rust-ipfs #132

chunningham · 2022-11-01T13:32:08Z

The rust-ipfs project has been archived in favour of a new rust implementation. Simultaneously we desire to decouple the design from the orthodox IPFS architecture for several reasons:

p2p communication should be authorised via the peer capabilities model
the need for a limit on block size is ameliorated by use of the graphsync protocol and the different assumptions of the orbit p2p model
different transport and protocol requirements and concerns
maintenance burden of our fork of rust-ipfs

This PR removes the rust-ipfs dependancy while retaining the libp2p and libipld dependancy. In the process it:

removes everything imported from rust-ipfs
[REGRESSION] removes any p2p capability from Kepler. Until a further pr, Kepler nodes will not be able to communicate with each other
adds the ImmutableStore and StoreConfig traits, implemented for the local filesystem, AWS S3 and storage::either::Either
adds appropriate generics to Orbit and other structs
moves config objects to their appropriate modules
removes the limit on block sizes
removes the chunking object writer and chunked list graph from the KVStore, in favour of singular unbounded-size content blocks
attempts to make the configuration and initialisation/opening of stores and Orbits more ergonomic

krhoda

Code changes look great, I am curious about how some of the organization changed (i.e. using 1 CID for all content), that said, the only feedback is there are plenty of commented-out code that could just be removed (so no big deal if no action is taken).

I was hoping to test it locally if possible before approving. Is there any easy way to do so? Like a demo front-end app or anything of that sort? At the very least, I wanted to test with cargo test.

When I initially tried running cargo test on the branch, but it failed at the build step:

Updating git repository `https://github.com/spruceid/ssi`
error: failed to get `did-ethr` as a dependency of package `kepler-lib v0.1.0 (/Users/kelseyrhoda/magic/spruce/kepler/lib)`

Caused by:
  failed to load source for dependency `did-ethr`

Caused by:
  Unable to update https://github.com/spruceid/ssi?branch=feat/ucan-ipld-version#e5cf96b7

Caused by:
  object not found - no match for id (e5cf96b7efd899a080b8f2cc763fc39a434ce536); class=Odb (9); code=NotFound (-3)

Any idea why, or how I could test / build it locally? The build seems more important, I could understand if the test suites aren't ready or don't prove all that much. It isn't a big deal if there isn't an easy way to test locally, I can definitely approve, then open issues if I have issues later integrating with SSX Quest, but I am curious why my build is failing.

I'm happy to approve once the build error is understood (if not solved).

krhoda · 2023-01-11T16:25:49Z

lib/Cargo.toml

-did-webkey = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "main" }
-did-onion = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "main" }
-did-ion = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "main" }
+did-method-key = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "feat/ucan-ipld-version" }


Ideally these would point to a crates.io release, but hey, Rebase doesn't either.

currently did-ion v0.2 isn't yet released, that's the only blocker for using crates.io, but I can remove the branch ref

That sounds like a good idea for now given our current use-case

did-ion is blocked by an ownership issue. Hopefully it will get resolved soon.

src/capabilities/store.rs

src/kv/entries.rs

src/kv/store.rs

src/relay.rs

src/transport.rs

sbihel · 2023-01-11T16:10:44Z

.github/workflows/rust.yml

@@ -23,6 +23,9 @@ jobs:
      run: |
        rustup target add wasm32-unknown-unknown

+    - name: Install Protoc
+      uses: arduino/setup-protoc@v1


Do we need protobuf? Probably not, then you can use something like

libipld = { version = "0.14", default-features = false, features = ["dag-cbor", "dag-json", "derive", "serde-codec"]}

to avoid needing it for compilation

CI still fails after removing it, looks like this is waiting on this pr

Ah, well then do make sure the docker image can still build

sbihel · 2023-01-11T16:12:32Z

lib/Cargo.toml

-did-webkey = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "main" }
-did-onion = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "main" }
-did-ion = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "main" }
+did-method-key = { default-features = false, git = "https://github.com/spruceid/ssi", branch = "feat/ucan-ipld-version" }


You can use the published versions now

sbihel · 2023-01-12T09:11:33Z

src/lib.rs

+};
+
+pub type Block = OBlock<DefaultParams>;
+pub type BlockStores = Either<S3BlockStore, FileSystemStore>;


I don't understand how this is better than an enum with all the variants of storage options

Also, why use generics for ImmutableStore when in the end (I assume) it's always the Either enum?

it is yeah usually gonna be that enum, I figured it would make it easier to implement alternatives if we/someone else wanted to, and easier to change the generic in Orbit if someone wants to deploy or embed it in a different context. e.g. if someone wanted to run a node in wasm for the browser using browser storage they could implement HypotheticalBrowserSessionStore and just change the generic for that environment to that and avoid potential complications of building for wasm with dependancies like s3 (although I couldn't concretely name any such complications). in short: probably overzealous abstraction

chunningham added 30 commits October 20, 2022 17:49

add ImmutableStore trait and fs based impl

a316905

use multihashes for content IDs (ignore codec)

eeace02

remove ipfs trait impls

5de83ab

remove Ipfs from kv, remove ObjectReader for single-block contents

a3be0b8

remove ipfs from cap store and service

65e4a7d

remove ipfs from orbit, remove more ipfs fns

4f4ad2f

remove various ipfs crate imports

bf44fa4

fix dependancies

f212870

remove SupportedCodecs, unused

b308413

fixes, stub kv behaviour

62f48cb

impl EitherStore to combine stores, make Orbit generic

cfa1fea

minor core fixes

22e16f2

hollow out relay.rs

2a18ad9

remove src/ipfs.rs

53bcce7

fix siwe and ssi versions

6c644fc

fix up storage trait, stub out impls

0459cf7

add clone and stderror for stores

9607007

bump aws and tokio-util versions

e2e90dd

make everything to do with ImmutableStore Send + Sync

874b629

fix up the KV mod to use ImmutableStore

57f9f3d

fix caps and auth to use immutablestore

2b2b576

warnings

c5f6382

use ImmutableStore in Orbit

36eefbe

fix up routes/responses to use AsyncRead

a764327

store minor fix

75d43dd

add StorageConfig trait, move config structs to respective modules

5cece4f

rename eitherstore -> either

1a194a8

add StorageConfig::create, Orbit::create and Orbit::open

3fa79c2

add OrbitPeerConfig and builder

b22942f

add ProviderUtils and refactor orbit load/create

e7b9125

chunningham added 2 commits November 3, 2022 13:52

fix protoc in clippy workflow

52fe2cb

fix tests

4c10cbb

chunningham marked this pull request as ready for review November 3, 2022 14:04

chunningham requested review from sbihel and cobward and removed request for sbihel November 3, 2022 14:25

chunningham added 3 commits November 4, 2022 14:09

update cached

0d57503

add hash code choice to store write, add to all usage

d26f359

code golf with macros

125abd8

chunningham force-pushed the feat/remove-ipfs branch from a29c6a8 to 125abd8 Compare November 4, 2022 17:14

chunningham added 2 commits November 4, 2022 18:18

remove unnecessary extra macro case

46c109f

remove unnecessary AsyncReadEither type, use future::Either

549ca13

chunningham mentioned this pull request Nov 23, 2022

reimplement relay peers #133

Closed

implement ImmutableStore::write_keyed

ca00323

obstropolos requested review from krhoda and lilyannehall January 11, 2023 16:06

krhoda reviewed Jan 11, 2023

View reviewed changes

sbihel approved these changes Jan 12, 2023

View reviewed changes

chunningham added 5 commits January 18, 2023 13:19

address pr feedback

d44cbb4

remove more commented out sections

10bf35e

cargo update

172d57b

satisfy clippy

057fb42

remove protoc step from workflow

1ca47cb

chunningham force-pushed the feat/remove-ipfs branch from 50a92f1 to 1ca47cb Compare January 18, 2023 13:57

re-add protoc to workflows

57812ef

krhoda approved these changes Jan 18, 2023

View reviewed changes

use released ssi and did method crates, remove did-ion

6762edc

chunningham merged commit c74c18c into main Jan 19, 2023

chunningham deleted the feat/remove-ipfs branch January 19, 2023 09:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove rust-ipfs #132

remove rust-ipfs #132

chunningham commented Nov 1, 2022 •

edited

Loading

krhoda left a comment

krhoda Jan 11, 2023

chunningham Jan 13, 2023

obstropolos Jan 13, 2023

sbihel Jan 18, 2023

sbihel Jan 11, 2023

chunningham Jan 18, 2023

sbihel Jan 18, 2023

sbihel Jan 11, 2023

sbihel Jan 12, 2023

sbihel Jan 12, 2023

chunningham Jan 13, 2023

remove rust-ipfs #132

remove rust-ipfs #132

Conversation

chunningham commented Nov 1, 2022 • edited Loading

krhoda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chunningham commented Nov 1, 2022 •

edited

Loading