Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

⚡️ v0.27.0 RELEASE 🚀 #1095

Closed
24 tasks done
daviddias opened this issue Nov 20, 2017 · 41 comments
Closed
24 tasks done

⚡️ v0.27.0 RELEASE 🚀 #1095

daviddias opened this issue Nov 20, 2017 · 41 comments
Assignees

Comments

@daviddias
Copy link
Member

daviddias commented Nov 20, 2017

🚀 This release brings a new transport to the libp2p family, perf improvements and stability (add files with 10x the size!), Windows support, bug fixes and more!
🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟

🗺 What's left to release

🔦 Highlights

🗂 The Files API has been upgraded with a new Streaming + Buffered interface

Now you can add and fetch Files from IPFS using your favorite streams library, Readable Streams (aka Node.js Streams) or Pull Stream, you pick! Also, if you don't need to handle large files or if you are ok with buffering the whole thing, you can also use the Callback or Promises API for simplicity.

See function signatures, descriptions and examples at: https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md

Ref:

🌟 New transport! websocket-star

js-ipfs has a new WebSockets relayed transport, a similar transport to webrtc-star but more stable (less memory hungry). With this addition, we disable webrtc by default and added instructions on how to enabled it to our FAQ so that you can pick the best transport (webrtc-star vs websocket-star) for your needs.

Big thank you to @mkg20001 for creating websocket-star in the first place! :)

📦 You are now able to add any directory size and depth to IPFS with the new multipart stream builder

js-ipfs-api got a big update with ipfs-inactive/js-ipfs-http-client#629, upgrading the way we constructed multipart streams to send large amounts of Files to an IPFS daemon. Thanks to this, now js-ipfs CLI can any any directory size to IPFS.

Thanks to @pgte for shipping it in a flash!

🏡 Windows Support

Yeeaaas! js-ipfs now supports Windows -- #1017 (comment) --!! Please send a huge hi5 to @richardschneider for slaying all the dragons on this one.

🦁 Safari Support (both Mobile and Desktop)

This is not necessarily a 0.27 update, but it is something you can do now! What this means is that now there are two mobile browsers that can run your Distributed Applications! (Chrome on Android and Safari on iOS)

🚥 More interop tests, including PubSub

Out set of interoperability tests is growing to make sure that JS and Go IPFS stay compatible at all times. The biggest new addition were PubSub tests, see these and more at https://github.com/ipfs/js-ipfs/tree/master/test/interop

👢 Bootstraper tests

Quick bootstraper tests were added -- #899 -- so that our Infrastructure team can always check that js-ipfs is able to contact the Bootstraper and Gateways after a configuration change. This should mitigate downtime issues we observer in the past.

ipfs.ls is now implemented!

This was a long time coming. Thank you to @pgte for shipping this one.

💫 ipfs file ls is now implemented too!

You know when they say that an implementation of S3 is bug by bug compatible? Well, in our case, js-ipfs is getting command by command compatible with go-ipfs so that any app uses go-ipfs as a daemon can use js-ipfs too. This one was shipped by @richardschneider, thank you!

📶 Progress bar!

You can now get progress reports while adding a file to IPFS, both through the CLI (progress bar) or through the API, see how here: https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md#add

Thank you @dryajov for shipping this!

🎢 Performance and Memory Improvements with pull-block

@Beanow improved the performance of pull-block, one of the js-ipfs-unixfs-engine dependencies, reducing vastly its memory fingerprint. What this translates too is to a faster and less memory hungry ipfs.add. I've successfully managed to added multiple files of 750Mb to a browser application using js-ipfs, this is more than a 10x improvement over previous reports!!

✅ Release Checklist

  • Robustness and quality
    • Ensure that all tests are passing, this includes:
      • unit
      • interop
      • sharness
    • Run tests of the following projects with the new release:
      • ipfs-pubsub-room
      • peerpad
  • Documentation
    • Ensure that README.md is up to date
    • Ensure that all the examples run
  • Communication
    • Create the release issue
    • Announcements (both pre-release and post-release)
      • Twitter
      • IRC
      • Reddit
    • Blog post

🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.

@daviddias daviddias self-assigned this Nov 20, 2017
@haadcode
Copy link
Member

@diasdavid, I ran tests from master against fresh version of ipfs-log and soon-to-be-released new version or OrbitDB and the tests are running all fine! 👍 🎉 There are a bunch of replication and pubsub tests as well as file and object API tests, so it looks solid. There's couple of API breaking changes that I found that I think should be highlighted:

  • ipfs.cat() --> ipfs.catReadableStream() (when previously used with readable-stream)
  • pubsub message topicCIDs --> topicIDs

Feel free to use the above PRs for testing the before final release and let me know if you need any help.

@pgte
Copy link
Contributor

pgte commented Nov 21, 2017

@diasdavid ran the tests for ipfs-pubsub-room, and released version v0.6 of it that supports the new pubsub message format. All tests pass.

Updated y-ipfs-connector (released version 1.6.0 of that) and tested peerpad with it. Also, all looking good! 👍

@daviddias
Copy link
Member Author

@pgte @haadcode mind testing again with latest master?

@daviddias
Copy link
Member Author

Question: Shall we disable WebRTC by default?

I'm leaning towards yes. Seems that we are not yet ready to use WebRTC without causing confusion to the users (see: #1089). #1088 is one of our top priorities and I would be more comfortable enabling WebRTC-Star or WebSockets-Star once we make more progress.

This doesn't mean that WebRTC won't work, you just need to add the multiaddr manually so that it gets activated.

Thoughts?

@Beanow
Copy link

Beanow commented Nov 25, 2017

Question: Shall we disable WebRTC by default?

Yes for me as well.

Regardless of how you use WebRTC there's always the risk of using one too many resources and crashing. Though the primary reasons to be using it is having out-of-the-box peer discovery. Which you either want to go "true P2P" the easy way, or to have pubsub.

For both I think we should add clear configuration examples. As I would expect that to top the FAQs when WebRTC is not available by default.

@interfect
Copy link

Without WebRTC enabled by default, how will browser nodes be able to communicate with each other? Will WebSockets allow that?

Will turning on WebRTC manually just bring back the browser crashes?

@daviddias
Copy link
Member Author

@interfect, it will prevent the crashes and avoid causing confusion to users that do not require to connect to every browser node.

The proper solution for the problem will be #1088

@daviddias
Copy link
Member Author

Update: PR here #1111

@daviddias
Copy link
Member Author

I'm working towards libp2p/js-libp2p#122 to give another option for Browser connectivity which in theory and from tests run, will be way more stable than the current WebRTC situation.

@daviddias
Copy link
Member Author

I believe that this PR -- #1111 -- will make a lot of people happy :)

@Beanow
Copy link

Beanow commented Nov 27, 2017

@diasdavid yay! 🎉

As a sidenote this may be a good moment to review how we should warn users about the unsafe rendezvous. As this is the first time it should work out-of-the-box. The WebRTC crashes should have been a big enough deterrent for most production use until now.

Ideally it would be clear during development and runtime, but at the same time allow for upgrading to requiring safe variants without config changes later (provided they didn't manually configure an unsafe rendezvous server).

@daviddias
Copy link
Member Author

@Beanow I realized today that we can actually have their pseudo safe version :) (I wouldn't call anything safe or secure until properly audited). See https://github.com/ipfs/js-ipfs/blob/master/src/core/runtime/libp2p-browser.js#L15

@Beanow
Copy link

Beanow commented Nov 27, 2017

Ah that's right. It's only if you want to add the module yourself it becomes an issue.
Perhaps then we should add something like: new WebSocketStar({allowUnsafe: true}) as an explicit requirement before attempting to connect without ID challenge. For those that wish to pass the module.

@daviddias
Copy link
Member Author

@Beanow The way it is built right now is that the crypto puzzle validator is the rendezvous point, if someone wants to "pass in the module" they will have to use another rendezvous point.

@Beanow
Copy link

Beanow commented Nov 27, 2017

@diasdavid which would be OK if there is a distinction in addresses between the two modes and the client can validate the expected behavior of the rendezvous. Like http vs https.

In other words I'm suggesting WebSocketStar should be the https example and downgrading security properties should be explicit. Similar to how you would need to allow self-signed certs explicitly in an https client.

@daviddias
Copy link
Member Author

Understood and agreed, @Beanow, although I believe it is not clear what the "unsafe" wording being used in these rendezvous points means.

To clarify, libp2p and therefore IPFS as well, will always validate that they are trying to dial to the right node through the SECIO handshake.

That the crypto puzzle that the rendezvous does is to enable that point to also be able to check that it is being dialed by the right now, avoiding someone to pollute the routing table.

This is a known issue and our proposal to fix goes in two stages:

  • a) enable the security guard that @mkg20001 implemented in both webrtc-star and websocket-star (already done).
  • b) Upgrade libp2p module loading structure so that transports can use other components already loaded (as per initial design but then it didn't get into the implementation) so that instead of having to use rendezvous points, they can dial to any IPFS node and make any IPFS node a rendezvous point. I've discussed this proposal in length with multiple people and now @pgte is taking the lead for it, track here dependency injection during boot time libp2p/js-libp2p#130

@mkg20001
Copy link
Contributor

implemented in both webrtc-star and websocket-star

@diasdavid webrtc-star only has type checking to prevent server crashes. crypto challenge isn't added.

@daviddias
Copy link
Member Author

@mkg20001 opened a issue to track that libp2p/js-libp2p-webrtc-star#128 :)

@Beanow
Copy link

Beanow commented Nov 27, 2017

@diasdavid true, safe / unsafe is pretty polarizing for a subject like security in general, we'd need an unambiguous term for the flag.

While DI and SECIO are great, those depend on the transport client, it doesn't make directly instantiating the transport client more explicit in it's behavior.

Though in spite of SECIO, being able to trivially proxy traffic still lets you collect metadata and gives you handshakes and cyphertext to store, replay or attempt any other kind of attack on. A situation that proved pretty detrimental for WEP and WPA.

@daviddias
Copy link
Member Author

@Beanow absolutely correct :) Just so you know, we are indeed working towards getting IPFS audited. Both known and unknown issues will get a priority tag so that we can work on patching them.

@mkg20001
Copy link
Contributor

A good solution would be to add an "allowJoinWithDisabledChallenge" (let's make the name long and annoying to type so nobody uses it :D ) parameter and throw listening errors if a user tries to listen on an insecure server and the flag is false.

@daviddias
Copy link
Member Author

Let's make sure to continue those discussions, especially in the issues for js-libp2p and the transports.

In another news, just went through the items on the Release checklist:

  • docs are updated
  • tests are passing
  • pubsub-room and peerpad work with new js-ipfs and also with the new transport!

It is time to hit the drums and announce that the release is ready and invite everyone to try it out before publishing to npm! 🚀

@Beanow
Copy link

Beanow commented Nov 27, 2017

@diasdavid hate to be nitting over this buuuut. I like @mkg20001's solution, but it's the very definition of an API breaking change.

Which means: libp2p-websocket-star goes from ^0.5.1 to ^0.6.0 in the direct dependencies and doesn't automatically end up in this release. Which is why I'm bringing it up now and on this issue.

@daviddias
Copy link
Member Author

daviddias commented Nov 27, 2017

@Beanow I'm confused by your statement. libp2p-websocket-star latest is v0.5.1, it has the crypto puzzle and there is nothing in the pipeline for that one now. What were you trying to convey?

Update: If you were trying to say that adding the Crypto Puzzle to webrtc-star will be a breaking change, I do not necessarily agree or disagree because it depends on the migration path we set up for users. We could:

  • Start by allowing both to work (as it is now) and in a future release of js-ipfs disabling the handshake without the Crypto Puzzle. This way we mitigate the effects and leave space for smooth migration.
  • Just ship it right now and encourage all users to upgrade. This would result in delaying this release or just shipping another version once it is done.

I'm more inclined to the smooth transition path.

mkg20001 added a commit to libp2p/js-libp2p-websocket-star that referenced this issue Nov 27, 2017
@mkg20001
Copy link
Contributor

@diasdavid @Beanow Added the requested changes to libp2p-websocket-star

Should I ship it as 0.5.2 or 0.6.0?

@Beanow
Copy link

Beanow commented Nov 27, 2017

@diasdavid adding a constructor argument such as allowJoinWithDisabledChallenge is an API change. And if we release [email protected] with ^0.5.1 for the dependency, it won't satisfy this API change when it lands.

Anyone using the built-in [email protected] from [email protected] would be vulnerable to silent downgrades. For example by changing the rendezvous server address.

@daviddias
Copy link
Member Author

daviddias commented Nov 27, 2017

We do not need to set up this flag allowJoinWithDisabledChallenge at all, the transport has access to the PeerId, it can always execute the Crypto Puzzle. See https://github.com/ipfs/js-ipfs/blob/master/src/core/runtime/libp2p-nodejs.js#L16

Also, thanks for catching the ^0.5.1, that is a typo. We don't release pre 1.0.0 deps with ^, only ~. That said:

If in the future a peer agrees to join without the Crypto Puzzle (because its users set it up like that), a previous node that doesn't support that mode won't get downgraded as it will just try to do the Puzzle with the other peer and then fail.

I might be missing something really obvious here. My apologies for making this convo long and possibly frustrating. I'm sure we can get to the bottom of it :)

@mkg20001
Copy link
Contributor

@diasdavid No if the id is set it can still join to a server without the puzzle! (Fixed in 0.6.0)

@daviddias
Copy link
Member Author

@mkg20001 ah, it was a bug on js-libp2p-websocket-star then, got it. Glad to you already fixed it. Let's make sure to track these things in issues when they pop up.

@ghost
Copy link

ghost commented Nov 27, 2017

The /p2p-websocket-star addresses need fixing, they lack the /tcp part which is essential for TCP-based addresses, see libp2p/js-libp2p-websocket-star#35

@daviddias
Copy link
Member Author

@lgierth the transport actually supports multiple variations https://github.com/whyrusleeping/js-mafmt/blob/master/test/index.spec.js#L87-L92. We have been doing the same for p2p-webrtc-star for while too https://github.com/whyrusleeping/js-mafmt/blob/master/test/index.spec.js#L75-L80

Are you proposing that we should fail on a multiaddr that doesn't explicitly call the tcp+443+wss?

@ghost
Copy link

ghost commented Nov 27, 2017

Are you proposing that we should fail on a multiaddr that doesn't explicitly call the tcp+443+wss?

Yes that would be good -- they're incorrect multiaddrs: /p2p-websocket-star requires a stream underneath, and /ip4 and /ip6 don't provide that capability. (In the case of /p2p-webrtc-star it's okay since there's backward-compat concerns, but the old-style addresses should be removed at some point too.)

It's not critical to remove the support right now, but at least the docs and tests should be updated before release, so that nobody starts to use these addresses in the first place.

@mitra42
Copy link

mitra42 commented Nov 28, 2017

is pubsub intended to replace YJS, or is YJS using ipfs-connector running over it, or is the best practice to configure YJS differently to use pubsub ?

Obviously there are parts of YJS crucial to pubsub type applications - which dont appear to be in the API - especially the ability to get the history of a channel in order for a newly joining peer to build the current state. Is it intended that YJS should be the way to do this going forward ?

@haadcode
Copy link
Member

@diasdavid tests in orbit-db running fine with the latest master 👍

@daviddias
Copy link
Member Author

daviddias commented Nov 28, 2017

Hi all, I created an issue for us to track the path moving out of the centralized rendezvous points which also brings a lot of context to why they exist and why they still exist -- libp2p/js-libp2p#134.

It's not critical to remove the support right now, but at least the docs and tests should be updated before release, so that nobody starts to use these addresses in the first place.

Alright, I'll update it everywhere! Thanks for reviewing, this why I like to create an announcement first and wait a bit to release :)

Update: Track: #1117

@diasdavid tests in orbit-db running fine with the latest master 👍

Woot! Thanks @haadcode! Does that mean that there will be a new orbit.chat soon with latest js-ipfs?

is pubsub intended to replace YJS, or is YJS using ipfs-connector running over it, or is the best practice to configure YJS differently to use pubsub ?

Good question. y.js using it's ipfs-connector uses PubSub already to propagate it's events (see the code: https://github.com/ipfs-shipyard/y-ipfs-connector/blob/master/src/index.js#L33).

One is not designed to replace the other, in fact, one enables the other. What you see happening is:

  • PubSub is innovation that has made its way into the IPFS Core Protocol
  • a sync'able KeyValue Store such as the one powered by y.js is still in IPFS Userland. That is why you have to attach it

We do have the intention to have a KeyValue Store inside IPFS but that is still at its early stage -- ipfs-inactive/interface-js-ipfs-core#157 --. The first step to that will be getting the API and a KeyValue store that works only locally (like MFS) and then design its syncing layer. Again, that is still very early stage, until then continue to check out things like y.js.

@mitra42
Copy link

mitra42 commented Nov 28, 2017

@diasdavid - thanks for the clarification. To be clear, we don't use YJS for KeyValue store, we use it for Append-Only-Logs, a critical building block for a lot of apps, and very hard to do in IPFS by any other means.

@haadcode
Copy link
Member

Woot! Thanks @haadcode! Does that mean that there will be a new orbit.chat soon with latest js-ipfs?

@diasdavid I won't have time to update orbit.chat anytime soon, but it is on my TODO list (we have API breaking changes in the new orbitdb, so orbit.chat needs a bit of work too).

we don't use YJS for KeyValue store, we use it for Append-Only-Logs

@mitra42 you might find https://github.com/orbitdb/orbit-db and its Log Database helpful for out-of-the-box append-only logs "natively" on IPFS. We just released a major new version that fixes many of the problems you might've had with previous versions.

@mitra42
Copy link

mitra42 commented Nov 29, 2017

@haadcode - the semantics of orbit-db append only logs, dont quite match, but I'll send you an email since its off-topic for release v0.27.0

@daviddias
Copy link
Member Author

Thanks @alexsicart for trying out this pre-release \o/ Quoting: https://twitter.com/alexsicart/status/937305396263473152

"Thanks! I've tried and the performance improvement is hugee!! Also less memory usage."

@daviddias
Copy link
Member Author

Notes on upgrading from js-ipfs 0.26 to 0.27

@daviddias
Copy link
Member Author

js-ipfs 0.27 released!

Tweet: https://twitter.com/daviddias/status/937565486912606208

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants