Release v0.4.22

We're releasing a PATCH release of go-ipfs based on 0.4.21 containing some critical fixes.

The IPFS network has scaled to the point where small changes can have a wide-reaching impact on the entire network. To keep this situation from escalating, we've put a hold on releasing new features until we can improve our release process (which we've trialed in this release) and testing procedures.

This release includes fixes for the following regressions:

1. A major bitswap throughput regression introduced in 0.4.21 (ipfs/go-ipfs#6442).
2. High bitswap CPU usage when connected to many (e.g. 10,000) peers. See ipfs/go-bitswap#154.
3. The local network discovery service sometimes initializes before the networking module, causing it to announce the wrong addresses and sometimes complain about not being able to determine the IP address (ipfs/go-ipfs#6415).

It also includes fixes for:

1. Pins not being persisted after ipfs block add --pin (ipfs/go-ipfs#6441).
2. Panic due to concurrent map access when adding and listing pins at the same time (ipfs/go-ipfs#6419).
3. Potential pin-set corruption given a concurrent ipfs repo gc and ipfs pin rm (ipfs/go-ipfs#6444).
4. Build failure due to a deleted git tag in one of our dependencies (ipfs/go-ds-badger#64).

Thanks to:

• @hannahhoward for fixing both bitswap issues.
• @sanderpick for catching and fixing the local discovery bug.
• @campoy for fixing the build issue.

Release v0.4.22-rc1

Track progress on #6506.

Release v0.4.21

We're happy to announce go-ipfs 0.4.21. This release has some critical bug fixes and a handful of new features so every user should upgrade.

Key bug fixes:

• Too many open file descriptors/too many peers (#6237).
• Adding multiple files at the same time doesn't work (#6254).
• CPU utilization spikes and then holds at 100% (#5613).

Key features:

• Experimental TLS1.3 support (to eventually replace secio).
• OpenSSL support for SECIO handshakes (performance improvement).

IMPORTANT: This release fixes a bug in our security transport that could potentially drop data from the channel. Note: This issue affects neither the privacy nor the integrity of the data with respect to a third-party attacker. Only the peer sending us data could trigger this bug.

ALL USERS MUST UPGRADE. We intended to introduce a feature this release that, unfortunately, reliably triggered this bug. To avoid partitioning the network, we've decided to postpone this feature for a release or two.

Specifically, we're going to provide a minimum one month upgrade period. After that, we'll start testing the impact of deploying the proposed changes.

If you're running the mainline go-ipfs, please upgrade ASAP. If you're building a separate app or working on a forked go-ipfs, make sure to upgrade github.com/libp2p/go-libp2p-secio to at least v0.0.3.

Contributors

First off, we'd like to give a shout-out to all contributors that participated in this release (including contributions to ipld, libp2p, and multiformats):

Contributor	Commits	Lines ±	Files Changed
Steven Allen	220	+6078/-4211	520
Łukasz Magiera	53	+5039/-4557	274
vyzo	179	+2929/-1704	238
Raúl Kripalani	44	+757/-1895	134
hannahhoward	11	+755/-1005	49
Marten Seemann	16	+862/-203	44
keks	10	+359/-110	12
Jan Winkelmann	8	+368/-26	16
Jakub Sztandera	4	+361/-8	7
Adrian Lanzafame	1	+287/-18	5
Erik Ingenito	4	+247/-28	8
Reid 'arrdem' McKenzie	1	+220/-20	3
Yusef Napora	26	+98/-130	26
Michael Avila	3	+116/-59	8
Raghav Gulati	13	+145/-26	13
tg	1	+41/-33	1
Matt Joiner	6	+41/-30	7
Cole Brown	1	+37/-25	1
Dominic Della Valle	2	+12/-40	4
Overbool	1	+50/-0	2
Christopher Buesser	3	+29/-16	10
myself659	1	+38/-5	2
Alex Browne	3	+30/-8	3
jmank88	1	+27/-4	2
Vikram	1	+25/-1	2
MollyM	7	+17/-9	7
Marcin Rataj	1	+17/-1	1
requilence	1	+11/-4	1
Teran McKinney	1	+8/-2	1
Oli Evans	1	+5/-5	1
Masashi Salvador Mitsuzawa	1	+5/-1	1
chenminjian	1	+4/-0	1
Edgar Lee	1	+3/-1	1
Dirk McCormick	1	+2/-2	2
ia	1	+1/-1	1
Alan Shaw	1	+1/-1	1

Bug Fixes And Enhancements

This release includes quite a number of critical bug fixes and performance/reliability enhancements.

Error when adding multiple files

The last release broke the simple command ipfs add file1 file2. It turns out we simply lacked a test case for this. Both of these issues (the bug and the lack of a test case) have now been fixed.

SECIO

As noted above, we've fixed a bug that could cause data to be dropped from a SECIO connection on read. Specifically, this happens when:

1. The capacity of the read buffer is greater than the length.
2. The remote peer sent more than the length but less than the capacity in a single secio "frame".

In this case, we'd fill the read buffer to it's capacity instead of its length.

Too many open files, too many peers, etc.

Go-ipfs automatically closes the least useful connections when it accumulates too many connections. Unfortunately, some relayed connections were blocking in Close(), halting the entire process.

Out of control CPU usage

Many users noted out of control CPU usage this release. This turned out to be a long-standing issue with how the DHT handled provider records (records recording which peers have what content):

1. It wasn't removing provider records for content until the set of providers completely emptied.
2. It was loading every provider record into memory whenever we updated the set of providers.

Combined, these two issues were trashing the provider record cache, forcing the DHT to repeatedly load and discard provider records.

More Reliable Connection Management

Go-ipfs has a subsystem called the "connection manager" to close the least-useful connections when go-ipfs runs low on resources.

Unfortunately, other IPFS subsystems may learn about connections before the connection manager. Previously, if some IPFS subsystem tried to mark a connection as useful before the connection manager learned about it, the connection manager would discard this information. We believe this was causing #6271. It no longer does that.

Improved Bitswap Connection Management

Bitswap now uses the connection manager to mark all peers downloading blocks as important (while downloading). Previously, it only marked peers from which it was downloading blocks.

Reduced Memory Usage

The most noticeable memory reduction in this release comes from fixing connection closing. However, we've made a few additional improvements:

• Bitswap's "work queue" no longer remembers every peer it has seen indefinitely.
• The peerstore now interns protocol names.
• The per-peer goroutine count has been reduced.
• The DHT now wastes less memory on idle peers by pooling buffered writers and returning them to the pool when not actively using them.

Increased File Descriptor Limit

The default file descriptor limit has been raised to 8192 (from 2048). Unfortunately, go-ipfs behaves poorly when it runs out of file descriptors and it uses a lot of file descriptors.

Luckily, most modern kernels can handle thousands of file descriptors without any difficulty.

Commands

This release brings no new commands but does introduce a few changes, bugfixes, and enhancements. This section is hardly complete but it lists the most noticeable changes.

Take note: this release also introduces a few breaking changes.

[DEPRECATION] The URLStore Command Deprecated

The experimental ipfs urlstore command is now deprecated. Please use ipfs add --nocopy URL instead.

[BREAKING] The DHT Command Base64 Encodes Values

When responding to an ipfs dht get command, the daemon now encodes the returned value using base64. The ipfs command will automatically decode this value before returning it to the user so this change should only affect those using the HTTP API directly.

Unfortunately, this change was necessary as DHT records are arbitrary binary blobs which can't be directly stored in JSON strings.

[BREAKING] Base32 Encoded v1 CIDs By Default

Both js-ipfs and go-ipfs now encode CIDv1 CIDs using base32 by default, instead of base58. Unfortunately, base58 is case-sensitive and doesn't play well with browsers (see #4143.

Human Readable Numbers

The ipfs bitswap stat and and ipfs object stat commands now support a --humanize flag that formats numbers with human-readable units (GiB, MiB, etc.).

Improved Errors

This release improves two types of errors:

1. Commands that take paths/multiaddrs now include the path/multiaddr in the error message when it fails to parse.
2. ipfs swarm connect now returns a detailed error describing which addresses were tried and why the dial failed.

Ping Improvements

The ping command has received some small improvements and fixes:

1. It now exits with a non-zero exit status on failure.
2. It no longer succeeds with zero successful pings if we have a zombie but non-functional connection to the peer being pinged (#6298).
3. It now prints out the average latency when canceled with ^C (like the unix ping command).

Features

This release is primarily a bug fix release but it still includes two nice features from libp2p.

Experimental TLS1.3 support

G...

Release 0.4.20

We're happy to release go-ipfs 0.4.20. This release includes some critical
performance and stability fixes so all users should upgrade ASAP.

This is also the first release to use go modules instead of GX. While GX has
been a great way to dogfood an IPFS-based package manager, building and
maintaining a custom package manager is a lot of work and we haven't been able
to dedicate enough time to bring the user experience of gx to an acceptable
level. You can read #5850 for
some discussion on this matter.

Docker

As of this release, it's now much easier to run arbitrary IPFS commands within
the docker container:

> docker run --name my-ipfs ipfs/go-ipfs:v0.4.20 config profile apply server # apply the server profile
> docker start my-ipfs # start the daemon

This release also reverts a change that
caused some significant trouble in 0.4.19. If you've been running into Docker
permission errors in 0.4.19, please upgrade.

WebUI

This release contains a major
WebUI release with some
significant improvements to the file browser and new opt-in, privately hosted,
anonymous usage analytics.

Commands

As usual, we've made several changes and improvements to our commands. The most
notable changes are listed in this section.

New: ipfs version deps

This release includes a new command, ipfs version deps, to list all
dependencies (with versions) of the current go-ipfs build. This should make it
easy to tell exactly how go-ipfs was built when tracking down issues.

New: ipfs add URL

The ipfs add command has gained support for URLs. This means you can:

1. Add files with ipfs add URL instead of downloading the file first.
2. Replace all uses of the ipfs urlstore command with a call to ipfs add --nocopy. The ipfs urlstore command will be deprecated in a future
   release.

Changed: ipfs swarm connect

The ipfs swarm connect command has a few new features:

It now marks the newly created connection as "important". This should ensure
that the connection manager won't come along later and close the connection if
it doesn't think it's being used.

It can now resolve /dnsaddr addresses that don't end in a peer ID. For
example, you can now run ipfs swarm connect /dnsaddr/bootstrap.libp2p.io to
connect to one of the bootstrap peers at random. NOTE: This could connect you to
an arbitrary peer as DNS is not secure (by default). Please do not rely on
this except for testing or unless you know what you're doing.

Finally, ipfs swarm connect now returns all errors on failure. This should
make it much easier to debug connectivity issues. For example, one might see an
error like:

Error: connect QmYou failure: dial attempt failed: 6 errors occurred:
	* <peer.ID Qm*Me> --> <peer.ID Qm*You> (/ip4/127.0.0.1/tcp/4001) dial attempt failed: dial tcp4 127.0.0.1:4001: connect: connection refused
	* <peer.ID Qm*Me> --> <peer.ID Qm*You> (/ip6/::1/tcp/4001) dial attempt failed: dial tcp6 [::1]:4001: connect: connection refused
	* <peer.ID Qm*Me> --> <peer.ID Qm*You> (/ip6/2604::1/tcp/4001) dial attempt failed: dial tcp6 [2604::1]:4001: connect: network is unreachable
	* <peer.ID Qm*Me> --> <peer.ID Qm*You> (/ip6/2602::1/tcp/4001) dial attempt failed: dial tcp6 [2602::1]:4001: connect: network is unreachable
	* <peer.ID Qm*Me> --> <peer.ID Qm*You> (/ip4/150.0.1.2/tcp/4001) dial attempt failed: dial tcp4 0.0.0.0:4001->150.0.1.2:4001: i/o timeout
	* <peer.ID Qm*Me> --> <peer.ID Qm*You> (/ip4/200.0.1.2/tcp/4001) dial attempt failed: dial tcp4 0.0.0.0:4001->200.0.1.2:4001: i/o timeout

Changed: ipfs bitswap stat

ipfs bitswap stat no longer lists bitswap partners unless the -v flag is
passed. That is, it will now return:

> ipfs bitswap stat
bitswap status
	provides buffer: 0 / 256
	blocks received: 0
	blocks sent: 79
	data received: 0
	data sent: 672706
	dup blocks received: 0
	dup data received: 0 B
	wantlist [0 keys]
	partners [197]

Instead of:

> ipfs bitswap stat -v
bitswap status
	provides buffer: 0 / 256
	blocks received: 0
	blocks sent: 79
	data received: 0
	data sent: 672706
	dup blocks received: 0
	dup data received: 0 B
	wantlist [0 keys]
	partners [203]
		QmNQTTTRCDpCYCiiu6TYWCqEa7ShAUo9jrZJvWngfSu1mL
		QmNWaxbqERvdcgoWpqAhDMrbK2gKi3SMGk3LUEvfcqZcf4
		QmNgSVpgZVEd41pBX6DyCaHRof8UmUJLqQ3XH2qNL9xLvN
        ... omitting 200 lines ...

Changed: ipfs repo stat --human

The --human flag in the ipfs repo stat command now intelligently picks a
size unit instead of always using MiB.

Changed: ipfs resolve (ipfs dns, ipfs name resolve)

All of the resolve commands now:

1. Resolve recursively (up to 32 steps) by default to better match user
   expectations (these commands used to be non-recursive by default). To turn
   recursion off, pass -r false.
2. When resolving non-recursively, these commands no longer fail when partially
   resolving a name. Instead, they simply return the intermediate result.

Changed: ipfs files flush

The ipfs files flush command now returns the CID of the flushed file.

Performance And Reliability

This release has the usual collection of performance and reliability
improvements.

Badger Memory Usage

Those of you using the badger datastore should notice reduced memory usage in
this release due to some upstream changes. Badger still uses significantly more
memory than the default datastore configuration but this will hopefully continue
to improve.

Bitswap

We fixed some critical CPU utilization regressions in bitswap for this release.
If you've been noticing CPU regressions in go-ipfs 0.4.19, especially when
running a public gateway, upgrading to 0.4.20 will likely fix them.

Relays

After AutoRelay was introduced in go-ipfs 0.4.19, the number of peers connecting
through relays skyrocketed to over 120K concurrent peers. This highlighted some
performance issues that we've now fixed in this release. Specifically:

• We've significantly reduced the amount of memory allocated per-peer.
• We've fixed a bug where relays might, in rare cases, try to actively dial a
  peer to relay traffic. By default, relays only forward traffic between peers
  already connected to the relay.
• We've fixed quite a number of performance issues that only show up when
  rapidly forming new connections. This will actually help all nodes but will
  especially help relays.

If you've enabled relay hop (Swarm.EnableRelayHop) in go-ipfs 0.4.19 and it
hasn't burned down your machine yet, this release should improve things
significantly. However, relays are still under heavy load so running an open
relay will continue to be resource intensive.

We're continuing to investigate this issue and have a few more patches on the
way that, unfortunately, won't make it into this release.

Panics

We've fixed two notable panics in this release:

• We've fixed a frequent panic in the DHT.
• We've fixed an occasional panic in the experimental QUIC transport.

Content Routing

IPFS announces and finds content by sending and retrieving content routing
("provider") records to and from the DHT. Unfortunately, sending out these
records can be quite resource intensive.

This release has two changes to alleviate this: a reduced number of initial
provide workers and a persistent provider queue.

We've reduced the number of parallel initial provide workers (workers that send
out provider records when content is initially added to go-ipfs) from 512 to 6.
Each provide request (currently, due to some issues in our DHT) tries to
establish hundreds of connections, significantly impacting the performance of
go-ipfs and crashing some
routers.

We've introduced a new persistent provider queue for files added via ipfs add
and ipfs pin add. When new directory trees are added to go-ipfs, go-ipfs will
add the root/final CID to this queue. Then, in the background, go-ipfs will walk
the queue, sequentially sending out provider records for each CID.

This ensures that root CIDs are sent out as soon as possible and are sent even
when files are added when the go-ipfs daemon isn't running.

By example, let's add a directory tree to go-ipfs:

> # We're going to do this in "online" mode first so let's start the daemon.
> ipfs daemon &
...
Daemon is ready
> # Now, we're going to create a directory to add.
> mkdir foo
> for i in {0..1000}; do echo do echo $i > foo/$i; done
> # finally, we're going to add it.
> ipfs add -r foo
added QmUQcSjQx2bg4cSe2rUZyQi6F8QtJFJb74fWL7D784UWf9 foo/0
...
added QmQac2chFyJ24yfG2Dfuqg1P5gipLcgUDuiuYkQ5ExwGap foo/990
added QmQWwz9haeQ5T2QmQeXzqspKdowzYELShBCLzLJjVa2DuV foo/991
added QmQ5D4MtHUN4LTS4n7mgyHyaUukieMMyCfvnzXQAAbgTJm foo/992
added QmZq4n4KRNq3k1ovzxJ4qdQXZSrarfJjnoLYPR3ztHd7EY foo/993
added QmdtrsuVf8Nf1s1MaSjLAd54iNqrn1KN9VoFNgKGnLgjbt foo/994
added QmbstvU9mnW2hsE94WFmw5WbrXdLTu2Sf9kWWSozrSDscL foo/995
added QmXFd7f35gAnmisjfFmfYKkjA3F3TSpvUYB9SXr6tLsdg8 foo/996
added QmV5BxS1YQ9V227Np2Cq124cRrFDAyBXNMqHHa6kpJ9cr6 foo/997
added QmcXsccUtwKeQ1SuYC3YgyFUeYmAR9CXwGGnT3LPeCg5Tx foo/998
added Qmc4mcQcpaNzyDQxQj5SyxwFg9ZYz5XBEeEZAuH4cQirj9 foo/999
added QmXpXzUhcS9edmFBuVafV5wFXKjfXkCQcjAUZsTs7qFf3G foo

In 0.4.19, we would have sent out provider records for files foo/{0..1000}
before sending out a provider record for foo. If you were ask a friend to
download /ipfs/QmUQcSjQx2bg4cSe2rUZyQi6F8QtJFJb74fWL7D784UWf9, they would
(baring other issues) be able to find it pretty quickly as this is the first CID
you'll have announced to the netwo...

Release 0.4.19

We're happy to announce go 0.4.19. This release contains a bunch of important fixes and a slew of new and improved features. Get pumped and upgrade ASAP to benefit from all the new goodies! 🎁

Features

🔌 Initializing With Random Ports

Go-ipfs can now be configured to listen on a random but stable port (across restarts) using the new randomports configuration profile. This should be helpful when testing and/or running multiple go-ipfs instances on a single machine.

To initialize a go-ipfs instance with a randomly chosen port, run:

> ipfs init --profile=randomports

👂 Gateway Directory Listing

IPNS (and/or DNSLink) directory listings on the gateway, e.g. https://ipfs.io/ipns/dist.ipfs.io/go-ipfs/, will now display the ipfs hash of the current directory. This way users can more easily create permanent links to otherwise mutable data.

📡 AutoRelay and AutoNAT

This release introduces two new experimental features (courtesy of libp2p): AutoRelay and AutoNAT.

AutoRelay is a new service that automatically chooses a public relay when it detects that the go-ipfs node is behind a NAT. While relaying connections through a third-party node isn't the most efficient way to route around NATs, it's a reliable fallback.

To enable AutoRelay, set the Swarm.EnableAutoRelay option in the config.

AutoNAT is the service AutoRelay uses to detect if the node is behind a NAT. You don't have to set any special config flags to enable it.

In this same config section, you may also notice options like EnableRelayHop, EnableAutoNATService, etc. You do not need to enable these:

• EnableRelayHop -- Allow other nodes to use your node as a relay (disabled by default).
• EnableAutoNATService -- Help other nodes detect if they're behind a NAT (disabled by default).

📵 Offline Operation

There are two new "offline" features in this release: a global --offline flag and an option to configure the gateway to not fetch files.

Most go-ipfs commands now support the --offline flag. This causes IPFS to avoid network operations when performing the requested operation. If you've ever used the --local flag, the --offline flag is the (almost) universally supported replacement.

For example:

• If the daemon is started with ipfs daemon --offline, it won't even connect to the network. (note: this feature isn't new, just an example).
• ipfs add --offline some_file won't send out provider records.
• ipfs cat --offline Qm... won't fetch any blocks from the network.
• ipfs block stat --offline Qm... is a great way to tell if a block is locally available.

Note: It doesn't yet work with the refs, urlstore, or tar commands (#6002).

On to the gateway, there's a new Gateway.NoFetch option to configure the gateway to only serve locally present files. This makes it possible to run an IPFS node as a gateway to serve content of your choosing without acting like a public proxy. 🤫

📍 Adding And Pinning Content

There's a new --pin flag for both ipfs block put and ipfs urlstore add to match the --pin flag in ipfs add. This allows one to atomically add and pin content with these APIs.

NOTE 1: For ipfs urlstore add, --pin has been enabled by default to match the behavior in ipfs add. However, ipfs block put does not pin by default to match the current behavior.

NOTE 2: If you had previously used the urlstore and weren't explicitly pinning content after adding it, it isn't pinned and running the garbage collector will delete it. While technically documented in the ipfs urlstore add helptext, this behavior was non-obvious and bears mentioning.

🗂 File Listing

The ipfs ls command has two significant changes this release: it reports file sizes instead of dag sizes and has gained a new --stream flag.

First up, ipfs ls now reports file sizes instead of dag sizes. Previously, for historical reasons, ipfs ls would report the size of a file/directory as seen by IPFS including all the filesystem datastructures and metadata. However, this meant that ls -l and ipfs ls would print different sizes:

> ipfs ls /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv

QmZTR5bcpQD7cFgTorqxZDYaew1Wqgfbd2ud9QqGPAkK2V 1688 about
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y 200  contact
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7 322  help
QmejvEPop4D7YUadeGqYWmZxHhLc4JBUCzJJHWMzdcMe2y 12   ping
QmXgqKTbzdh83pQtKFb19SpMCpDDcKR2ujqk3pKph9aCNF 1692 quick-start
QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB 1102 readme
QmQ5vhrL7uv6tuoN9KeVBwd4PwfQkXdVVmDLUZuTNxqgvm 1173 security-notes

> ipfs get /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
Saving file(s) to QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
 6.39 KiB / 6.39 KiB [================================] 100.00% 0s

> ls -l QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
total 28
-rw------- 1 user group 1677 Feb 14 17:03 about
-rw------- 1 user group  189 Feb 14 17:03 contact
-rw------- 1 user group  311 Feb 14 17:03 help
-rw------- 1 user group    4 Feb 14 17:03 ping
-rw------- 1 user group 1681 Feb 14 17:03 quick-start
-rw------- 1 user group 1091 Feb 14 17:03 readme
-rw------- 1 user group 1162 Feb 14 17:03 security-notes

This is now no longer the case. ipfs ls and ls -l now return the same sizes. 🙌

Second up, ipfs ls now has a new --stream flag. In IPFS, very large directories (e.g., Wikipedia) are split up into multiple chunks (shards) as there are too many entries to fit in a single block. Unfortunately, ipfs ls buffers the entire file list in memory and then sorts it. This means that ipfs ls /ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki (wikipedia) will take a very long time to return anything (it'll also use quite a bit of memory).

However, the new --stream flag makes it possible to stream a directory listing as new chunks are fetched from the network. To test this, you can run ipfs ls --stream --size=false --resolve-type=false /ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki. You probably won't want to wait for that command to finish, Wikipedia has a lot of entries. 😉

🔁 HTTP Proxy

This release sees a new (experimental) feature contributed by our friends at Peergos: HTTP proxy over libp2p. When enabled, the local gateway can act as an HTTP proxy and forward HTTP requests to libp2p peers. When combined with the ipfs p2p command, users can use this to expose HTTP services to other go-ipfs nodes via their gateways. For details, check out the documentation.

Performance And Reliability

This release introduces quite a few performance/reliability improvements and, as usual, fixes several memory leaks. Below is a non-exhaustive list of noticeable changes.

📞 DHT

This release includes an important DHT fix that should significantly:

1. Reduce dialing.
2. Speed up DHT queries.
3. Improve performance of the gateways.

Basically, in the worst case, a DHT query would turn into a random walk of the entire IPFS network. Yikes!

Relevant PR: libp2p/go-libp2p-kad-dht#237

🕸 Bitswap

Bitswap sessions have improved and are now used for all requests. Sessions allow us to group related content and ask peers most likely to have the content instead of broadcasting the request to all connected peers. This gives us two significant benefits:

1. Less wasted upload bandwidth. Instead of broadcasting which blocks we want to everyone, we can ask fewer peers thus reducing the number of requests we send out.
2. Less wasted download bandwidth. Because we know which peers likely have content, we can ask an individual peer for a block and expect to get an answer. In the past, we'd ask every peer at the same time to optimize for latency at the expense of bandwidth (getting the same block from multiple peers). We had to do this because we had to assume that most peers didn't have the requested block.

‼️ Pubsub

This release includes some significant reliability improvements in pubsub subscription handling. If you've previously had issues with connected pubsub peers not seeing each-other's messages, please upgrade ASAP.

♻️ Reuseport

In this release, we've rewritten our previously error-prone go-reuseport library to not duplicate a significant portion of Go's low-level networking code. This was made possible by Go's new Control net.Dialer option.

In the past, our first suggestion to anyone experiencing weird resource or connectivity issues was to disable REUSEPORT (set IPFS_REUSEPORT to false). This should no longer be necessary.

🐺 Badger Datastore

Badger has reached 1.0. This release brings an audit and numerous reliability fixes. We are now reasonably confident that badger will become the default datastore in a future release. 👍

This release also adds a new Truncate configuration option for the badger datastore (enabled by default for new IPFS nodes). When enabled, badger will delete any un-synced data on start instead of simply refusing to start. This should be safe on all filesystems where the sync operation is safe and removes the need for manual intervention when restarting an IPFS node after a crash.

Assuming you initialized your badger repo with ipfs init --profile=badgerds, you can enable truncate on an existing repo by running: ipfs config --json "Datastore.Spec.child.truncate" true.

Refactors and Endeavors

🕹 Commands Library

The legacy commands library shim has now been completely removed...

Release 0.4.18

This is probably one of the largest go-ipfs releases in recent history, 3 months
in the making.

Features

The headline features this release are experimental QUIC support, the gossipsub
pubsub routing algorithm, pubsub message signing, and a refactored ipfs p2p
command. However, that's just scratching the surface.

QUIC

First up, on the networking front, this release has also introduced experimental
support for the QUIC protocol. QUIC is a new UDP-based network transport that
solves many of the long standing issues with TCP.

For us, this means (eventually):

• Fewer local resources. TCP requires a file-descriptor per connection while
  QUIC (and most UDP based transports) can share a single file descriptor
  between all connections. This should allow us to dial faster and keep more
  connections open.
• Faster connection establishment. When client authentication is included,
  QUIC has a three-way handshake like TCP. However, unlike TCP, this handshake
  brings us from all the way from 0 to a fully encrypted, authenticated, and
  multiplexed connection. In theory (not yet in practice), this should
  significantly reduce the latency of DHT queries.
• Behaves better on lossy networks. When multiplexing multiple requests over
  a single TCP connection, a single dropped packet will bring the entire
  connection to a halt while the packet is re-transmitted. However, because QUIC
  handles multiplexing internally, dropping a single packets affects only the
  related stream.
• Better NAT traversal. TL;DR: NAT hole-punching is significantly easier
  and, in many cases, more reliable with UDP than with TCP.

However, we still have a long way to go. While we encourage users to test this,
the IETF QUIC protocol is still being actively developed and will change. You
can find instructions for enabling it
here.

Pubsub

In terms of pubsub, go-ipfs now supports the gossipsub routing algorithm and
message signing.

The gossipsub routing algorithm is significantly more efficient than the
current floodsub routing algorithm. Even better, it's fully backwards compatible
so you can enable it and still talk to nodes using the floodsub algorithm. You
can find instructions to enable gossipsub in go-ipfs
here.

Messages are now signed by their authors. While signing has now been enabled by
default, strict signature verification has not been and will not be for at least
one release (probably multiple) to avoid breaking existing applications. You can
read about how to configure this feature
here.

Commands

In terms of new toys, this release introduces a new ipfs cid subcommand for
working with CIDs, a completely refactored ipfs p2p command, streaming name
resolution, and complete inline block support.

The new ipfs cid command allows users to both inspect CIDs and convert them
between various formats and versions. For example:

# Print out the CID metadata (prefix)
> ipfs cid format -f %P QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
cidv0-protobuf-sha2-256-32

# Get the hex sha256 hash from the CID.
> ipfs cid format -b base16 -f '0x%D' QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
0x46d44814b9c5af141c3aaab7c05dc5e844ead5f91f12858b021eba45768b4c0e

# Convert a base58 v0 CID to a base32 v1 CID.
> ipfs cid base32 QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby

The refactored ipfs p2p command allows forwarding TCP streams through two IPFS
nodes from one host to another. It's ssh -L but for IPFS. You can find
documentation
here.
It's still experimental but we don't expect too many breaking changes at this
point (it will very likely be stabilized in the next release). Quick summary of
breaking changes:

• We don't stop listening for local (forwarded) connections after accepting a
  single connection.
• ipfs p2p stream ls output now returns more useful output, first address is
  always the initiator address.
• ipfs p2p listener ls is renamed to ipfs p2p ls
• ipfs p2p listener close is renamed to ipfs p2p close
• Protocol names have to be prefixed with /x/ and are now just passed to
  libp2p as handler name. Previous version did this 'under the hood' and with
  /p2p/ prefix. There is a --allow-custom-protocol flag which allows you
  to use any libp2p handler name.
• ipfs p2p listener open and ipfs p2p stream dial got renamed:
  • ipfs p2p listener open p2p-test /ip4/127.0.0.1/tcp/10101
    new becomes ipfs p2p listen /x/p2p-test /ip4/127.0.0.1/tcp/10101
  • ipfs p2p stream dial $NODE_A_PEERID p2p-test /ip4/127.0.0.1/tcp/10102
    is now ipfs p2p forward /x/p2p-test /ip4/127.0.0.1/tcp/10102 /ipfs/$NODE_A_PEERID

There is now a new flag for ipfs name resolve - --stream. When the command
is invoked with the flag set, it will start returning results as soon as they
are discovered in the DHT and other routing mechanisms. This enables certain
applications to start prefetching/displaying data while the discovery is still
running. Note that this command will likely return many outdated records
before it finding and returning the latest. However, it will always return
valid records (even if a bit stale).

Finally, in the previous release, we added support for extracting blocks inlined
into CIDs. In this release, we've added support for creating these CIDs. You can
now run ipfs add with the --inline flag to inline blocks less than or equal
to 32 bytes in length into a CID, instead of writing an actual block. This
should significantly reduce the size of filesystem trees with many empty
directories and tiny files.

IPNS

You can now publish and resolve paths with namespaces other than /ipns and
/ipfs through IPNS. Critically, IPNS can now be used with IPLD paths (paths
starting with /ipld).

WebUI

Finally, this release includes the shiny updated
webui. You can view it by
installing go-ipfs and visiting http://localhost:5001/webui.

Performance

This release includes some significant performance improvements, both in terms
of resource utilization and speed. This section will go into some technical
details so feel free to skip it if you're just looking for shiny new features.

Resource Utilization

In this release, we've (a) fixed a slow memory leak in libp2p and (b)
significantly reduced the allocation load. Together, these should improve both
memory and CPU usage.

Datastructures

We've changed two of our most frequently used datastructures, CIDs and
Multiaddrs, to reduce allocation load.

First, we now store CIDs encode as strings, instead of decoded in structs
(behind pointers). In addition to being more compact, our Cid type is now a
valid map key so we no longer have to encode CIDs every time we want to use
them in a map/set. Allocations when inserting CIDs into maps/sets was showing up
as a significant source of allocations under heavy load so this change should
improve memory usage.

Second, we've changed many of our multiaddr parsing/processing/formatting
functions to allocate less. Much of our DHT related-work includes processing
multiaddrs so this should reduce CPU utilization when heavily using the DHT.

Streams and Yamux

Streams have always plagued us in terms of memory utilization. This was
partially solved by introducing the connection manager, keeping our maximum
connection count to a reasonable number but they're still a major memory sink.

This release sees two improvements on this front:

1. A memory leak in identify
   has been fixed. This was slowly causing us to leak connections (locking up
   the memory used by the connections' streams).
2. Yamux streams now use a buffer-pool backed, auto shrinking read buffer.
   Before, this read buffer would grow to its maximum size (a few megabytes) and
   never shrink but these buffers now shrink as they're emptied.

Bitswap Performance

Bitswap will now pack multiple small blocks into a single message thanks
ipfs/go-bitswap#5. While this won't
help when transferring large files (with large blocks), this should help when
transferring many tiny files.

Refactors and Endeavors

This release saw yet another commands-library refactor, work towards the
CoreAPI, and the first step towards reliable base32 CID support.

Commands Lib

We've completely refactored our commands library (again). While it still needs
quite a bit of work, it now requires significantly less boilerplate and should
be significantly more robust. The refactor immediately found two broken tests
and probably fixed quite a few bugs around properly returning and handling
errors.

CoreAPI

CoreAPI is a new way to interact with IPFS from Go. While it's still not
final, most things you can do via the CLI or HTTP interfaces, can now be done
through the new API.

Currently there is only one implementation, backed by go-ipfs node, and there are
plans to start http-api backed one soon. We are also looking into creating RPC
interface using this API, which could help performance in some use cases.

You can track progress in #4498

IPLD paths

We introduced new path type which introduces distinction between IPLD and
IPFS (unixfs) paths. From now on paths prefixed with /ipld/ ...

Release 0.4.17

Ipfs 0.4.17 is a quick release to fix a major performance regression in bitswap (mostly affecting go-ipfs -> js-ipfs transfers). However, while motivated by this fix, this release contains a few other goodies that will excite some users.

The headline feature in this release is urlstore support. Urlstore is a generalization of the filestore backend that can fetch file blocks from remote URLs on-demand instead of storing them in the local datastore.

Additionally, we've added support for extracting inline blocks from CIDs (blocks inlined into CIDs using the identity hash function). However, go-ipfs won't yet create such CIDs so you're unlikely to see any in the wild.

Features:

• URLStore (ipfs/go-ipfs#4896)
• Add trickle-dag support to the urlstore (ipfs/go-ipfs#5245).
• Allow specifying how the data field in the object get is encoded (ipfs/go-ipfs#5139)
• Add a -U flag to files ls to disable sorting (ipfs/go-ipfs#5219)
• Add an efficient --size-only flag to the repo stat (ipfs/go-ipfs#5010)
• Inline blocks in CIDs (ipfs/go-ipfs#5117)

Changes/Fixes:

• Make ipfs files ls -l correctly report the hash and size of files (ipfs/go-ipfs#5045)
• Fix sorting of files ls (ipfs/go-ipfs#5219)
• Improve prefetching in ipfs cat and related commands (ipfs/go-ipfs#5162)
• Better error message when ipfs cp fails (ipfs/go-ipfs#5218)
• Don't wait for the peer to close it's end of a bitswap stream before considering the block "sent" (ipfs/go-ipfs#5258)
• Fix resolving links in sharded directories via the gateway (ipfs/go-ipfs#5271)
• Fix building when there's a space in the current directory (ipfs/go-ipfs#5261)

Documentation:

• Improve documentation about the bloomfilter config options (ipfs/go-ipfs#4924)

General refactorings and internal bug fixes:

• Remove the Offset() method from the DAGReader (ipfs/go-ipfs#5190)
• Fix TestLargeWriteChunks seek behavior (ipfs/go-ipfs#5276)
• Add a build tag to disable dynamic plugins (ipfs/go-ipfs#5274)
• Use FSNode instead of the Protobuf structure in PBDagReader (ipfs/go-ipfs#5189)
• Remove support for non-directory MFS roots (ipfs/go-ipfs#5170)
• Remove UnixfsNode from the balanced builder (ipfs/go-ipfs#5118)
• Fix truncating files (internal) when already at the correct size (ipfs/go-ipfs#5253)
• Fix dagTruncate (internal) to preserve the node type (ipfs/go-ipfs#5216)
• Add an internal interface for unixfs directories (ipfs/go-ipfs#5160)
• Refactor the CoreAPI path types and interfaces (ipfs/go-ipfs#4672)
• Refactor precalcNextBuf in the dag reader (ipfs/go-ipfs#5237)
• Update a bunch of dependencies that haven't been updated for a while (ipfs/go-ipfs#5268)

---

Download:

• /ipfs/QmNuyz8s1MZHPBp3nu4hnAXJyXbyZXdVkJfKu38JiSv7KC/go-ipfs/v0.4.17/
• dist.ipfs.io

You can also download them from GitHub (but we recommend IPFS, obviously).