potential memory leak in ipfs add

[whyrusleeping]

Several users have reported ipfs using way too much memory when adding large datasets.

This could be that the mfs structure is not ever flushing directories from memory, we probably need to periodically call Flush on parts of the structure during longer adds.

[Stebalien]

Several users have reported ipfs using way too much memory when adding large datasets.

That may be providing. We queue up a bunch of go routines to provide blocks and have no backpressure mechanism. Ask them to try again with --local and see if that makes a difference.

[Stebalien]

Actually, it looks like you're right.

[Stebalien]

Several issues:

1. When adding, we need to stream CIDs back to the user instead of queuing them all up.
2. When providing, we need to not queue CIDs in memory.
3. Our CIDs are large. The 48 bytes for the CID structure and 48 bytes for the Cid structure, 48 bytes for the CID multihash (according to the profiler). We should really jump on Suggestion to avoid exra layer of indirection by using a more compact representation of Cid go-cid#3.
4. We allocate 256KiB buffers when importing but don't make any attempts to shrink them down if we happen to read a smaller file. This means we end up batching ParallelBatchCommits(2*NumCPU) * MatchBatchBlocks(128) 256KiB blocks in our DAG batch writer. That's not good.

[whyrusleeping]

@Stebalien hrm... thats troublesome, but the users reporting issues here said that their machines had at least 8GB of ram. That wouldnt account for that would it?

[Stebalien]

We should try adding an LRU (or ARC) cache for CIDs.

[Stebalien]

@whyrusleeping

1. How many CPUs do they have.
2. You have to take GC into account. GC will lag a bit even if we're allocating a ton of buffers.

[Stebalien]

(old)