add-code-flow.md

IPFS : The Add command demystified

The goal of this document is to capture the code flow for adding a file (see the coreapi package) using the IPFS CLI, in the process exploring some datastructures and packages like ipld.Node (aka dagnode), FSNode, MFS, etc.

Concepts

• Files

---

Try this yourself

# Convert a file to the IPFS format.
echo "Hello World" > new-file
ipfs add new-file
added QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u new-file
12 B / 12 B [=========================================================] 100.00%

# Add a file to the MFS.
NEW_FILE_HASH=$(ipfs add new-file -Q)
ipfs files cp /ipfs/$NEW_FILE_HASH /new-file

# Get information from the file in MFS.
ipfs files stat /new-file
# QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u
# Size: 12
# CumulativeSize: 20
# ChildBlocks: 0
# Type: file

# Retrieve the contents.
ipfs files read /new-file
# Hello World

Code Flow

UnixfsAPI.Add() - Entrypoint into the Unixfs package

The UnixfsAPI.Add() acts on the input data or files, to build a merkledag node (in essence it is the entire tree represented by the root node) and adds it to the blockstore. Within the function, a new Adder is created with the configured Blockstore and DAG service`.

• adder.AddAllAndPin(files) - Entrypoint to the Add logic encapsulates a lot of the underlying functionality that will be investigated in the following sections.

  Our focus will be on the simplest case, a single file, handled by Adder.addFile(file files.File).

  • adder.addFile(file files.File) - Create the DAG and add to MFS

    The addFile(file) method takes the data and converts it into a DAG tree and adds the root of the tree into the MFS.

    https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L508-L521

    There are two main methods to focus on -

    1. adder.add(io.Reader) - Create and return the root DAG node

       This method converts the input data (io.Reader) to a DAG tree, by splitting the data into chunks using the Chunker and organizing them in to a DAG (with a trickle or balanced layout. See balanced for more info).

       The method returns the root ipld.Node of the DAG.

    2. adder.addNode(ipld.Node, path) - Add root DAG node to the MFS

       Now that we have the root node of the DAG, this needs to be added to the MFS file system. Fetch (or create, if doesn't already exist) the MFS root using mfsRoot().

       NOTE: The MFS root is an ephemeral root, created and destroyed solely for the add functionality.

       Assuming the directory already exists in the MFS file system, (if it doesn't exist it will be created using mfs.Mkdir()), the root DAG node is added to the MFS File system using the mfs.PutNode() function.

       • [MFS] PutNode(mfs.Root, path, ipld.Node) - Insert node at path into given MFS

         The path param is used to determine the MFS Directory, which is first looked up in the MFS using lookupDir() function. This is followed by adding the root DAG node (ipld.Node) in to this Directory using directory.AddChild() method.

       • [MFS] Add Child To UnixFS

         • directory.AddChild(filename, ipld.Node) - Add root DAG node under this directory

           Within this method the node is added to the Directory's DAG service using the dserv.Add() method, followed by adding the root DAG node with the given name, in the directory.addUnixFSChild(directory.child{name, ipld.Node}) method.

         • [MFS] directory.addUnixFSChild(child) - Add child to inner UnixFS Directory

           The node is then added as a child to the inner UnixFS directory using the (BasicDirectory).AddChild() method.

           NOTE: This is not to be confused with the directory.AddChild(filename, ipld.Node), as this operates on the UnixFS BasicDirectory object.

         • [UnixFS] (BasicDirectory).AddChild(ctx, name, ipld.Node) - Add child to BasicDirectory

           IMPORTANT: It should be noted that the BasicDirectory object uses the ProtoNode type object which is an implementation of the ipld.Node interface, seen and used throughout this document. Ideally the ipld.Node should always be used, unless we need access to specific functions from ProtoNode (like Copy()) that are not available in the interface.

           This method first attempts to remove any old links (ProtoNode.RemoveNodeLink(name)) to the ProtoNode prior to adding a link to the newly added ipld.Node, using ProtoNode.AddNodeLink(name, ipld.Node).

           • [Merkledag] AddNodeLink()

             The AddNodeLink() method is where an ipld.Link is created with the ipld.Node's CID and size in the ipld.MakeLink(ipld.Node) method, and is then appended to the ProtoNode's links in the ProtoNode.AddRawLink(name) method.

  • adder.Finalize() - Fetch and return the DAG root from the MFS and UnixFS directory

    The Finalize method returns the ipld.Node from the UnixFS Directory.

  • adder.PinRoot() - Pin all files under the MFS root

    The whole process ends with PinRoot recursively pinning all the files under the MFS root