RFC: Handling of Alternative Multibases

[kevina]

As we switch to using CidV1 base32 encoding the question is how we should handle this displaying of CID in alternative multibases.

This issue is only about the output. For the input will we already handle CIDv1 in any base supported by go-multibase.

Here are the alternatives, as I see it, and what I estimate it will take to do it.

1. Don't Preserve the multibase, Use Settings or Defaults to determine multibase

Accept CidV1 in any multibase, but ignore the base that is used. Output the CIDs based on either a global --cid-base flag or a config value.

This means that the string a user passes in may not be the same string that is returned. For example

$ ipfs pin add zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x
pinned bafkreiekhi5qqy64ovh7jckh5ewnyt6lcvfzqfubv3l66cazo5dknu6dre recursively

This also means that #5234 (ipfs resolve should respect CID base) will not be implemented.

The most straightforward way to implement this is replacing every call of cid.String() with something like cid.Encode(base) (see ipfs/go-cid#60) and make sure the base is available at the point when this method needs to be called. The WithValue functionally on contexts can be helpful here.

As most CID's are converted to strings on the server side, rather than the client, doing a quick hack such as setting a global variable is not going to work.

Another way to implement this, at least when working with the API, is to modify our API to make CID's a special type and then set the multibase as part of the writer that formats the output for this user. This means extra CPU cycles as the CID is converted from a binary represenation to a string twice. If we go this route I recommend the API be hardcoded to use base32 as is a more efficient encoding than our current default of base58btc.

2. Preserve the multibase. Children of DAGs inherit multi-base of parent when known.

When a CID is decoded preserve the multibase used with the CID so when it is converted back to a string the same multibase will be used. In addition associate the CID of any children of the DAG with the same multibase prefix. When a CID does not have a multibase associated use the settings from the command line or config option.

ipfs pin add will then display the same base. In addition commands like ipfs ls will use the same base to display directory entries as the hash used for the parent.

$ ipfs ls f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06
f01551220f6ba0f2491f2371dee0ea66e8812331b78f162697d2f41b5d55b667e5ae6bc67 3606  diff.go
f01551220e43f16413e6e1aa73eee0194d40363b7082d06faae8d6e94f18bd5f8cc08dfee 18299 object.go
f015512201985aa3462ff163c46de0f5e6634bde41f2bd537780ccbe5401aea1cccc49577 9057  patch.go

Implementing this will involve associating a multibase with CID's when it is known (see ipfs/go-cid#66). It will also involve setting the multibase of children CID's in the DAG code. New CID's will also need to get the base set somewhere.

After thinking about this I think this will involve less code change than #1 as the base does not need to be set everywhere, although the changes may not be as straightforward.

I think this is the most user friendly approach.

3. Attempt to Preserve the multibase.

This solution does not store the multi-base with the CID but makes an attempt to output the same multibase on a ad-hoc bases. For the current implementation of ipfs resolve it turned out to be easy. ipfs pin add is more difficult. It was in fact after looking at ipfs pin add that I decided that #2 will be required to do it properly, especially when multiple CIDs are given on the command line.

Implementing this involves doing #1 and in addition detecting the multibase of any CIDs used and setting and using the correct base for the output. Things can get tricky when CID's with different mutlibases are used at once. This will involve more code that either other solution.

#5289 started effort towards this solution (see the code for ipfs resolve and ipfs ls)

I do not like this solution as it will only work some of the time and prefer we go with #1 or #2 for more predictable output.

[kevina]

New CID's will also need to get the base set somewhere.

If we allow CIDs to carry a multibase, setting the base should be fairly easy to do with the new Builder interface (see ipfs/go-cid#53) and would require very little code change.

[Stebalien]

2. Preserve the multibase. Children of DAGs inherit multi-base of parent when known.

I'm not actually sure this is user-friendly. If I run:

ipfs pin f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x

And get back a bunch of different CIDs in different bases, that's going to be a bit confusing.

3. Attempt to Preserve the multibase.

I agree with your analysis here. Predictability is key.

---

Personally, I'd go with 1 and add a global flag (possibly with an environment variable as a fallback?).

As most CID's are converted to strings on the server side, rather than the client, doing a quick hack such as setting a global variable is not going to work.

Damn... We should be converting them back and forth (well, we should be using a binary RPC transport). Oh well.

Regardless, we can use an environment variable, we just have to forward it as an argument.

We could also start converting them back. That is, just replace every "string that is actually a CID" with a CID (and make sure that go can unmarshal everything correctly). That would actually be better from a backwards compat standpoint. How does that sound?

---

We could also mix in a bit of 3 but I'm wary about that. It's easy for commands that take one argument and spit out one argument but not something we can do reliably for commands that take many arguments.

[Stebalien]

Regardless of what we do, I really don't want to carry display information with CIDs themselves.

[kevina]

I'm not actually sure this is user-friendly. If I run:
ipfs pin f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x
And get back a bunch of different CIDs in different bases, that's going to be a bit confusing.

We will have to agree to disagree. I don't think it will be confusing at all if we return the same string. I.e.

$ ipfs pin add f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x
pinned f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 recursively
pinned zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x recursively

[kevina]

That is, just replace every "string that is actually a CID" with a CID (and make sure that go can unmarshal everything correctly). That would actually be better from a backwards compat standpoint. How does that sound?

This will be an API change. As {"cid": "<cid-string>"} will be replaced with {"cid": {"/": "<cid-string>"}} based on how the UnmarshalJSON and MarshalJSON functions are written: https://github.com/ipfs/go-cid/blob/master/cid.go#L387

[kevina]

@lidel @lgierth I would also appreciate some feedback on this from a users perspective.

[lidel]

@kevina If our endgame is to move to base32 as the universal default, users should not need to think about getting "the right" encoding unless they explicitly choose to do so.

If we look at this from that perspective, option 1 is not a problem, it is a feature. It may be actually a good UX idea to "normalize" to base32 by default, enforcing its universal use. If user needs to interact or get different encoding then config or CLI --cid-base ah-hoc flag should be enough for those rare cases.

That being said, option 2 is also fine (but not ideal, as users may need to always pass --cid-base base32 to normalize all outputs to base32).

[Stebalien]

We will have to agree to disagree. I don't think it will be confusing at all if we return the same string. I.e.

In that case, I agree. I was thinking about the refs -r case.

This will be an API change. As {"cid": ""} will be replaced with {"cid": {"/": ""}} based on how the UnmarshalJSON and MarshalJSON functions are written:

Hm. Yeah, I forgot about that (technically the correct solution but a massive breaking change).

[kevina]

This will be an API change. As {"cid": ""} will be replaced with {"cid": {"/": ""}} based on how the UnmarshalJSON and MarshalJSON functions are written.

One way to fix this is to introduce a new type, maybe called 'APICid` that has a type:

  var APICidBase := mbase.Base58BTC // effects the as String method, not JSON output
  const APICidJSONBase := mbase.Base32 
  type APICid stuct {
    str string // always in APICidJSONBase
  }

  func FromCid(*cid Cid) {}

  func FromString(str string) {} // may not be necessary

  func (c APICid) AsString() {
    return WithBase(APICidBase)
  }

  func (c APICid) WithBase(enc mbase.Encoder) string {
    if CidV0 string or already in enc {
      return c.str
    }
    // convert Cid to base enc
  }
 
  func (c APICid) AsCid() *cid.Cid {...}

  func (c APICid) UnmarshalJSON(b []byte) error {...}

  // encodes the cid as a JSON string
  func (c APICid) MarshalJSON() ([]byte, error) {
    return json.Marshal(c.str)
  }

Then convert the type of anything that is a Cid String to APICid. We might be able to get away with just calling APICid as Cid, especially if we put this in its own package.

To change the global base the variable APICidBase can be changed. This will be a client side change so we can use a global variable now.

[Stebalien]

Another solution is to use refmt. It has support for explicitly specifying how different types should be encoded to/from JSON.

[kevina]

Another solution is to use refmt. It has support for explicitly specifying how different types should be encoded to/from JSON.

Perhaps. I would need to spend some time investigating that option. The idea of a new type is that it works within our existing framework....

[Stebalien]

Fair enough. That's actually a pretty clean way to do it.

APICidJSONBase

Won't that cause problems with CIDv0?

[kevina]

Won't that cause problems with CIDv0?

I am still of the option that will should just allow multibase prefixes on CidV0. @whyrusleeping was against it but I discussion feels unfinished as I do not see how it can cause harm.

However, since we made that decision that multibase prefixes on CidV0 are not allowed, CidV0 will simply remain the same while CidV1 will be converted to APICidJSONBase. I am against the idea of upgrading CidV0 to CidV1 except at the U.I. level as it is not technically correct.

[kevina]

The client side option didn't work out quite as planned so now I am of the opinion that it is better to do it on the server side in order to move things forward.

There are now two outstanding p.r. #5777 and #5789. There are mostly identical except one does most of the conversion on the client side and the other the server side. From a comment in #5777:

The client side solution ended up being not so nice because sometime we return formatted string with CID in them (see ipfs refs for example). Maybe when we clean these up and start passing real CIDs via the API and we can switch to client side, but for now server side makes more sense.

My current thinking is to switch to server side. Most of the time CIDs are encoded in the commands package. When there are not, for example with the CoreAPI the plan is to reencode the CIDs after we get them back before calling Emit to avoid having to thread the parameter though outside of the commands package. Eventually I think that the CoreAPI should return CIDs and not strings and we should perhaps not reuse the same datastructures for the CoreAPI and what we use to send the result over the HTTP API.

For completeness there is also a third alternative solution that @magik6k left as a review comment in #5789 with regard to do the conversion within the CoreAPI which I responded to in that p.r.

I created two separate p.r. so we can compare the impact of the change and make an informed decision. In both implementations I simplified things as best I could so hopefully one of them can get in A.S.A.P. and before any more command lib changes.

Once this is in the next step is to provide (at minimum) a CidBase config option which will change the default option of the --cid-base parameter. For now it will default to base58btc but in the not so far future we can change it to base32 for new installations. The details of how this will work is best left to a separate issue.

CC: @Stebalien

[magik6k]

So if this is badly time constrained, I'm probably fine with the current server-side approach, it's not super optimal, but gives us something to iterate on.

[kevina]

@magik6k thanks, that the idea, something as non-invasive as possible so we can clean it up later.

[kevina]

@Stebalien @alanshaw if it will help move things along may I offer the following compromise:

1. Paths and CID hashes that are really strings will be encoded server side and thus respect the ?cid-base= HTTP end point.
2. Moving forward true CID's (in JSON those encoded as {'/': ""}) will not be effected by the ?cid-base= HTTP end point. If we switch to a binary API then they will be the raw binary bytes and thus not have a multibase prefix. If JSON is used the multibase used will not be effected by the ?cid-base= HTTP end point. For now this means they will still be encoded using base58btc, however eventually we will switch this to base32 for CIDv1.

Right now we don't use true CIDs in many places in the API in go-ipfs. We do in some parts of ipfs filestore and ipfs dag. I can't speak for ipfs dag but the use of CID's in ipfs filestore was accidental due to the serialization of the internal structure which uses a CID.

Thoughts?

[alanshaw]

IMHO it should be supported on all the HTTP endpoints that return CIDs or none. From an outsider's point of view supporting it on some endpoints but not others will seem arbitrary, will be frustrating and might cause some confusion.

I didn't realise this would be quite so problematic and I don't wish to use up your time unnecessarily. I feel this has gone on for a bit too long now and I don't want the reason to be because of something I have suggested. I think your time would be better used to actually work on the switch to base32.

All I really need is consensus with the go team on whether to support it or not on the HTTP endpoints and I can get ipfs/js-ipfs#1552 merged. I believe we should add support but if go-ipfs isn't able and there's no intention to do so then there's no point in js-ipfs supporting it and I will back out my changes.

For the record, there are no bad feelings here, lets just get the decision made and push this forwards 😄 🚢

[kevina]

@alanshaw

I think the JS and Go implementation are structured differently. The Go implemenation uses a strict client/server model. For go-ipfs supporting the HTTP endpoints basically mean doing it server side. That being said:

IMHO it should be supported on all the HTTP endpoints that return CIDs or none

None is probably not a good option because things like paths are best done via the server side, otherwise we will have to reparse the path client side to fix the CID (at least in the go-ipfs implementation)

I proposed this because I believe @Stebalien wanted to make it possible to switch to a more compact binary API such as using CBOR for the response rather than JSON. If a binary API is used it would make sense that the CIDs are stored as binary strings rather then encoded using a multibase. In this case the CIDs will need to be converted client side.

I didn't realise this would be quite so problematic and I don't wish to use up your time unnecessarily. I feel this has gone on for a bit too long now and I don't want the reason to be because of something I have suggested.

Your suggestion was only part of the reason I decided to to use a server side implementation and you are not the reason this has gone on for so long.

[Stebalien]

None is probably not a good option because things like paths are best done via the server side, otherwise we will have to reparse the path client side to fix the CID (at least in the go-ipfs implementation)

Very good point.

---

I agree with all the arguments for doing this server side (paths) and I also agree that we can't really do it for "real" (not just text) CIDs. Given some IPLD object, we don't want to reach in and change the version of it's CIDs (internally).

This also gives us a pretty clear boundary: if we have a string CID, we convert on the server; if we have a CID "object", we convert on the client.

It's a bit annoying that we don't do everything either on the server or the client but I can't see a way to do that.

[Stebalien]

Thank you @kevina. This was a well thought out and clear analysis of the problem.

[alanshaw]

None is probably not a good option because things like paths are best done via the server side, otherwise we will have to reparse the path client side to fix the CID

Yes, agree. I wasn't actually including endpoints that return paths when I said it should be supported on all the HTTP endpoints that return CIDs or none.

---

Thanks for your input @Stebalien and also @kevina for taking the time to consider this so carefully!

The following is a list of endpoints that JS IPFS supports that return a CID (or a CID in a path). I've put ✅ next to the ones that will support ?cid-base query param and ❌ next to the ones that will not based on @kevina's assertion that the ones that return CIDs as { "/": "..." } will not be supported. Could I get confirmation that this is correct? 🙏

• /api/v0/bitswap/wantlist ❌
• /api/v0/bitswap/stat ❌
• /api/v0/block/put ✅
• /api/v0/block/stat ✅
• /api/v0/add ✅
• /api/v0/ls ✅
• /api/v0/object/new ✅
• /api/v0/object/get ✅
• /api/v0/object/put ✅
• /api/v0/object/stat ✅
• /api/v0/object/links ✅
• /api/v0/object/patch/append-data ✅
• /api/v0/object/patch/set-data ✅
• /api/v0/object/patch/add-link ✅
• /api/v0/object/patch/rm-link ✅
• /api/v0/pin/ls ✅
• /api/v0/pin/add ✅
• /api/v0/pin/rm ✅
• /api/v0/resolve ✅

[kevina]

/api/v0/bitswap/wantlist
/api/v0/bitswap/stat

Sorry I overlooked those commands. @Stebalien do we want to do these server side?

/api/v0/object/get

I skipped ipfs object in my p.r. due to the delicate nature that although outputting CIDs in an alternative base would be okay upgrading to CIDv1 is not okay when outputting a dag contents, because if a user uses that data to recreate the dag they will get a different object. What I am unsure of is we we should auto-upgrade the CID bases on any of the command in ipfs object. @Stebalien thoughts?

[Stebalien]

/api/v0/object/get

So, really, I'd like to avoid changing the CID version for the dag and object commands as these are working with IPLD objects where the CID version really matters. We should probably support converting the base encoding (when using CIDv1) but I don't feel too strongly about that.

However, having the options available but not applying them may be a bit confusing. I wonder if these options shouldn't, in fact, be global but should be added to every command where they apply. Thoughts?

/api/v0/bitswap/wantlist
/api/v0/bitswap/stat

We just need to consistently draw the line. If we format CIDs in wantlist/stat, we should also do so in the filestore commands.

• The current line is: The API returns (crappy) DagJSON objects where true DagJSON CIDs are left unformatted.
• Another reasonable line is: The API always uses the requested base for CIDs and auto-upgrades CIDs from v0 to v1 when necessary except in the dag and object commands.

I'm fine either way but I think this matters more for JavaScript. @alanshaw?

[kevina]

@Stebalien form an implementation standpoint fixing ipfs dag to convert the CIDs on either side will be messy. Right now we return the abstract (i.e. an interface and not a struct) Node object which gets directly converted to JSON. To fix this server side we will need a stateful encoder so we can instruct it how to encode CIDs. To fix this client side we can use a global variable but we the default will effect how all CIDs are converted.

[Stebalien]

So, I think we can do this by specifying the JSON encoder for the command (server-side) and using something like refmt to encode instead of encoding/json. Unlike encoding/json, refmt allows the programmer to specify how encoding should work separate from the type itself.

Example because it's a bit confusing:

package main

import (
	"fmt"
	"github.com/ipfs/go-cid"
	"github.com/multiformats/go-multibase"
	"github.com/polydawn/refmt/json"
	"github.com/polydawn/refmt/obj/atlas"
)

func main() {
	at := atlas.MustBuild(
		atlas.BuildEntry(cid.Cid{}).Transform().TransformMarshal(atlas.MakeMarshalTransformFunc(func(c cid.Cid) (interface{}, error) {
			b, _ := c.StringOfBase(multibase.Base32)
			return map[string]string{"/": b}, nil
		})).Complete(),
	)
	c, _ := cid.Parse("zb2rhkm1A8HXsfcGeuvKvAJtcksYSGspSj5udWoCZ6cpZk7Zr")
	obj := map[string]interface{}{"foo": c}
	marshaled, _ := json.MarshalAtlased(
		json.EncodeOptions{},
		obj,
		at,
	)
	fmt.Printf("%s", marshaled)
}

We can probably use the same logic client-side.

---

However, I'm just throwing this out there so we consider it. I'll leave any decisions up to you and @alanshaw (and @lidel and anyone else needing this from the browser).

[alanshaw]

I wonder if these options shouldn't, in fact, be global but should be added to every command where they apply. Thoughts?

In the JS PR the cid-base option is added per command.

We just need to consistently draw the line. If we format CIDs in wantlist/stat, we should also do so in the filestore commands

We don't have a filestore API yet in HTTP, core or CLI so whatever gets decided here can be implemented when the time comes.

Likewise we don't have a dag HTTP API yet (although we do have core and partial CLI) hence why I didn't list /api/v0/dag above.

The API always uses the requested base for CIDs and auto-upgrades CIDs from v0 to v1 when necessary except in the dag and object commands.

When you say API, do you mean HTTP, CLI or core or all of them?

We're trying to get away from core returning string CIDs (paths are fine) but I'm totally fine with the CLI and HTTP API to "always uses the requested base for CIDs and auto-upgrades CIDs from v0 to v1 when necessary except in the dag and object commands.".

[kevina]

However, having the options available but not applying them may be a bit confusing. I wonder if these options shouldn't, in fact, be global but should be added to every command where they apply. Thoughts?

I think it is better as a global option because it allows you to blindly use the option in scripts without having to worry if the command supports it. You can also do something like:

alias ipfs="ipfs --cid-base=base64url"

if for example you want or need all output in a different multibase.

I do plan to propose config options for changing the defaults, but sometimes you want to change the default just for that script or local environment. Also the config options as I envision them will have slightly different effects that the command line option. For example --cid-base may change the base used in a path so that the input is different from the output for /ipfs paths but the config option won't as it will change the default base only.

[kevina]

So, I think we can do this by specifying the JSON encoder for the command (server-side) and using something like refmt to encode instead of encoding/json. Unlike encoding/json, refmt allows the programmer to specify how encoding should work separate from the type itself.

I think this is something we should consider in the future. What I want to do now is make sure anything we do now allows us to change to something like this in the future, preferable in a way that is invisible to the end-user.

[kevina]

except in the dag and object commands

I would add bitswap to this list. It is a fairly low-level command and it could make a difference if a CidV0 or CidV1 is on the list.

[Stebalien]

However, if we have this list of excluded commands, we need to be clear when we don't apply these functions.

[kevina]

@Stebalien

Another reasonable line is: The API always uses the requested base for CIDs

Until we switch to something like refmt doing this in the general case will be difficult. In addition because we don't store the multibase with the CIDs the client still needs to respect the --cid-base option because the multibase is lost once we have the go object. We could use a new type in some places, but this isn't going to really work when the CID is embedded deep inside the object such as the case for ipfs dag get output.

I agree somewhat that having the server also respect the --cid-base option will be helpful, especially for the non-ipfs clients that make use of the JSON output. So my vote is we aim to support it, but right now some commands will be broken in this regard (the command will still respect the option in general, but the JSON output won't).

[Stebalien]

Let's just do the best we can.

[kevina]

@alanshaw go-ipfs will not support the cid-base HTTP API endpoints right away for bitswap, but that is because implementing them without switching to refmt will be to difficult and not worth the effort at this time. Since you already implemented them, there is no harm is keeping them. The go-ipfs will support the cid-base option for botswap but only on the command line. Also note the dag, object, and bitswap commands should not auto-upgrade CIDv0 to CIDv1 as discussed above.

go-ipfs will support a flag to change the default behavior for auto-upgrading CIDs but we have not come to a consensus yet on what that flag will be.

Let me know if anything is unclear.