IPIP: include ipns-record in Gateway CAR responses

[lidel]

IPIP-351: IPNS Signed Records Response Format on HTTP Gateways will return a raw IPNS record, allowing a custom light client to verify IPNS without having to run DHT client.

Users asked for the ability to export both CIDs and IPNS record into a CAR, transport over sneakernet and load this on airgapped node, which is a good use case.

Another use case is Decentralized Gateway WG, which could benefit from the ability to have a single round-trip that includes full or partial CAR with content + ipns-record.

Broad strokes UX we want to accommodate:

$ curl http://gateway.foo/ipns/{id}?format=car&something > ipns-site-dump.car
$ ipfs import ipns-site-dump.car # imaginary command that goes beyond DAGs 
$ ipfs resolve /ipns/{id}   # works, even in offline mode
$ ipfs dag stat  /ipns/{id} # works, even in offline mode

Initial notes / subtasks

Tasks

Beta Give feedback

1. need to add a new ipns-record codec, to indicate special meaning of this additional block. Done: Add IPNS Record code multiformats/multicodec#312
2. Extend Trustless Gateway spec to support verifiable responses of /ipns/{libp2p-key} paths: decide if returned CAR should include two roots. One for data, second for ipns-record block. Specify if ipfs-records should be at the beginning of the CAR returned by /ipns/{id}?format=car[&something?] - to allow quick validation of response without wasting time for hashing invalid blocks that do not belong to CID in IPNS record.
3. Kubo CLI should have top level import / export that operates on content paths (instead of raw dags like ipfs dag versions), include all parent blocks and not just the final terminating element recursively, and be smart enough to use ipns-record and import it to local namesys cache if a block with ipns-record codec is present in a CAR

cc @hacdias (for now just fyi, tbd if this is something we need to prioritize for Saturn work any time soon)

[jphastings]

This is great! Am I understanding correctly from your second initial note that you expect a car to contain only ever one IPNS node? (I’m wondering why “at beginning” is important)

[lidel]

@jphastings these details are highly tbd. We may punt this until we have something like "content path aware CARv3".

If one would want to implement PoC today, they would want ipns-record(s) first in the stream, so the client can fail-fast if none of records is valid (performing signature check from IPNS spec + picking ipns record with highest sequence number if there is more than one). There is no point in parsing blocks for root CID if IPNS→CID mapping is not verified.

[jphastings]

My original reply collapsed; it was based on a misunderstanding!

Original reply

Thanks for the reply @lidel! I don't follow why an invalid IPNS record stored in a CAR would make the other blocks stored in that same CAR invalid/not useful?

An example case might be: I create a CAR that holds the data for two IPLD-encoded Podcast RSS feeds I like. Each podcast is represented by a number of blocks for the MP3s and feed metadata, and an IPNS record which becomes a signed statement that "the entity holding this IPNS private key defined this CID to be the root of the IPLD tree for their podcast at the specified time".

I could have both these podcasts in one CAR. The podcasts could share one episode (ie. have blocks in common). One of the IPNS records could be invalid, but I'd still want to be able to import the valid podcast (and possibly all the blocks).

I agree that putting the IPLD records first is a great optimisation (to help speed up access to large CAR files), but I don't see how it becomes a requirement?

[willscott]

@jphastings - if i ask for a car representing 'domain.com', and i get back an IPNS record of 'domain.com' pointing to a CID, and then the contents of the CID, then i should be cautious about rendering the CID if i can't validate the IPNS record.

@lidel - I'm curious what you see as requiring a bump of format to "CARv3" vs using the multicodec for IPNS and including such IPNS blocks within the existing car wrapper?

[jphastings]

@willscott my mistake, I misinterpreted @lidel’s comment as implying only one IPNS record per CAR, but on re-reading I can see that’s not the case — thanks for helping me spot that!

[jphastings]

Hey @lidel, I think the first part of your second bullet in the Task checklist is complete: the Trustless Gateway spec has been extended to include verifiable responses for IPLD records (with ?format=ipns-record). (Though I can't find this record in the CARs produced, yet)

I believe the second part isn't yet complete; unless the DFS ordering parameter requires IPNS records to arrive first? (It somewhat implies that, bu it only seems to be an implication)

[hacdias]

@jphastings I think the point is about having the IPNS record available in CAR files: /ipns/{key}?format=car. Right now, you need to download the CAR and separately get the record with ?format=ipns-record. The idea here is to return the record directly in the CAR itself so that you can fully verify your response without needing further HTTP requests. And as far as I know, the records are not yet sent in the CAR.

[jphastings]

Ahh; you’re absolutely right — that’s still not viable as it stands! Thank you.