New consignment format

[dr-orlovsky]

Consignments are blobs of client-side-validated data which has to be sent between users in order for RGB state transfers to happen.

Each consignment must include all information required for its validation - except information about bitcoin transactions involved in the contract (this info is provided by a bitcoin core instance).

Such information includes:

• schema;
• contract genesis;
• set of strict type libraries (providing information about data types used by the contract);
• set of AluVM libraries running validation programs;
• collection of anchors with corresponding state transition bundles which may contain partially concealed information (NB: data attachments for state transitions are not a part of a consignment!);
• set of state extensions used by any of state transitions.

Alongside these consensus data consignment must also provide the following meta information:

• contract manifest which provides information about used RGB interface standards may include different media metadata;
• certificate and signature over schema for the schema developer;
• certificate and signature over genesis and manifest for the contract issuer;
• optional opentimestamp proof for the genesis.

Consignments may be small - just several kbytes - and grow quite large up to multiple gigabytes in size. They can be distributes via different channels, starting with dynamic QR codes (consignments with issued assets), instant messages, e-mail - or as files via FTP and HTTP. For small consignments a Base64 ascii armoring is provided to support distribution via QRs and messages. For large consignments sent over HTTP a streaming mode should be available such that the receiving party may run validation in parallel with downloading - and abort if the consignment is invalid without the need to download multiple gigabytes of data (this should prevent some of DoS attacks).

Thus, consignment binary data must support different forms:

• base64 armored text block;
• single binary file which must support simple seeking to the desired data piece (see the list of consignment components above);
• FS package (directory or package for OSes like MacOS);
• streamed data.

Content of the consignment is highly entropic (due to txids, Petersen commitments, bulletproofs, anchor data being >80% of the most of the consignments) - thus, application of additional compression is not desired.

Here I propose to discuss which format must be the best to work with such requirements.

[cryptoquick]

Compression does help rgbc consignment files, from my experience, it's one of Carbonado's test files:
https://github.com/diba-io/carbonado/tree/main/tests/samples
For FS, Carbonado could be a really good container format, since it can be encrypted on-disk using wallet keys, and can be handed off to other hosts to be verified remotely. It also builds in optional error correction on a binary level (not like bech32 bch).

[cryptoquick]

Also, if the streaming is broken, can I request to download from a specific point? (I assume yes, it is more implementation thing rather then file format).

Yes, picking up from a byte offset would be something that we'd like to support on HTTP frontends for the carbonado-node.

I mean does Carbonado provides some sort of header with offsets to all data components for a multi-component files (like consignments are)?

That's planned with Catalogs, it's mentioned in the README, and there's also an issue:
diba-io/carbonado#3

So my take was that Carbonado is designed for storing stash backups, not consignments - consignments are temporary and do not persist at all, thus they do not need a mayday protection (a sender can create a new consignment as long as it is needed).

The RGB methods in bitmask-core are stateless, so a consignment is effectively the same as a stash. They can also be encrypted with ECDH keys, so they can be privately shared with peers. This helps if an insecure or unauthenticated means of communication were used, such as email.

[dr-orlovsky]

Consignment can't be a stash since stash usually covers multiple contracts, while consignments are always single-contract specific. Stash can be exported only in form of a disclosure, but this is also a highly suboptimal.

[cryptoquick]

Yes, we export them and persist them separately.

[cryptoquick]

Also, when multiple assets are transferred in a single transaction, are multiple consignments produced?

[dr-orlovsky]

In RGB there are no "transactions", so it is not clear what do you mean "assets transfered by a single transaction".

If you mean a single witness transaction, than yes, it may have many RGB and non-RGB data for tapret commitments, but it has one-to-many relations with consignments.