Explore instruction bundle nodes

[lorisleiva]

There is still a lot to explore here so this issue mainly offer a space to discuss this in more detail before committing to this.

The idea here is to describe bundles of instructions that are typically used together to create more complex operations.

For instance createAccount from the System program and initializeMint from the Token program could both be part of a createMint bundle which would link to these two instructions and provide a mapping regarding the accounts and arguments that refer to the same reference on both instructions. For instance, the created account on the createAccount instruction would also match the mint account of the initializeMint instruction.

By describing these instruction bundles, we allow renderers to offer helper methods for these, CLIs to offer more useful commands and documentation to provide more useful examples.

[etodanik]

This would be super useful for us. For context, we have a program with over 120 instructions, with most of those instructions existing as part of a bundle like this.

I propose to use JSON-Schema for validation since it also it has the ability to have resolvable $ref links to both in-document targets and URLs.

What I imagine we'd need:

• A way to describe custom parameters and refer instruction accounts / parameters to them for ergonomics (shown here)
• Some default way to also just "inherit" the parameters from the instructions if we're lazy and all our instructions live in the IDL... (not shown here)
• Defaults
• Optionals

Here's how I imagine it vaguely (This doesn't yet address how we refer to IDL, although it isn't outrageous to just establish a $ref convention to refer to a program within the IDL):

{
  "bundleName": "createAndInitializeMintAccount",
  "accounts": [
    {
      "name": "fromPubkey",
      "description": "The public key of the account funding the creation"
    },
    {
      "name": "toPubkey",
      "description": "The public key of the new account to be created"
    },
    {
      "name": "owner",
      "description": "The public key of the owner of the new account"
    },
    {
      "name": "tokenProgramId",
      "description": "The public key of the token program"
    },
    {
      "name": "mintAuthorityPubkey",
      "description": "The public key of the mint authority",
      "default": {
        "$ref": "#/accounts/0"
      }
    },
    {
      "name": "freezeAuthorityPubkey",
      "description": "The public key of the freeze authority (optional)",
      "isOptional": true
    }
  ],
  "args": [
    {
      "name": "lamports",
      "type": "number",
      "description": "Number of lamports to fund the new account"
    },
    {
      "name": "space",
      "type": "number",
      "description": "Space to allocate for the new account"
    },
    {
      "name": "decimals",
      "type": "u8",
      "description": "Number of decimals for the mint"
    }
  ],
  "instructions": [
    {
      "name": "createAccount",
      "program": "11111111111111111111111111111111 ", // This one is of course challenging. But I'd imagine that being able to reference to programs by public key will be needed
      "params": {
        "from_pubkey": {
          "$ref": "#/accounts/0"
        },
        "to_pubkey": {
          "$ref": "#/accounts/1"
        },
        "owner": {
          "$ref": "#/accounts/2"
        },
        "lamports": {
          "$ref": "#/args/0"
        },
        "space": {
          "$ref": "#/args/1"
        }
      }
    },
    {
      "name": "initializeMint",
      "program": "ATokenGPvbdGVxr1b2hvZbsiqW5xWH25efTNsLJA8knL",
      "params": {
        "token_program_id": {
          "$ref": "#/accounts/3"
        },
        "mint_pubkey": {
          "$ref": "#/accounts/1"
        },
        "mint_authority_pubkey": {
          "$ref": "#/accounts/4"
        },
        "freeze_authority_pubkey": {
          "$ref": "#/accounts/5"
        },
        "decimals": {
          "$ref": "#/args/2"
        }
      }
    }
  ]
}

[lorisleiva]

Thanks for that Danny. I love the idea of defining the expectations for the "bundle" first and then linking these expectations to actual instruction accounts/arguments. My only comment here is, since we already have InstructionNodes, can we just point to these instead — maybe using a new InstructionLinkNode and provide a mapping for their accounts/arguments?

[etodanik]

EDIT:
I'm using $ref here, but we might as well just be idiomatic to the way codama does things and use an InstructionLinkNode (similar to stuff like https://github.com/codama-idl/codama/blob/main/packages/nodes/docs/linkNodes/ProgramLinkNode.md), so just read it as that. I used $ref for my example, because they're generic JSON-Schema (which is a standard) , but I realize that in Codama we'd use link nodes instead.

EDIT 2:
Looks like we'll need roughly the following:

• A helper to add additional programs (Add an addProgramsVisitor #82)
• A new InstructionLinkNode (to be used where I use $ref in my example pseudo-IDL)
• A new InstructionBundleNode

There's an open question here @lorisleiva about where the InstructionBundleNode[] would go? On RootNode?

I think since we have about 120 instructions that need this, it'd make sense for me to start prototyping in a PR (with some internal happy path testing), and we can reason about how it feels once I have a working prototype?

Thanks for that Danny. I love the idea of defining the expectations for the "bundle" first and then linking these expectations to actual instruction accounts/arguments. My only comment here is, since we already have InstructionNodes, can we just point to these instead — maybe using a new InstructionLinkNode and provide a mapping for their accounts/arguments?

Yes! I was just thinking about it as you typed this.

1. It would be lovely to just inherit instructions that we have in the IDL, and provide mappings. Do we currently have a way to feed Codama multiple IDL's?
2. I imagine that referring to those would be easiest with a $ref convention of sorts?

Basically, the use cases I see are:

1. Multiple IX needing some arg to default to an account / arg from a previous IX (but keeping it use-overridable)
2. Overriding some sensible defaults for a use case?

Any IX that defaults in one of its' accounts / args to another previous IX account / arg would then render that specific arg optional.

For instance, a lazy default could look like:

{
  "bundleName": "createAndInitializeMintAccount",
  "instructions": [
    {
      "$ref": "system#/instructions/create_account" // assuming that this is the name of the ProgramNode in an IDL? 
      // another variation could imagine everything as one big JSON and then do: 
      "$ref": "#/programs/0/instructions/create_account"
    },
    {
      "$ref": "token#/instructions/initialize_mint"
    }
  ]
}

Then a more elaborate one could be:

{
  "bundleName": "createAndInitializeMintAccount",
  "instructions": [
    {
      "$ref": "#/program/instructions/create_account"
    },
    {
      "$ref": "#/additionalPrograms/0/instructions/initialize_mint"
    }
  ],
  // this could let us customize some of the parameters
  "accounts": [
    {
      "name": "mint_pubkey",
      "replace": [
        {
          "$ref": "#/program/instructions/create_account/arguments/2"
        },
        {
          "$ref": "#/additionalPrograms/0/instructions/initialize_mint/arguments/3"
        }
      ]
    },
    {
      // let's say that here we're replacing/overriding just for the sake of defaulting to something else
      "name": "mint_authority_pubkey",
      "replace": {
        "$ref": "#/additionalPrograms/0/instructions/initialize_mint/arguments/2"
      },
      "default": {
        "$ref": "#/program/instructions/create_account/arguments/2"
      }
    }
  ]
}

In this case I'd imagine we will then have to pass on accounts / params grouped by the IX name:

{
  "createAccount": {
    "accounts": {
      // ... all the accounts for this IX
    },
    "args": {
      // ... all the args for this IX
    }
  },
  "initializeMint": {
    "accounts": {
      // ... all the accounts for this IX
    },
    "args": {
      // ... all the args for this IX
    }
  }
}

[lorisleiva]

Great! I love your idea of using “overrides” for accounts/arguments of the underlying linked instructions. I think this is a great way to build bundles bottom up. We could even consider nesting bundles by having a InstructionBundleLinkNode.

Regarding your question on where these bundles should live: under the ProgramNode would be my answer. Everything should be under the context of a program IMO. For instance, the createMint may use the createAccount instruction from the system program but that operation is very much a construct of the token program.

I also think using the additionalPrograms array for that purpose is perfect because that is completely idiomatic to how Codama currently handles programs depending on each other. In the future, we may push this further by downloading these additional programs on-demand (say when that information is available on-chain) so when we do, this bundle logic will automatically benefit from that change.

[etodanik]

Tracking progress here:
#143