-
-
Notifications
You must be signed in to change notification settings - Fork 16
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #124 from DefiantLabs/pharr117/custom-chain-module…
…s-examples-docs pharr117/custom chain modules examples and docs
- Loading branch information
Showing
13 changed files
with
129 additions
and
40 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
44 changes: 43 additions & 1 deletion
44
docs/reference/custom_cosmos_module_extensions/cosmos_indexer_modules.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,45 @@ | ||
| # Cosmos Indexer Modules | ||
|
|
||
| Coming Soon | ||
| The [`cosmos-indexer-modules`](https://github.com/DefiantLabs/cosmos-indexer-modules) package provides a set of modules that extend the supported transaction message types in the Cosmos Indexer. These modules are used to extend the supported transaction message types in the Cosmos Indexer by providing custom message types for Cosmos SDK modules that are not part of the base Cosmos SDK. | ||
|
|
||
| ## Strategy | ||
|
|
||
| The `cosmos-indexer-modules` package provides a set of packages that allow access to a type URL mapping for custom message types. These mappings can be used to register custom message types with the codec in the Cosmos Indexer. | ||
|
|
||
| The types defined in the subpackages must fit the Cosmos SDK `Msg` interface, which is used to define the transaction message types in the Cosmos SDK. These types are protobuf messages that are used to define the transaction messages that are sent to the blockchain and returned in the blockchain responses. | ||
|
|
||
| The `cosmos-indexer-modules` package includes full `Msg` implementations for various (and growing) Cosmos SDK modules. This is achieved by generating the protobuf message types for the modules and implementing the `Msg` interface for each message type. These are then provided in a module-specific type URL mapping that can be used to register the custom message types with the codec in the Cosmos Indexer. | ||
|
|
||
| ## Usage | ||
|
|
||
| The following shows usage of how one of the `cosmos-indexer-modules` packages can be used to extend the supported transaction message types in the Cosmos Indexer. The `cosmos-indexer-modules` package contains a [`block-sdk`](https://github.com/DefiantLabs/cosmos-indexer-modules/tree/main/block-sdk) subpackage that provides a set of custom message types for the [Skip MEV `blocksdk`](https://github.com/skip-mev/block-sdk) module. | ||
|
|
||
| The `block-sdk` package defines a [`GetBlockSDKTypeMap` function](https://github.com/DefiantLabs/cosmos-indexer-modules/blob/main/block-sdk/msg_types.go#L17-L26) that returns a map of type URLs to the custom message types for the `blocksdk` module. This map can be used to register the custom message types with the codec in the Cosmos Indexer. The underlying types have been generated using protobuf definitions for the `blocksdk` module. | ||
|
|
||
| These can be passed to the `RegisterCustomMsgTypesByTypeURLs` method in the `Indexer` type in the Cosmos Indexer to register the custom message types with the codec. This allows the Cosmos Indexer to decode and encode the custom message types to Go types at runtime. | ||
|
|
||
| ```go | ||
| package main | ||
|
|
||
| import ( | ||
| "log" | ||
|
|
||
| blockSDKModules "github.com/DefiantLabs/cosmos-indexer-modules/block-sdk" | ||
| "github.com/DefiantLabs/cosmos-indexer/cmd" | ||
| ) | ||
|
|
||
| func main() { | ||
| indexer := cmd.GetBuiltinIndexer() | ||
|
|
||
| indexer.RegisterCustomMsgTypesByTypeURLs(blockSDKModules.GetBlockSDKTypeMap()) | ||
|
|
||
| err := cmd.Execute() | ||
| if err != nil { | ||
| log.Fatalf("Failed to execute. Err: %v", err) | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| By providing the custom message types for the `blocksdk` module, the Cosmos Indexer can now decode custom message types to Go types at runtime. This allows the Cosmos Indexer to extend the supported transaction message types and index the custom message types for the `blocksdk` module. | ||
|
|
||
| This example can be found in the `cosmos-indexer-modules` [examples/block-sdk-indexer](https://github.com/DefiantLabs/cosmos-indexer/tree/d020840f44775bf1680765867d54338592ac3caa/examples/block-sdk-indexer) codebase. This example also provides an example `filter.json` file for indexing only the `blocksdk` module messages, which conforms to the filter file creation requirements documented in the [Filtering](../../usage/filtering.md) doc. |
41 changes: 40 additions & 1 deletion
41
docs/reference/custom_cosmos_module_extensions/custom_message_type_registration.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,42 @@ | ||
| # Custom Message Type Registration | ||
|
|
||
| Coming Soon | ||
| The Cosmos Indexer comes with a built-in codec that is used to decode JSON RPC responses and their Protobuf encoded transaction messages. The codec is used to decode the response into Go types where appropriate. By default, the codec is set up to handle the base Cosmos SDK modules. | ||
|
|
||
| However, the codec provides a way to register custom message types with the codec, allowing for decoding and encoding of custom message types to Go types at runtime by type URL. | ||
|
|
||
| The two main ways to register custom message types with the codec are: | ||
|
|
||
| 1. Using the Cosmos SDK `AppModuleBasics` interface to register an entire Cosmos SDK module with the codec by providing the module's `AppModuleBasic` implementation to the `Indexer` type before application execution | ||
| 2. Using the custom message type URL tied to an underlying type to register custom message types with the codec | ||
|
|
||
| These methods are described in detail below. | ||
|
|
||
| ## Module Registration using AppModuleBasics | ||
|
|
||
| In normal usage of the Cosmos SDK, message types are registered with the codec using the `AppModuleBasics` interface. The `RegisterInterfaces` method of the `module.BasicManager` interface is used to register custom message types with the codec. This is how the base Cosmos SDK modules are provided in the `probe` package and are used by the Indexer by default. | ||
|
|
||
| This is done by: | ||
|
|
||
| 1. Pulling the base Cosmos SDK modules into the `probe` client package and providing them in the `DefaultModuleBasics` variable, as can be seen in the [probe client package config.go file](https://github.com/DefiantLabs/probe/blob/main/client/config.go#L30-L31) | ||
| 2. During `probe` client creation, using these module basics to register the base Cosmos SDK modules with the `probe` `ChainClientConfig` type, as can be seen in the [cosmos-indexer probe probe.go file](https://github.com/DefiantLabs/cosmos-indexer/blob/main/probe/probe.go#L26-L27) | ||
| 3. These module basics then have thier module-specific interfaces registered with the codec during `probe` client creation, as can be seen in the [probe client encoding.go file](https://github.com/DefiantLabs/probe/blob/main/client/encoding.go#L30) `MakeCodec` function. | ||
|
|
||
| The list of AppModuleBasics registered to the probe client can be extended to include new modules. The `Indexer` type provides a `RegisterCustomModuleBasics` in the [indexer package types.go file](https://github.com/DefiantLabs/cosmos-indexer/blob/main/indexer/registration.go#L14-L16) method that registers custom module basics with the indexer. This provides the `probe` client with the ability to register module-specific message types with the codec. | ||
|
|
||
| The main difficulty with this approach is that it requires the developer to have access to the module's `AppModuleBasic` implementation. This is not always possible, especially when dealing with custom modules that are not part of the base Cosmos SDK. For example, the following list of reasons, amongst others, may prevent the developer from using the `AppModuleBasic` interface: | ||
|
|
||
| 1. The module is not part of the base Cosmos SDK and does not use the exact version of the Cosmos SDK that the `cosmos-indexer` package is built on. | ||
| 2. The module is not open source and the developer does not have access to the module's `AppModuleBasic` implementation for registration. | ||
|
|
||
| In these cases, the developer can use the custom message type URL registration method instead. | ||
|
|
||
| ## Custom Message Type Registration using Type URL | ||
|
|
||
| This is useful for extending the supported transaction message types in the Cosmos Indexer. For instance, the `RegisterCustomTypeURL` function in the [client codec types package interface_registry.go file](https://github.com/DefiantLabs/probe/blob/main/client/codec/types/interface_registry.go) can be used to register custom message types with the codec. | ||
|
|
||
| This is exactly how the Cosmos Indexer extends the supported transaction message types. The `Indexer` provides a [`RegisterCustomMsgTypesByTypeURLs`]((https://github.com/DefiantLabs/cosmos-indexer/blob/main/indexer/registration.go#L18-L19)) method that registers custom message types with the indexer. During application setup, custom message types are registered with the `probe` package codec during `ChainClient` creation. This process is handled in the setup in the following manner: | ||
|
|
||
| 1. During the setup of the Indexer in the [cosmos-indexer/cmd package index.go file](https://github.com/DefiantLabs/cosmos-indexer/blob/main/cmd/index.go#L200-L201), the `GetProbeClient` function is called with the registered custom message types. | ||
| 2. The `GetProbeClient` function in the [cosmos-indexer/probe package probe.go file](https://github.com/DefiantLabs/cosmos-indexer/blob/main/probe/probe.go#L10) creates a `ChainClientConfig` with the custom message types registered | ||
| 3. The `ChainClientConfig` is passed to the `NewChainClient` function in the [probe/client package client.go file](https://github.com/DefiantLabs/probe/blob/main/client/client.go#L28) | ||
| 4. The `ChainClient` is created with the custom message types registered with the codec during the `MakeCodec` function in the [probe client encoding.go file](https://github.com/DefiantLabs/probe/blob/main/client/encoding.go#L30) `MakeCodec` function. |
24 changes: 23 additions & 1 deletion
24
docs/reference/custom_cosmos_module_extensions/probe_codec_walkthrough.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,25 @@ | ||
| # Probe Codec Walkthrough | ||
|
|
||
| Coming Soon | ||
| The Cosmos Indexer uses the [probe](https://github.com/DefiantLabs/probe) package for Cosmos SDK protobuf codec management, RPC client generation, and blockchain RPC data querying/processing. | ||
|
|
||
| ## Cosmos SDK Codec Management | ||
|
|
||
| The Cosmos SDK uses Protobuf for encoding and decoding transaction messages, as well as other blockchain data returned in RPC responses. The `probe` package provides a codec for decoding JSON RPC responses and their Protobuf encoded transaction messages. Types are registered with the codec to allow for decoding and encoding of Protobuf messages to Go types at runtime. | ||
|
|
||
| This allows data to be passed from the blockchain to the `probe` package in a format that can be decoded into Go types, allowing for easy processing and indexing of blockchain data. | ||
|
|
||
| ## RPC Client Generation | ||
|
|
||
| The `probe` package also provides a client generator that can be used to generate a client for a specific blockchain. The client provides methods for querying the blockchain for data, such as blocks, transactions, and events. | ||
|
|
||
| The `ChainClient` type defined in the [client package client.go file](https://github.com/DefiantLabs/probe/blob/main/client/client.go) is used to generate the client for a specific blockchain. The client is generated using the `NewChainClient` function, which takes a `ChainClientConfig` type as an argument that contains the configuration for the client, such as RPC endpoint, chain ID and others. | ||
|
|
||
| ## Blockchain RPC Data Querying/Processing | ||
|
|
||
| The `ChainClient` type is attached to a `Query` type defined in the [query package query.go file](https://github.com/DefiantLabs/probe/blob/main/query/query.go) that provides methods for querying the blockchain for data. During JSON RPC response decoding, the codec is used to decode the response into Go types where appropriate. | ||
|
|
||
| ## Probe Interface Registry | ||
|
|
||
| The `probe` package provides an interface registry that allows for registering custom message types with the codec. This registry provides methods for registering custom message types with the codec, allowing for decoding and encoding of custom message types to Go types at runtime by type URL. | ||
|
|
||
| There are two main ways to register custom message types with the codec, using the Cosmos SDK `AppModuleBasics` interface to register an entire Cosmos SDK module with the codec, or by using the custom message type URL tied to an underlying type. See the [Custom Message Type Registration](./custom_message_type_registration.md) docs for more details. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| SELECT b.height, tx.hash, met.message_type from txes tx | ||
| JOIN messages me on me.tx_id = tx.id | ||
| JOIN blocks b on b.id = tx.block_id | ||
| JOIN message_types met on met.id = me.message_type_id; | ||
| JOIN message_types met on met.id = me.message_type_id; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,4 +10,4 @@ | |
| "message_type": "/sdk.auction.v1.MsgAuctionBid" | ||
| } | ||
| ] | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters