From be828783486fec07b4c826dd7dbfe99511cc8352 Mon Sep 17 00:00:00 2001 From: Max Efremov <51917427+mefrem@users.noreply.github.com> Date: Thu, 10 Aug 2023 11:55:47 -0500 Subject: [PATCH 1/5] Updated "chainhooks service with bitcoind" doc --- ...n-chainhook-as-a-service-using-bitcoind.md | 234 ++++-------------- 1 file changed, 53 insertions(+), 181 deletions(-) diff --git a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md index b4c9a9f5b..5f9378b02 100644 --- a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md +++ b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md @@ -8,12 +8,17 @@ You can run Chainhook as a service to evaluate Bitcoin against your predicates. ### Setting up a Bitcoin Node -Install bitcoind using [this](https://bitcoin.org/en/bitcoin-core/) link. Ingesting blocks using bitcoind happens through zeromq, an embedded networking library in the bitcoind installation package. +Bitcoind is a program that implements the Bitcoin protocol for remote procedure call (RPC) use. Chainhook can be set up to interact with the Bitcoin chainstate through bitcoind's ZeroMQ interface, its embedded networking library. -Bitcoind installation will download binaries in a zip format, `bitcoin-22.0-osx64.tar.gz`. You can extract the zip file to view the folders. Expand the `bin` folder to see the bitcoind executable files. +This guide is written to work with the latest Bitcoin Core software containing bitcoind, [Bitcoin Core 25.0](https://bitcoincore.org/bin/bitcoin-core-25.0/). + +> **_NOTE:_** +> +> While bitcoind can and will start syncing a Bitcoin node, customizing this node to your use cases beyond supporting a Chainhook is out of scope for this guide. See the Bitcoin wiki for [bitcoind](https://en.bitcoin.it/wiki/Bitcoind), ["Running Bitcoin"](https://en.bitcoin.it/wiki/Running_Bitcoin), or bitcoin.org's [Running A Full Node guide](https://bitcoin.org/en/full-node). - Navigate to your project folder, create a new file, and rename it to `bitcoin.conf` on your local machine. Copy the below configuration to the `bitcoin.conf` file. -- Get the downloaded path of the bitcoind from the [prerequisites section](#prerequisites) and use it in the `datadir` configuration below. +- a) if you already have a bitcoin node, or b) you are syncing and starting a node from scratch. +- The Chainhook will scan against bitcoin blockchain data. Copy the path of your Bitcoin directory to the `bitcoin.conf`'s `datadir` option. See the Bitcoin wiki for the [list of default directories by operating system](https://en.bitcoin.it/wiki/Data_directory) - Set a username of your choice for bitcoind and use it in the `rpcuser` configuration below. - Set a password of your choice for bitcoind and use it in the `rpcpassword` configuration below. @@ -22,10 +27,12 @@ Bitcoind installation will download binaries in a zip format, `bitcoin-22.0-osx6 > Make a note of the `rpcuser`, `rpcpassword` and `rpcport` values to use them later in the chainhook configuration. ```conf -datadir= +# Bitcoin Core Configuration + +datadir= # Path to your Bitcoin folder server=1 -rpcuser="bitcoind_username" # You can set the username here -rpcpassword="bitcoind_password" # You can set the password here +rpcuser=bitcoind_username # You can set the username here +rpcpassword=bitcoind_password # You can set the password here rpcport=8332 # You can set your localhost port number here rpcallowip=0.0.0.0/0 rpcallowip=::/0 @@ -51,13 +58,13 @@ In the command below, use the path to your `bitcoin.conf` file from your machine > **_NOTE:_** > -> The below command is a startup process that might take a few hours to run. +> The below command is a startup process that, if this is your first time syncing a node, might take a few hours to a few days to run. ```console -./bitcoind -conf=/bitcoin.conf +./bitcoind -conf= ``` -Once the above command runs, you will see `zmq_url` entries in the output, enabling zeromq. +Once the above command runs, you will see `zmq_url` entries in the output, enabling ZeroMQ. ### Configure Chainhook @@ -106,67 +113,65 @@ tsv_file_url = "https://archive.hiro.so/mainnet/stacks-blockchain-api/mainnet-st In the `Chainhook.toml` file that gets generated, you'll need to match the network parameters to the `bitcoin.config` that was generated earlier in [this](#setting-up-a-bitcoin-node) section to allow Chainhook to connect to the bitcoin layer. -The Bitcoin node is exposing the rpc endpoints. To protect the endpoints, we are using rpc username and password fields. To run Chainhook as a service using Bitcoin, you must match the rpc endpoints username, password, and network ports. +The Bitcoin node is exposing the RPC endpoints. To protect the endpoints, we are using RPC username and password fields. To run Chainhook as a service using Bitcoin, you must match the RPC endpoints username, password, and network ports. As such, some configurations must match Chainhook and the bitcoind node. In the `Chainhook.toml`, update the following network parameters to match the `bitcoin.conf`: -| bitcoin.conf | Chainhook.toml | -| ----------- | ----------- | -| rpcuser | bitcoind_rpc_username | -| rpcpassword | bitcoind_rpc_password | -| rpcport | bitcoind_rpc_url | -| zmqpubhashblock | bitcoind_zmq_url | +| bitcoin.conf | Chainhook.toml | +| --------------- | --------------------- | +| rpcuser | bitcoind_rpc_username | +| rpcpassword | bitcoind_rpc_password | +| rpcport | bitcoind_rpc_url | +| zmqpubhashblock | bitcoind_zmq_url | In the `Chainhook.toml` file, - Update the `bitcoind_rpc_username` to use the username set for `rpcuser` earlier. -- Update the `bitcoind_rpc_password` to use the password set for `rpcpassword` earlier. +- Update the `bitcoind_rpc_password` to use the password set for `rpcpassword` earlier. - Update the `bitcoind_rpc_url` to use the same host and port for the `rpcport` earlier. - Next, update the `bitcoind_zmq_url` to use the same host and port for the `zmqpubhashblock` that was set earlier. ## Scan blockchain based on predicates -Now that your bitcoind and Chainhook configurations are done, you can scan your blocks by defining your [predicates](../overview.md#if-this-predicate-design). This section helps you with an example JSON file to scan a range of blocks in the blockchain and render the results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). - -The following are the two examples to walk you through `file_append` and `http_post` `then-that` predicate designs. +Now that your bitcoind and Chainhook configurations are done, you can scan your blocks by defining your [predicates](../overview.md#if-this-predicate-design). This section helps you with an example JSON file to scan a range of blocks in the blockchain to trigger results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). -Example 1 uses the `ordinals.json` file to scan the predicates and render results using `file_append`. -Example 2 uses the `ordinals_protocol.json` to scan the predicates and render results using `http_post`. +The following is an example to walk you through `file_append` `then-that` predicate design. -You can choose between the following examples to scan the predicates. +The example collects bitcoin transactions from a particular address, specifically the bitcoin address associated with a Stacking pool, [Friedgar Pool](https://pool.friedger.de/). This address has been collecting payouts from Stacks miners since cycle 55. We are scanning a portion of the bitcoin blockchain to capture the last few of these payouts (to shorten predicate scanning example). ### Example 1 Run the following command in your terminal to generate a sample JSON file with predicates. ```console -chainhook predicates new ordinals.json --bitcoin +touch stacking.json ``` -A JSON file `ordinals.json` is generated. +Paste the following contents into the `stacking.json` file. ```json { - "uuid": "1", - "name": "Hello Ordinals", - "chain": "bitcoin", - "version": 1, - "networks": { - "testnet": { - "start_block": 777534, - "end_block": 777540, - "if_this": { - "scope": "ordinals_protocol", - "operation": "inscription_feed" - }, - "then_that": { - "file_append": { - "path": "inscription_feed.txt" - } - } + "chain": "bitcoin", + "uuid": "13b3fce6-eace-4552-a2f6-7672cd94cf7e", + "name": "Friedgar's Stacking Pool", + "version": 1, + "networks": { + "mainnet": { + "start_block": 800000, + "end_block": 802000, + "if_this": { + "scope": "outputs", + "p2wpkh": { + "equals": "bc1qs0kkdpsrzh3ngqgth7mkavlwlzr7lms2zv3wxe" + } + }, + "then_that": { + "file_append": { + "path": "btc-transactions.txt" } - + } } + } } ``` @@ -174,156 +179,23 @@ A JSON file `ordinals.json` is generated. > > You can get blockchain height and current block by referring to https://explorer.hiro.so/blocks?chain=mainnet -Now, use the following command to scan the blocks based on the predicates defined in the `ordinals.json` file. +Now, use the following command to scan the blocks based on the predicates defined in the `stacking.json` file. ```console -chainhook predicates scan ordinals.json --config-path=./Chainhook.toml +chainhook predicates scan stacking.json --config-path=./Chainhook.toml ``` -The output of the above command will be a text file `inscription_feed.txt` generated based on the predicate definition. +The output of the above command will be a text file `btc-transactions.txt` generated based on the predicate definition. > **_TIP:_** > > To optimize your experience with scanning, the following are a few knobs you can play with: +> > - Use of adequate values for `start_block` and `end_block` in predicates will drastically improve the performance. > - Reducing the number of network hops between the Chainhook and the bitcoind processes can also help, so your network setup can play a major role in performance. -### Example 2 - -Run the following command to generate a sample JSON file with predicates in your terminal. - -```console -chainhook predicates new ordinals_protocol.json --bitcoin -``` - -A JSON file `ordinals_protocol.json` is generated. You can now edit the JSON based on the available predicates for Bitcoin. To understand the available predicates, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). - -```json -{ - "uuid": "1", - "name": "Hello Ordinals", - "chain": "bitcoin", - "version": 1, - "networks": { - "testnet": { - "if_this": { - "scope": "ordinals_protocol", - "operation": "inscription_feed" - }, - "then_that": { - "http_post": { - "url": "http://localhost:3000/events", - "authorization_header": "123904" - } - }, - "start_block": 777534, - } - - } -} -``` - -> **_NOTE:_** -> -> The `start_block` is the required field to use the `http_post` `then-that` predicate. - -Now, use the following command to scan the blocks based on the predicates defined in the `ordinals_protocol.json` file. - -```console -chainhook predicates scan ordinals_protocol.json --config-path=./Chainhook.toml -``` - -The above command posts events to the URL, `http://localhost:3000/events` mentioned in the JSON file. - -## Initiate Chainhook Service - -In this section, you'll learn how to initiate the chainhook service using the following two ways and use the REST API call to post the events onto a server. - -- Initiate the chainhook service by passing the predicate path to the command as shown below. - - ```console - chainhook service start --predicate-path=ordinals_protocol.json --config-path=Chainhook.toml - ``` - - The above command registers the predicates based on the predicate definition in the `ordinals_protocol.json` file. - -- You can also dynamically register predicates via the predicate registration server. To do this: - - Uncomment the following lines of code in the `Chainhook.toml` file to enable the predicate registration server. - ``` - [http_api] - http_port = 20456 - database_uri = "redis://localhost:6379/" - ``` - - Start the Chainhook service by running ```chainhook service start --config-path=Chainhook.toml```. - - Now, the predicate registration server is running at `localhost:20456`. To dynamically register a new predicate, send a POST request to `localhost:20456/v1/chainhooks` with the new predicate, in JSON format, included in the request body. For complete documentation on the API endpoints available, see the [OpenAPI](https://raw.githubusercontent.com/hirosystems/chainhook/develop/docs/chainhook-openapi.json) specification. - - ![Example post request](../images/chainhook-post-request.jpeg) - -The sample payload response should look like this: - -```jsonc -{ - "chainhook": { - "predicate": { - "operation": "inscription_feed", - "scope": "ordinals_protocol" - }, - "uuid": "1" - }, - "apply": [{ - "block_identifier": { - "hash": "0x00000000000000000003e3e2ffd3baaff2cddda7d12e84ed0ffe6f7778e988d4", - "index": 777534 - }, - "metadata": {}, - "parent_block_identifier": { - "hash": "0x0000000000000000000463a1034c59e6dc94c7e52855582af11882743b86e2a7", - "index": 777533 - }, - "timestamp": 1676923039, - "transactions": [{ - "transaction_identifier": { - "hash": "0xca20efe5e4d71c16cd9b8dfe4d969efdd225ef0a26136a6a4409cb3afb2e013e" - }, - "metadata": { - "ordinal_operations": [{ - "inscription_revealed": { - "content_bytes": "", - "content_length": 12293, - "content_type": "image/jpeg", - "inscriber_address": "bc1punnjva5ayg84kf5tmvx265uwvp8py3ux24skz43aycj5rzdgzjfq0jxsuc", - "inscription_fee": 64520, - "inscription_id": "ca20efe5e4d71c16cd9b8dfe4d969efdd225ef0a26136a6a4409cb3afb2e013ei0", - "inscription_number": 0, - "inscription_output_value": 10000, - "ordinal_block_height": 543164, - "ordinal_number": 1728956147664701, - "ordinal_offset": 1147664701, - "satpoint_post_inscription": "ca20efe5e4d71c16cd9b8dfe4d969efdd225ef0a26136a6a4409cb3afb2e013e:0:0", - "transfers_pre_inscription": 0 - } - }], - "proof": null - }, - "operations": [] - // Other transactions - }] - }], - "rollback": [], -} -``` - -Understand the output of the above JSON file with the following details. - -- The `apply` payload includes the block header and the transactions that triggered the predicate. - -- The `rollback` payload includes the block header and the transactions that triggered the predicate for a past block that is no longer part of the canonical chain and must be reverted. - -> **_TIP:_** -> -> You can also run chainhook service by passing multiple predicates. -> Example: ```chainhook service start --predicate-path=predicate_1.json --predicate-path=predicate_2.json --config-path=Chainhook.toml``` ## References - To learn more about Ordinals, refer to [Introducing Ordinals Explorer and Ordinals API](https://www.hiro.so/blog/introducing-the-ordinals-explorer-and-ordinals-api). -- The [OpenAPI specification for chainhook](https://raw.githubusercontent.com/hirosystems/chainhook/develop/docs/chainhook-openapi.json) is available to understand the scope of chainhook. +- The [OpenAPI specification for chainhook](https://raw.githubusercontent.com/hirosystems/chainhook/develop/docs/chainhook-openapi.json) is available to understand the scope of chainhook. \ No newline at end of file From c44afef2371017c6ad097ae27e13950f859acb91 Mon Sep 17 00:00:00 2001 From: Max Efremov <51917427+mefrem@users.noreply.github.com> Date: Thu, 10 Aug 2023 12:43:51 -0500 Subject: [PATCH 2/5] Update how-to-run-chainhook-as-a-service-using-bitcoind.md Some typos and language changes. --- ...n-chainhook-as-a-service-using-bitcoind.md | 35 ++++++++----------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md index 5f9378b02..e7acd2da4 100644 --- a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md +++ b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md @@ -14,11 +14,10 @@ This guide is written to work with the latest Bitcoin Core software containing b > **_NOTE:_** > -> While bitcoind can and will start syncing a Bitcoin node, customizing this node to your use cases beyond supporting a Chainhook is out of scope for this guide. See the Bitcoin wiki for [bitcoind](https://en.bitcoin.it/wiki/Bitcoind), ["Running Bitcoin"](https://en.bitcoin.it/wiki/Running_Bitcoin), or bitcoin.org's [Running A Full Node guide](https://bitcoin.org/en/full-node). +> While bitcoind can and will start syncing a Bitcoin node, customizing this node to your use cases beyond supporting a Chainhook is out of scope for this guide. See the Bitcoin wiki for ["Running Bitcoin"](https://en.bitcoin.it/wiki/Running_Bitcoin) or bitcoin.org's [Running A Full Node guide](https://bitcoin.org/en/full-node). - Navigate to your project folder, create a new file, and rename it to `bitcoin.conf` on your local machine. Copy the below configuration to the `bitcoin.conf` file. -- a) if you already have a bitcoin node, or b) you are syncing and starting a node from scratch. -- The Chainhook will scan against bitcoin blockchain data. Copy the path of your Bitcoin directory to the `bitcoin.conf`'s `datadir` option. See the Bitcoin wiki for the [list of default directories by operating system](https://en.bitcoin.it/wiki/Data_directory) +- The Chainhook will scan against bitcoin blockchain data. Copy the path of your Bitcoin directory to the `bitcoin.conf`'s `datadir` field. See the Bitcoin wiki for the [list of default directories by operating system](https://en.bitcoin.it/wiki/Data_directory) - Set a username of your choice for bitcoind and use it in the `rpcuser` configuration below. - Set a password of your choice for bitcoind and use it in the `rpcpassword` configuration below. @@ -29,10 +28,10 @@ This guide is written to work with the latest Bitcoin Core software containing b ```conf # Bitcoin Core Configuration -datadir= # Path to your Bitcoin folder +datadir= # Path to existing Bitcoin folder. New data directory will be created here otherwise server=1 -rpcuser=bitcoind_username # You can set the username here -rpcpassword=bitcoind_password # You can set the password here +rpcuser=devnet # You can set the username here +rpcpassword=devnet # You can set the password here rpcport=8332 # You can set your localhost port number here rpcallowip=0.0.0.0/0 rpcallowip=::/0 @@ -58,7 +57,7 @@ In the command below, use the path to your `bitcoin.conf` file from your machine > **_NOTE:_** > -> The below command is a startup process that, if this is your first time syncing a node, might take a few hours to a few days to run. +> The below command is a startup process that, if this is your first time syncing a node, might take a few hours to a few days to run. Alternatively, if the directory pointed to in the `datadir` field above contains bitcoin blockchain data, syncing will resume. ```console ./bitcoind -conf= @@ -73,7 +72,7 @@ In this section, you will configure chainhook to match the network configuration Next, you will generate a `Chainhook.toml` file to connect Chainhook with your bitcoind node. Navigate to the directory where you want to generate the `Chainhook.toml` file and use the following command in your terminal: ```console -chainhook config generate --testnet +chainhook config generate --mainnet ``` The following `Chainhook.toml` file should be generated: @@ -111,11 +110,12 @@ tsv_file_url = "https://archive.hiro.so/mainnet/stacks-blockchain-api/mainnet-st ``` -In the `Chainhook.toml` file that gets generated, you'll need to match the network parameters to the `bitcoin.config` that was generated earlier in [this](#setting-up-a-bitcoin-node) section to allow Chainhook to connect to the bitcoin layer. +Several of the network parameters in the generated `Chainhook.toml` configuration file need to match the network parameters contained in the `bitcoin.conf` that was created earlier in the [Setting up a Bitcoin Node](#setting-up-a-bitcoin-node) section: -The Bitcoin node is exposing the RPC endpoints. To protect the endpoints, we are using RPC username and password fields. To run Chainhook as a service using Bitcoin, you must match the RPC endpoints username, password, and network ports. - -As such, some configurations must match Chainhook and the bitcoind node. In the `Chainhook.toml`, update the following network parameters to match the `bitcoin.conf`: +- Update the `bitcoind_rpc_username` to use the username set for `rpcuser` earlier. +- Update the `bitcoind_rpc_password` to use the password set for `rpcpassword` earlier. +- Update the `bitcoind_rpc_url` to use the same host and port for the `rpcport` earlier. +- Next, update the `bitcoind_zmq_url` to use the same host and port for the `zmqpubhashblock` that was set earlier. | bitcoin.conf | Chainhook.toml | | --------------- | --------------------- | @@ -124,20 +124,15 @@ As such, some configurations must match Chainhook and the bitcoind node. In the | rpcport | bitcoind_rpc_url | | zmqpubhashblock | bitcoind_zmq_url | -In the `Chainhook.toml` file, -- Update the `bitcoind_rpc_username` to use the username set for `rpcuser` earlier. -- Update the `bitcoind_rpc_password` to use the password set for `rpcpassword` earlier. -- Update the `bitcoind_rpc_url` to use the same host and port for the `rpcport` earlier. -- Next, update the `bitcoind_zmq_url` to use the same host and port for the `zmqpubhashblock` that was set earlier. ## Scan blockchain based on predicates -Now that your bitcoind and Chainhook configurations are done, you can scan your blocks by defining your [predicates](../overview.md#if-this-predicate-design). This section helps you with an example JSON file to scan a range of blocks in the blockchain to trigger results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). +Now that your bitcoind and Chainhook configurations are complete, you can define the [predicates](../overview.md#if-this-predicate-design) you would like to scan against bitcoin blocks [predicates](../overview.md#if-this-predicate-design). These predicates are where the user specifies the kinds of blockchain events they want their Chainhook to trigger an action. This section helps you with an example JSON file to scan a range of blocks in the blockchain to trigger results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). The following is an example to walk you through `file_append` `then-that` predicate design. -The example collects bitcoin transactions from a particular address, specifically the bitcoin address associated with a Stacking pool, [Friedgar Pool](https://pool.friedger.de/). This address has been collecting payouts from Stacks miners since cycle 55. We are scanning a portion of the bitcoin blockchain to capture the last few of these payouts (to shorten predicate scanning example). +The example collects bitcoin transactions from a particular address, specifically the bitcoin address associated with a Stacking pool, [Friedgar Pool](https://pool.friedger.de/). This address has been collecting payouts from Stacks miners since cycle 55. We are scanning a portion of the bitcoin blockchain to capture the last few of these payouts (to shorten predicate scanning for example purposes). ### Example 1 @@ -198,4 +193,4 @@ The output of the above command will be a text file `btc-transactions.txt` gener ## References - To learn more about Ordinals, refer to [Introducing Ordinals Explorer and Ordinals API](https://www.hiro.so/blog/introducing-the-ordinals-explorer-and-ordinals-api). -- The [OpenAPI specification for chainhook](https://raw.githubusercontent.com/hirosystems/chainhook/develop/docs/chainhook-openapi.json) is available to understand the scope of chainhook. \ No newline at end of file +- The [OpenAPI specification for chainhook](https://raw.githubusercontent.com/hirosystems/chainhook/develop/docs/chainhook-openapi.json) is available to understand the scope of chainhook. From 2c9c54608a411bce4fc2e172d8817955c8de9b17 Mon Sep 17 00:00:00 2001 From: Max Efremov <51917427+mefrem@users.noreply.github.com> Date: Thu, 10 Aug 2023 14:19:38 -0500 Subject: [PATCH 3/5] Update how-to-run-chainhook-as-a-service-using-bitcoind.md Hand off to @ryanwaits --- ...n-chainhook-as-a-service-using-bitcoind.md | 35 +++++++++++-------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md index e7acd2da4..999a8803d 100644 --- a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md +++ b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md @@ -30,9 +30,9 @@ This guide is written to work with the latest Bitcoin Core software containing b datadir= # Path to existing Bitcoin folder. New data directory will be created here otherwise server=1 -rpcuser=devnet # You can set the username here -rpcpassword=devnet # You can set the password here -rpcport=8332 # You can set your localhost port number here +rpcuser=devnet +rpcpassword=devnet +rpcport=8332 rpcallowip=0.0.0.0/0 rpcallowip=::/0 txindex=1 @@ -81,20 +81,26 @@ The following `Chainhook.toml` file should be generated: [storage] working_dir = "cache" -# The Http Api allows you to register/deregister +# The Http Api allows you to register / deregister # dynamically predicates. # Disable by default. # -[http_api] -http_port = 20456 -database_uri = "redis://localhost:6379/" +# [http_api] +# http_port = 20456 +# database_uri = "redis://localhost:6379/" [network] -mode = "testnet" -bitcoind_rpc_url = "http://localhost:8332" # Must match the rpcport in the bitcoin.conf -bitcoind_rpc_username = "" # Must match the rpcuser in the bitcoin.conf -bitcoind_rpc_password = "" # Must match the rpcpassword in the bitcoin.conf -stacks_node_rpc_url = "http://localhost:20443" +mode = "mainnet" +bitcoind_rpc_url = "http://localhost:8332" +bitcoind_rpc_username = "devnet" +bitcoind_rpc_password = "devnet" +# Bitcoin block events can be received by Chainhook +# either through a Bitcoin node's ZeroMQ interface, +# or through the Stacks node. The Stacks node is +# used by default: +# stacks_node_rpc_url = "http://localhost:20443" +# but zmq can be used instead: +bitcoind_zmq_url = "tcp://0.0.0.0:18543" [limits] max_number_of_bitcoin_predicates = 100 @@ -107,7 +113,6 @@ max_caching_memory_size_mb = 32000 [[event_source]] tsv_file_url = "https://archive.hiro.so/mainnet/stacks-blockchain-api/mainnet-stacks-blockchain-api-latest" - ``` Several of the network parameters in the generated `Chainhook.toml` configuration file need to match the network parameters contained in the `bitcoin.conf` that was created earlier in the [Setting up a Bitcoin Node](#setting-up-a-bitcoin-node) section: @@ -115,7 +120,8 @@ Several of the network parameters in the generated `Chainhook.toml` configuratio - Update the `bitcoind_rpc_username` to use the username set for `rpcuser` earlier. - Update the `bitcoind_rpc_password` to use the password set for `rpcpassword` earlier. - Update the `bitcoind_rpc_url` to use the same host and port for the `rpcport` earlier. -- Next, update the `bitcoind_zmq_url` to use the same host and port for the `zmqpubhashblock` that was set earlier. +- Make sure this line is commented out (and therefore inactive): `stacks_node_rpc_url = "http://localhost:20443"` +- Next, uncomment and update the `bitcoind_zmq_url` to use the same host and port for the `zmqpubhashblock` that was set earlier. | bitcoin.conf | Chainhook.toml | | --------------- | --------------------- | @@ -125,7 +131,6 @@ Several of the network parameters in the generated `Chainhook.toml` configuratio | zmqpubhashblock | bitcoind_zmq_url | - ## Scan blockchain based on predicates Now that your bitcoind and Chainhook configurations are complete, you can define the [predicates](../overview.md#if-this-predicate-design) you would like to scan against bitcoin blocks [predicates](../overview.md#if-this-predicate-design). These predicates are where the user specifies the kinds of blockchain events they want their Chainhook to trigger an action. This section helps you with an example JSON file to scan a range of blocks in the blockchain to trigger results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). From 9379505dddc69c7c9158a80aa284f736ad9646e1 Mon Sep 17 00:00:00 2001 From: Ryan Waits Date: Thu, 10 Aug 2023 19:39:43 -0500 Subject: [PATCH 4/5] add additional examples and service sections back / update copy and examples --- ...n-chainhook-as-a-service-using-bitcoind.md | 237 +++++++++++++++--- 1 file changed, 206 insertions(+), 31 deletions(-) diff --git a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md index 999a8803d..007d21f22 100644 --- a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md +++ b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md @@ -14,7 +14,7 @@ This guide is written to work with the latest Bitcoin Core software containing b > **_NOTE:_** > -> While bitcoind can and will start syncing a Bitcoin node, customizing this node to your use cases beyond supporting a Chainhook is out of scope for this guide. See the Bitcoin wiki for ["Running Bitcoin"](https://en.bitcoin.it/wiki/Running_Bitcoin) or bitcoin.org's [Running A Full Node guide](https://bitcoin.org/en/full-node). +> While bitcoind can and will start syncing a Bitcoin node, customizing this node to your use cases beyond supporting a Chainhook is out of scope for this guide. See the Bitcoin wiki for ["Running Bitcoin"](https://en.bitcoin.it/wiki/Running_Bitcoin) or bitcoin.org [Running A Full Node guide](https://bitcoin.org/en/full-node). - Navigate to your project folder, create a new file, and rename it to `bitcoin.conf` on your local machine. Copy the below configuration to the `bitcoin.conf` file. - The Chainhook will scan against bitcoin blockchain data. Copy the path of your Bitcoin directory to the `bitcoin.conf`'s `datadir` field. See the Bitcoin wiki for the [list of default directories by operating system](https://en.bitcoin.it/wiki/Data_directory) @@ -75,7 +75,13 @@ Next, you will generate a `Chainhook.toml` file to connect Chainhook with your b chainhook config generate --mainnet ``` -The following `Chainhook.toml` file should be generated: +Several network parameters in the generated `Chainhook.toml` configuration file need to match those in the `bitcoin.conf` file created earlier in the [Setting up a Bitcoin Node](#setting-up-a-bitcoin-node) section. Please update the following parameters accordingly: + +1. Update `bitcoind_rpc_username` with the username set for `rpcuser` in `bitcoin.conf`. +2. Update `bitcoind_rpc_password` with the password set for `rpcpassword` in `bitcoin.conf`. +3. Update `bitcoind_rpc_url` with the same host and port used for `rpcport` in `bitcoin.conf`. + +Additionally, if you want to receive events from the configured Bitcoin node, substitute `stacks_node_rpc_url` with `bitcoind_zmq_url`, as follows: ```toml [storage] @@ -115,14 +121,6 @@ max_caching_memory_size_mb = 32000 tsv_file_url = "https://archive.hiro.so/mainnet/stacks-blockchain-api/mainnet-stacks-blockchain-api-latest" ``` -Several of the network parameters in the generated `Chainhook.toml` configuration file need to match the network parameters contained in the `bitcoin.conf` that was created earlier in the [Setting up a Bitcoin Node](#setting-up-a-bitcoin-node) section: - -- Update the `bitcoind_rpc_username` to use the username set for `rpcuser` earlier. -- Update the `bitcoind_rpc_password` to use the password set for `rpcpassword` earlier. -- Update the `bitcoind_rpc_url` to use the same host and port for the `rpcport` earlier. -- Make sure this line is commented out (and therefore inactive): `stacks_node_rpc_url = "http://localhost:20443"` -- Next, uncomment and update the `bitcoind_zmq_url` to use the same host and port for the `zmqpubhashblock` that was set earlier. - | bitcoin.conf | Chainhook.toml | | --------------- | --------------------- | | rpcuser | bitcoind_rpc_username | @@ -130,44 +128,41 @@ Several of the network parameters in the generated `Chainhook.toml` configuratio | rpcport | bitcoind_rpc_url | | zmqpubhashblock | bitcoind_zmq_url | - ## Scan blockchain based on predicates Now that your bitcoind and Chainhook configurations are complete, you can define the [predicates](../overview.md#if-this-predicate-design) you would like to scan against bitcoin blocks [predicates](../overview.md#if-this-predicate-design). These predicates are where the user specifies the kinds of blockchain events they want their Chainhook to trigger an action. This section helps you with an example JSON file to scan a range of blocks in the blockchain to trigger results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). -The following is an example to walk you through `file_append` `then-that` predicate design. - -The example collects bitcoin transactions from a particular address, specifically the bitcoin address associated with a Stacking pool, [Friedgar Pool](https://pool.friedger.de/). This address has been collecting payouts from Stacks miners since cycle 55. We are scanning a portion of the bitcoin blockchain to capture the last few of these payouts (to shorten predicate scanning for example purposes). +The following is an example to walk you through an `if_this / then_that` predicate design that appends event payloads to the configured file destination. -### Example 1 +### Example 1 - `file_append` -Run the following command in your terminal to generate a sample JSON file with predicates. +To generate a sample JSON file with predicates, execute the following command in your terminal: ```console -touch stacking.json +chainhook predicates new stacking-pool.json --bitcoin ``` -Paste the following contents into the `stacking.json` file. +Replace the contents of the `stacking-pool.json` file with the following: ```json { "chain": "bitcoin", - "uuid": "13b3fce6-eace-4552-a2f6-7672cd94cf7e", - "name": "Friedgar's Stacking Pool", + "uuid": "1", + "name": "Stacking Pool", "version": 1, "networks": { "mainnet": { - "start_block": 800000, + "start_block": 801500, "end_block": 802000, "if_this": { "scope": "outputs", "p2wpkh": { "equals": "bc1qs0kkdpsrzh3ngqgth7mkavlwlzr7lms2zv3wxe" - } - }, + } + }, "then_that": { "file_append": { - "path": "btc-transactions.txt" + "path": "bitcoin-transactions.txt" } } } @@ -175,25 +170,205 @@ Paste the following contents into the `stacking.json` file. } ``` +This example demonstrates scanning a portion of the Bitcoin blockchain to capture specific outputs from a Bitcoin address associated with a Stacking pool, [Friedgar Pool](https://pool.friedger.de/). + > **_NOTE:_** > > You can get blockchain height and current block by referring to https://explorer.hiro.so/blocks?chain=mainnet -Now, use the following command to scan the blocks based on the predicates defined in the `stacking.json` file. +Now, use the following command to scan the blocks based on the predicates defined in the `stacking-pool.json` file. ```console -chainhook predicates scan stacking.json --config-path=./Chainhook.toml +chainhook predicates scan stacking-pool.json --config-path=./Chainhook.toml ``` -The output of the above command will be a text file `btc-transactions.txt` generated based on the predicate definition. +The output of the above command will be a text file `bitcoin-transactions.txt` generated based on the predicate definition. -> **_TIP:_** +### Example 2 - `http_post` + +Let's generate another sample predicate, this time we are going to send the payload to an API endpoint: + +```console +chainhook predicates new stacking-pool-api.json --bitcoin +``` + +Replace the contents of the `stacking-pool-api.json` file with the following: + +```json +{ + "chain": "bitcoin", + "uuid": "2", + "name": "Stacking Pool (API)", + "version": 1, + "networks": { + "mainnet": { + "start_block": 801500, + "if_this": { + "scope": "outputs", + "p2wpkh": { + "equals": "bc1qs0kkdpsrzh3ngqgth7mkavlwlzr7lms2zv3wxe" + } + }, + "then_that": { + "http_post": { + "url": "http://localhost:3000/events", + "authorization_header": "12345" + } + } + } + } +} +``` + +> **_NOTE:_** > -> To optimize your experience with scanning, the following are a few knobs you can play with: +> The `start_block` is a required field when using the `http_post` `then-that` predicate. + +Once you are finished setting up your endpoint, use the following command to scan the blocks based on the predicates defined in the `stacking-pool-api.json` file. + +```console +chainhook predicates scan stacking-pool-api.json --config-path=./Chainhook.toml +``` + +The above command posts events to the URL, http://localhost:3000/events mentioned in the JSON file. + +## Initiate Chainhook Service + +In this section, you'll learn how to initiate the chainhook service using the following two ways and use the REST API call to post the events onto a server. + +- Initiate the chainhook service by passing the predicate path to the command as shown below. + +``` +chainhook service start --predicate-path=stacking-pool-api.json --config-path=Chainhook.toml +``` + +The above command registers the predicate based on the predicate definition in the `stacking-pool-api.json` file. + +## Dynamically Register Predicates + +You can also dynamically register new predicates with your Chainhook service. + +First, we need to uncomment the following lines of code in the `Chainhook.toml` file to enable the predicate registration server. + +```toml +# ... + +[http_api] +http_port = 20456 +database_uri = "redis://localhost:6379/" + +# ... +``` + +> **_NOTE:_** > -> - Use of adequate values for `start_block` and `end_block` in predicates will drastically improve the performance. -> - Reducing the number of network hops between the Chainhook and the bitcoind processes can also help, so your network setup can play a major role in performance. +> This assumes you have a local instance of [Redis](https://redis.io/docs/getting-started/) running. + +Start the Chainhook service by running the following command: + +``` +chainhook service start --config-path=Chainhook.toml +``` +To dynamically register a new predicate, send a POST request to the running predicate registration server at `localhost:20456/v1/chainhooks`. Include the new predicate in JSON format within the request body. Use the following `curl` command as an example: + +```console +curl -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "chain": "bitcoin", + "uuid": "3", + "name": "Ordinals", + "version": 1, + "networks": { + "mainnet": { + "start_block": 777534, + "if_this": { + "scope": "ordinals_protocol", + "operation": "inscription_feed" + }, + "then_that": { + "http_post": { + "url": "http://localhost:3000/events", + "authorization_header": "12345" + } + } + } + } + }' \ + http://localhost:20456/v1/chainhooks +``` + +The sample response should look like this: + +```jsonc +{ + "chainhook": { + "predicate": { + "operation": "inscription_feed", + "scope": "ordinals_protocol" + }, + "uuid": "1" + }, + "apply": [ + { + "block_identifier": { + "hash": "0x00000000000000000003e3e2ffd3baaff2cddda7d12e84ed0ffe6f7778e988d4", + "index": 777534 + }, + "metadata": {}, + "parent_block_identifier": { + "hash": "0x0000000000000000000463a1034c59e6dc94c7e52855582af11882743b86e2a7", + "index": 777533 + }, + "timestamp": 1676923039, + "transactions": [ + { + "transaction_identifier": { + "hash": "0xca20efe5e4d71c16cd9b8dfe4d969efdd225ef0a26136a6a4409cb3afb2e013e" + }, + "metadata": { + "ordinal_operations": [ + { + "inscription_revealed": { + "content_bytes": "", + "content_length": 12293, + "content_type": "image/jpeg", + "inscriber_address": "bc1punnjva5ayg84kf5tmvx265uwvp8py3ux24skz43aycj5rzdgzjfq0jxsuc", + "inscription_fee": 64520, + "inscription_id": "ca20efe5e4d71c16cd9b8dfe4d969efdd225ef0a26136a6a4409cb3afb2e013ei0", + "inscription_number": 0, + "inscription_output_value": 10000, + "ordinal_block_height": 543164, + "ordinal_number": 1728956147664701, + "ordinal_offset": 1147664701, + "satpoint_post_inscription": "ca20efe5e4d71c16cd9b8dfe4d969efdd225ef0a26136a6a4409cb3afb2e013e:0:0", + "transfers_pre_inscription": 0 + } + } + ], + "proof": null + }, + "operations": [] + // Other transactions + } + ] + } + ], + "rollback": [] +} +``` + +Understand the output of the above JSON file with the following details. + +- The `apply` payload includes the block header and the transactions that triggered the predicate. + +- The `rollback` payload includes the block header and the transactions that triggered the predicate for a past block that is no longer part of the canonical chain and must be reverted. + +> **_TIP:_** +> +> You can also run chainhook service by passing multiple predicates. +> Example: `chainhook service start --predicate-path=predicate_1.json --predicate-path=predicate_2.json --config-path=Chainhook.toml` ## References From 892933f3b8dcd0864a85a6aef5289cc39ff998a2 Mon Sep 17 00:00:00 2001 From: Max Efremov <51917427+mefrem@users.noreply.github.com> Date: Fri, 11 Aug 2023 10:04:31 -0500 Subject: [PATCH 5/5] Update how-to-run-chainhook-as-a-service-using-bitcoind.md Clarifying language to bitcoind guide --- ...n-chainhook-as-a-service-using-bitcoind.md | 27 ++++++++++--------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md index 007d21f22..4a64496c5 100644 --- a/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md +++ b/docs/how-to-guides/how-to-run-chainhook-as-a-service-using-bitcoind.md @@ -2,7 +2,7 @@ title: Run Chainhook as a Service using Bitcoind --- -You can run Chainhook as a service to evaluate Bitcoin against your predicates. You can also dynamically register new predicates by enabling predicates registration API. +You can run Chainhook as a service to evaluate your `if_this / then_that` predicates against the Bitcoin blockchain, delivering results—either file appendations or HTTP POST requests to a server you designate—for your application's use case. You can also dynamically register new predicates as the service is running by enabling the predicates registration API. ## Prerequisites @@ -16,8 +16,8 @@ This guide is written to work with the latest Bitcoin Core software containing b > > While bitcoind can and will start syncing a Bitcoin node, customizing this node to your use cases beyond supporting a Chainhook is out of scope for this guide. See the Bitcoin wiki for ["Running Bitcoin"](https://en.bitcoin.it/wiki/Running_Bitcoin) or bitcoin.org [Running A Full Node guide](https://bitcoin.org/en/full-node). -- Navigate to your project folder, create a new file, and rename it to `bitcoin.conf` on your local machine. Copy the below configuration to the `bitcoin.conf` file. -- The Chainhook will scan against bitcoin blockchain data. Copy the path of your Bitcoin directory to the `bitcoin.conf`'s `datadir` field. See the Bitcoin wiki for the [list of default directories by operating system](https://en.bitcoin.it/wiki/Data_directory) +- Navigate to your project folder, create a new file, and rename it to `bitcoin.conf` on your local machine. Copy the configuration below to the `bitcoin.conf` file. +- Copy the path of your Bitcoin directory to the `bitcoin.conf`'s `datadir` field. See the Bitcoin wiki for the [list of default directories by operating system](https://en.bitcoin.it/wiki/Data_directory) - Set a username of your choice for bitcoind and use it in the `rpcuser` configuration below. - Set a password of your choice for bitcoind and use it in the `rpcpassword` configuration below. @@ -60,14 +60,14 @@ In the command below, use the path to your `bitcoin.conf` file from your machine > The below command is a startup process that, if this is your first time syncing a node, might take a few hours to a few days to run. Alternatively, if the directory pointed to in the `datadir` field above contains bitcoin blockchain data, syncing will resume. ```console -./bitcoind -conf= +./bitcoind -conf= ``` -Once the above command runs, you will see `zmq_url` entries in the output, enabling ZeroMQ. +Once the above command runs, you will see `zmq_url` entries in the console's stdout, displaying ZeroMQ. ### Configure Chainhook -In this section, you will configure chainhook to match the network configurations with the bitcoin config file. First, [install the latest version of chainhook](../getting-started.md#install-chainhook-from-source). +In this section, you will configure Chainhook to match the network configurations with the bitcoin config file. First, [install the latest version of Chainhook](../getting-started.md#install-chainhook-from-source). Next, you will generate a `Chainhook.toml` file to connect Chainhook with your bitcoind node. Navigate to the directory where you want to generate the `Chainhook.toml` file and use the following command in your terminal: @@ -121,6 +121,8 @@ max_caching_memory_size_mb = 32000 tsv_file_url = "https://archive.hiro.so/mainnet/stacks-blockchain-api/mainnet-stacks-blockchain-api-latest" ``` +Here is a table of the relevant parameters this guide changes in our configuration files. + | bitcoin.conf | Chainhook.toml | | --------------- | --------------------- | | rpcuser | bitcoind_rpc_username | @@ -130,7 +132,7 @@ tsv_file_url = "https://archive.hiro.so/mainnet/stacks-blockchain-api/mainnet-st ## Scan blockchain based on predicates -Now that your bitcoind and Chainhook configurations are complete, you can define the [predicates](../overview.md#if-this-predicate-design) you would like to scan against bitcoin blocks [predicates](../overview.md#if-this-predicate-design). These predicates are where the user specifies the kinds of blockchain events they want their Chainhook to trigger an action. This section helps you with an example JSON file to scan a range of blocks in the blockchain to trigger results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). +Now that your bitcoind and Chainhook configurations are complete, you can define the [predicates](../overview.md#if-this-predicate-design) you would like to scan against bitcoin blocks [predicates](../overview.md#if-this-predicate-design). These predicates are where the user specifies the kinds of blockchain events they want their Chainhook to trigger to the deliver a result (either a file appendation or an HTTP POST result). This section helps you with an example JSON file to scan a range of blocks in the blockchain to trigger results. To understand the supported predicates for Bitcoin, refer to [how to use chainhooks with bitcoin](how-to-use-chainhooks-with-bitcoin.md). The following is an example to walk you through an `if_this / then_that` predicate design that appends event payloads to the configured file destination. @@ -222,7 +224,7 @@ Replace the contents of the `stacking-pool-api.json` file with the following: > **_NOTE:_** > -> The `start_block` is a required field when using the `http_post` `then-that` predicate. +> The `start_block` is a required field when using the `http_post` `then_that` predicate. Once you are finished setting up your endpoint, use the following command to scan the blocks based on the predicates defined in the `stacking-pool-api.json` file. @@ -234,11 +236,13 @@ The above command posts events to the URL, http://localhost:3000/events mentione ## Initiate Chainhook Service -In this section, you'll learn how to initiate the chainhook service using the following two ways and use the REST API call to post the events onto a server. +In the examples above, our Chainhook scanned historical blockchain data against the user's predicates and delivered results. In this next section, let's learn how to set up a Chainhook that acts as an ongoing observer and event-streaming service. + +We can start a Chainhook service with an existing predicate. We will also see how we can dynamically register new predicates by making an API call to our Chainhook. In both of these instances, our predicates will be delivering their results to a server set up to recieve results. - Initiate the chainhook service by passing the predicate path to the command as shown below. -``` +```console chainhook service start --predicate-path=stacking-pool-api.json --config-path=Chainhook.toml ``` @@ -362,8 +366,7 @@ The sample response should look like this: Understand the output of the above JSON file with the following details. - The `apply` payload includes the block header and the transactions that triggered the predicate. - -- The `rollback` payload includes the block header and the transactions that triggered the predicate for a past block that is no longer part of the canonical chain and must be reverted. +- The `rollback` payload includes the block header and the transactions that triggered the predicate for a past block that is no longer part of the canonical chain and must be reverted. (Note: This is a chief component of Chainhook's reorg aware functionality, maintaining rollback data for blocks near the chaintip.) > **_TIP:_** >