Added detailed guide about DipDup indexer (#673)

* added detailed guide about DipDup indexer

* fixed some mistakes

* fixed misprint

* fixed notes on PR

---------

Co-authored-by: Joel Willmore <[email protected]>

docs/developers/tooling/data-indexers/dipdup/_category_.json

@@ -0,0 +1,3 @@
+{
+  "label": "DipDup"
+}

docs/developers/tooling/data-indexers/dipdup/overview.md

@@ -0,0 +1,19 @@
+---
+title: Overview
+sidebar_position: 1
+image: /img/socialCards/dipdup.jpg
+---
+
+[DipDup](https://dipdup.io/) is a Python framework for building smart contract indexers. It helps developers focus on business logic instead of writing a boilerplate to store and serve data. DipDup-based indexers are selective, which means only required data is requested. This approach allows to achieve faster indexing times and decreased load on underlying APIs.
+
+## Understanding DipDup
+DipDup is a software framework that helps web3 developers create selective indexers for decentralized applications. It uses blockchain data provided by various off-chain data sources. Some of the key features of DipDup include:
+
+- **Ready For Multichain**: DipDup supports dozens of blockchains, and we are constantly adding new ones. You can easily reuse your business logic for different networks or even index multiple chains in a single project.
+- **Declarative Configuration**: A whole indexer is defined by a single configuration file and a bunch of Python data handlers. Code is completely separated from the configuration and environment variables, making it easy to maintain and deploy your project.
+- **Lots of Integrations**: You can use SQLite, PostgreSQL, or TimescaleDB databases to store blockchain data, deploy to Compose or Swarm with a single command, monitor your indexer with Prometheus or Sentry, and more.
+- **Magic GraphQL API**: DipDup automatically generates a GraphQL API for your data using Hasura, so you can easily query it from your frontend or other services. You can easily extend API with custom queries and metadata requests.
+- **Powerful CLI**: DipDup CLI has everything you need to manage your project, from creating a new one to running and deploying. And it's very convenient. There are lots of templates for various blockchains and use cases, so you can start quickly.
+
+## Explore DipDup
+To learn more about DipDup features, visit the [official DipDup documentation](https://dipdup.io/docs). It offers an in-depth explanation of the framework concepts, lots of examples from basic to the most advanced, allowing rapid and efficient development of blockchain data indexers of any complexity.

docs/developers/tooling/data-indexers/dipdup/quickstart.md

@@ -0,0 +1,143 @@
+---
+title: Quickstart
+sidebar_position: 2
+image: /img/socialCards/dipdup.jpg
+---
+This page will guide you through the steps to get your first DipDup indexer up and running in a few minutes without getting too deep into the details.
+
+Let's create an indexer for output transactions from a specific address. We will need to set up the indexing environment, configure the indexer, and store the results in a database.
+
+## Prerequisites
+Here are a few things you need to get started with DipDup:
+
+- **Skills**: Basic Python 3 knowledge to implement data handlers.
+- **Operating System**: You can use any Linux/macOS distribution on amd64/arm64 platforms with Python installed.
+- **Python Version**: Python 3.11 is required for DipDup. You can check your Python version by running `python3 --version` in your terminal.
+
+## Step 1 — Install DipDup
+The easiest way to install DipDup as a CLI application is [pipx](https://pipx.pypa.io/stable/) with pipx install dipdup command. If you don't want to deal with tooling, we have a convenient installer script. Run the following command in your terminal:
+
+```bash
+curl -Lsf https://dipdup.io/install.py | python3
+```
+
+See the [Installation](https://dipdup.io/docs/installation) page for other options.
+
+## Step 2 — Create a project
+DipDup CLI has a built-in project generator with lots of templates. To create a new project interactively, run the following command:
+
+```bash
+dipdup new
+```
+
+For educational purposes, we'll create a project from scratch, so choose `[none]` network and `demo_blank` template.
+
+Follow the instructions; the project will be created in the new directory.
+
+## Step 3 — Configuration file
+The project root directory contains a bash file named `dipdup.bash`. It's the main configuration file of your indexer. Available options are described in detail on [this page](https://dipdup.io/docs/getting-started/config). For now, just replace its content with the following:
+
+```bash
+spec_version: 2.0
+package: linea
+
+datasources:
+  subsquid:
+    kind: evm.subsquid	
+    url: https://v2.archive.subsquid.io/network/linea-mainnet
+    node: evm_node
+
+  etherscan:
+    kind: abi.etherscan	
+    url: https://api.lineascan.build/api    
+
+  mainnet_node:
+    kind: evm.node	
+    url: https://linea-mainnet.infura.io/v3
+    ws_url: wss://linea-mainnet.infura.io/ws/v3
+
+contracts:
+  some_contract:
+    kind: evm
+    address: 0xa219439258ca9da29e9cc4ce5596924745e12b93
+    typename: not_typed
+
+indexes:
+  evm_index:
+    kind: evm.subsquid.transactions
+    datasource: mainnet_node
+    handlers:
+      - callback: on_output_transaction
+        from: some_contract
+    last_level: 4631
+
+
+database:
+  kind: sqlite
+  path: data/linea.sqlite
+```
+
+Now it's time to generate directories and files required for the project: callbacks stubs, types and other entities we defined in configuration, don't worry in this guide we will only need a small portion of those:
+
+```bash
+dipdup init
+```
+
+You can read more about the structure of the DipDup package [here](https://dipdup.io/docs/getting-started/package).
+
+## Step 4 — Define models and implement data handlers
+In this step, we define the business logic of our application. DipDup supports storing data in SQLite, PostgreSQL and TimescaleDB databases. We use custom ORM based on Tortoise ORM as an abstraction layer.
+
+First, you need to define a model class. Our schema will consist of a single `Transaction` model:
+
+```bash
+from dipdup import fields
+from dipdup.models import Model
+
+
+class Transaction(Model):
+    hash = fields.TextField(pk=True)
+    block_number = fields.IntField()
+    from_ = fields.TextField()
+    to = fields.TextField(null=True)
+
+    created_at = fields.DatetimeField(auto_now_add=True)
+```
+
+
+Our single handler will be responsible for processing output transactions as described in the index definition in config:
+
+```bash
+from dipdup.context import HandlerContext
+from dipdup.models.evm_node import EvmNodeTransactionData
+from dipdup.models.evm_subsquid import SubsquidTransactionData
+
+from linea import models as models
+
+
+async def on_output_transaction(
+    ctx: HandlerContext,
+    transaction: SubsquidTransactionData | EvmNodeTransactionData,
+) -> None:
+    await models.Transaction(
+        hash=transaction.hash,
+        block_number=transaction.block_number,
+        from_=transaction.from_,
+        to=transaction.to,
+    ).save()
+```
+
+## Step 5 — Results
+Time to run the indexer. Processed data will be written to the SQLite file defined in the configuration:
+
+```bash
+dipdup run
+```
+
+DipDup will fetch all the historical data and switch to realtime mode. You can check the progress in the logs.
+
+Query database to see the results:
+
+```bash
+sqlite3 /tmp/linea.sqlite 'SELECT * FROM transaction LIMIT 10'
+```

docusaurus.config.js

@@ -456,6 +456,10 @@ const config = {
             to: "/developers/quickstart",
             from: "/developers/quickstart/goerli-to-sepolia",
           },
+          {
+            to: "/developers/tooling/data-indexers/dipdup/overview",
+            from: "/developers/tooling/data-indexers/dipdup",
+          },
         ],
       },
     ],

project-words.txt

@@ -601,6 +601,7 @@ Pico
 picocolors
 picomatch
 Pimlico
+pipx
 playbtn
 pluggable
 plushie

sidebars.js

@@ -297,10 +297,17 @@ const sidebars = {
             "developers/tooling/data-indexers/arkham",
             "developers/tooling/data-indexers/covalent",
             "developers/tooling/data-indexers/decommas",
-            "developers/tooling/data-indexers/dipdup",
+            {
+              type: "category",
+              label: "DipDup",
+              items: [
+                "developers/tooling/data-indexers/dipdup/overview",
+                "developers/tooling/data-indexers/dipdup/quickstart",   
+              ],
+            },            
             "developers/tooling/data-indexers/dune",
             "developers/tooling/data-indexers/envio",
-            "developers/tooling/data-indexers/flair",
+            "developers/tooling/data-indexers/flair",            
             {
               type: "category",
               label: "Goldsky",