Skip to content
This repository has been archived by the owner on Oct 25, 2024. It is now read-only.

Commit

Permalink
up until 'designing a schema'
Browse files Browse the repository at this point in the history
  • Loading branch information
Rashad Alston authored and Rashad Alston committed Sep 18, 2023
1 parent dc1eeac commit 17bbba0
Show file tree
Hide file tree
Showing 25 changed files with 129 additions and 71 deletions.
4 changes: 2 additions & 2 deletions config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ indexer_net_config: false
# The number of WASM opcodes after which the indexer will stop execution.
metering_points: 30000000000

# Allow the web API to accept raw SQL queries.
# Allow the web server to accept raw SQL queries.
accept_sql_queries: false

# Amount of blocks to return in a request to a Fuel node.
Expand Down Expand Up @@ -64,7 +64,7 @@ web_api:
# Web API port.
port: 29987

# Max body size for web API requests.
# Max body size for web server requests.
max_body_size: "5242880"

# ******************************
Expand Down
4 changes: 2 additions & 2 deletions docs/src/forc-index/start.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ USAGE:
OPTIONS:
--accept-sql-queries
Allow the web API to accept raw SQL queries.
Allow the web server to accept raw SQL queries.
--auth-enabled
Require users to authenticate for some operations.
Expand Down Expand Up @@ -66,7 +66,7 @@ OPTIONS:
Indexer config file.
--max-body-size <MAX_BODY_SIZE>
Max body size for web API requests. [default: 5242880]
Max body size for web server requests. [default: 5242880]
--metering-points <METERING_POINTS>
The number of WASM opcodes after which the indexer's event handler will stop execution.
Expand Down
22 changes: 14 additions & 8 deletions docs/src/getting-started/dependencies.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,26 @@
# Dependencies

> This guide covers some of the basics with regard to installing dependencies for the Fuel indexer service. However, note that this guide is meant to be a general overview for most platforms and by no means covers all platforms. If you're having trouble with dependencies on your system, we recommend that you use `docker`.
> This guide covers some of the basics with regard to installing dependencies for the Fuel indexer service. However, note that this guide is meant to be a general overview for most platforms and by no means covers all platforms.
>
> If you're having trouble with dependencies on your system, we recommend that you use `docker`.
To run the Fuel indexer, you'll need to install a few dependencies on your system:

- The [Rust programming language, with its package manager](https://www.rust-lang.org/tools/install)
- [`fuelup`](#fuelup), the Fuel toolchain manager
- A [PostgresQL](#postgresql) server backend
- The [`wasm32-unknown-unknown`](#web-assembly-wasm) `rustup` target
- [`wasm-snip`](#web-assembly-wasm), a utility for stripping symbols from WebAssemly binaries.
1. The [Rust programming language, with its package manager](https://www.rust-lang.org/tools/install)
2. [`fuelup`](#fuelup), the Fuel toolchain manager
3. A [PostgresQL](#postgresql) server backend
4. The [`wasm32-unknown-unknown`](#web-assembly-wasm) `rustup` target
5. [`wasm-snip`](#web-assembly-wasm), a utility for stripping symbols from WebAssemly binaries.

> If you don't want to install a database directly onto your system, you can use Docker to run it as an isolated container. You can install Docker by following the [install instructions](https://docs.docker.com/get-docker/). For reference purposes, we provide a [`docker compose` file](https://github.com/FuelLabs/fuel-indexer/blob/develop/scripts/docker-compose.yaml) that comes with a PostgresSQL server and a Fuel indexer service.
> If you don't want to install a database directly onto your system, you can use Docker to run a database in an isolated container. You can install Docker by following its [installation instructions](https://docs.docker.com/get-docker/).
>
> For reference purposes, we provide a [`docker compose` file](https://github.com/FuelLabs/fuel-indexer/blob/develop/scripts/docker-compose.yaml) that comes with a PostgresSQL server and a Fuel indexer service.
## `fuelup`

We strongly recommend that you use the Fuel indexer through [`forc`, the Fuel orchestrator](https://fuellabs.github.io/sway/master/book/forc/index.html). You can get `forc` (and other Fuel components) by way of [`fuelup`, the Fuel toolchain manager](https://fuellabs.github.io/fuelup/latest). Install `fuelup` by running the following command, which downloads and runs the installation script.
We strongly recommend that you use the Fuel indexer that's made available via [`forc`, the Fuel orchestrator](https://fuellabs.github.io/sway/master/book/forc/index.html). You can get `forc` (and other Fuel components) by way of [`fuelup`, the Fuel toolchain manager](https://fuellabs.github.io/fuelup/latest).

Install `fuelup` by running the following command, which downloads and runs the installation script.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://install.fuel.network/fuelup-init.sh | sh
Expand Down
71 changes: 48 additions & 23 deletions docs/src/getting-started/indexer-service-infrastructure.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,44 @@
# Indexer Service Infrastructure

- [Service Components](#components)
- [Fuel Indexer Service](#fuel-indexer-service)
- [Starting the service via CLI options](#using-cli-options-indexer-service)
- [Starting the service via a config file](#using-a-configuration-file-indexer-service)
- [Fuel Indexer Web Server](#web-api-server)
- [Starting the service via CLI options](#using-cli-options-web-server)
- [Starting the service via a config file](#using-a-configuration-file-web-server)

A Fuel indexer service instance requires just three components:

- a **Fuel node**: Custom indexers monitor incoming blocks from a Fuel node and extract information about the state of the Fuel blockchain.
- a **Fuel Node**: Custom indexers monitor incoming blocks via a Fuel GraphQL server and extract information about the state of the Fuel blockchain.

- a **PostgresQL database server**: Extracted information is saved into a database.

- a **web server**: dApps can query indexers for up-to-date information and operators can deploy/remove indexers as needed.
- a **Web Server**: dApps can query indexers for up-to-date information and operators can deploy/remove indexers as needed.

---

The Fuel indexer service will connect to any Fuel node, which means you can run your own node or use a node provided by Fuel. The indexer service web API is included with the Fuel indexer; it's available as soon as the indexer is started through `fuel-indexer run`. The only component that isn't provided for you is a Postgres database server. You should set up a server according to your own needs and specifications.

## Components

| Component | Default Host | Default Port | CLI Argument | Environment Variable |
|---|---|---|---|---|
| Fuel Node | localhost | 4000 | `--fuel-node-host` / `--fuel-node-port` | |
| Database Server | localhost | 5432 | `--postgres-user` / `--postgres--password` / `--postgres-host` / `--postgres--port` / `--postgres-database` | |
| Indexer Service Web API | localhost | 29987 | `--web-api-host` / `--web-api-port` | |
| Fuel Node | localhost | 4000 | `--fuel-node-{host,port}` | $FUEL_NODE_{HOST,PORT} |
| Database Server | localhost | 5432 | `--postgres-{username,database,password,host,port}` | $POSTGRES_{USERNAME,DATABASE,PASSWORD,HOST,PORT} |
| Indexer Service Web API | localhost | 29987 | `--web-api-{host,port}` | $WEB_API_{HOST,PORT} |

---

## Fuel Indexer Service

The Fuel indexer service will connect to any Fuel GraphQL server, which means you can run your own node or use a node provided by Fuel. The indexer service web server is included with the Fuel indexer; it's available as soon as the indexer is started through `fuel-indexer run`. The only component that isn't provided for you is a Postgres database server. You should set up a server according to your own needs and specifications.

## Starting the Fuel indexer
> You can start the indexer service with an array of CLI options. Note that most (if not all) of these options include sensible defaults.
### Using CLI options
### Using CLI options (Indexer Service)

```bash
fuel-indexer run --help
```

```text
Standalone binary for the fuel indexer service.
Expand All @@ -32,7 +48,7 @@ USAGE:
OPTIONS:
--accept-sql-queries
Allow the web API to accept raw SQL queries.
Allow the web server to accept raw SQL queries.
--auth-enabled
Require users to authenticate for some operations.
Expand Down Expand Up @@ -84,7 +100,7 @@ OPTIONS:
Indexer config file.
--max-body-size <MAX_BODY_SIZE>
Max body size for web API requests. [default: 5242880]
Max body size for web server requests. [default: 5242880]
--metering-points <METERING_POINTS>
The number of WASM opcodes after which the indexer's event handler will stop execution.
Expand Down Expand Up @@ -141,39 +157,37 @@ OPTIONS:
```

### Using a configuration file
### Using a configuration file (Indexer Service)

```yaml
{{#include ../../../config.yaml}}
```

----

## Web API Server

The `fuel-indexer-api-server` crate of the Fuel indexer contains a standalone web API server that acts as a queryable endpoint on top of the database. Note that the main `fuel-indexer` binary of the indexer project also contains the same web API endpoint.
The `fuel-indexer-api-server` crate of the Fuel indexer contains a standalone web server server that acts as a queryable endpoint on top of the database. Note that the main `fuel-indexer` binary of the indexer project also contains the same web server endpoint.

> The `fuel-indexer-api-server` crate offers a _standalone_ web API endpoint, whereas the API endpoint offered in `fuel-indexer` is bundled with other Fuel indexer functionality (e.g., execution, handling, data-layer construction, etc). Offering the API server as a separate piece allows users to separate components and run them on different systems, if desired.
> The `fuel-indexer-api-server` crate offers a _standalone_ web server endpoint, whereas the API endpoint offered in `fuel-indexer` is bundled with other Fuel indexer functionality (e.g., execution, handling, data-layer construction, etc). Offering the API server as a separate piece allows users to separate components and run them on different systems, if desired.
### Usage
### Using CLI Options (Web Server)

To run the standalone Fuel indexer web API server using a configuration file:
> You can start the indexer service with an array of CLI options. Note that most (if not all) of these options include sensible defaults.
```bash
fuel-indexer-api-server run --config config.yaml
fuel-indexer-api-server run --help
```

In the above example, `config.yaml` is based on [the default service configuration file](https://github.com/FuelLabs/fuel-indexer/blob/develop/config.yaml).

### Options

```text
Fuel indexer web API
Fuel indexer web server
USAGE:
fuel-indexer-api-server run [OPTIONS]
OPTIONS:
--accept-sql-queries
Allow the web API to accept raw SQL queries.
Allow the web server to accept raw SQL queries.
--auth-enabled
Require users to authenticate for some operations.
Expand Down Expand Up @@ -254,3 +268,14 @@ OPTIONS:
--web-api-port <WEB_API_PORT>
Web API port. [default: 29987]
```

### Using A Configuration File (Web Server)

To run the standalone Fuel indexer web server server using a configuration file:

```bash
fuel-indexer-api-server run --config config.yaml
```

In the above example, `config.yaml` is based on [the default service configuration file](https://github.com/FuelLabs/fuel-indexer/blob/develop/config.yaml).

23 changes: 23 additions & 0 deletions docs/src/project-components/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,33 @@ We'll describe these three different use cases below.

The Fuel indexer provides functionality to make it easy to build and compile abitrary indexers by using the [`forc index`](../forc-index/index.md) CLI tool. Using `forc index`, users can create, build, deploy, and remove indexers, as well as authenticate against a running indexer service, and check the status of running indexers.

#### Example

Create, deploy, and check the status of a new indexer.

```bash
forc index new fuel && \
cd fuel && forc index deploy --url http://indexer.fuel.network && \
forc index status --url http://indexer.fuel.network --auth $MY_TOKEN
```

### As a standalone service

You can also start the Fuel indexer as a standalone service that connects to a Fuel node in order to monitor the Fuel blockchain for new blocks and transactions. To do so, run the requisite database migrations, adjust the configuration to connect to a Fuel node, and start the service.

#### Example

Create, deploy, and check the status of a new indexer.

```bash
fuel-indexer run \
--fuel-node-host beta-4.fuel.network \
--fuel-node-port 80 \
--run-migrations \
--accept-sql-queries \
--replace-indexer
```

### As part of a Fuel project

Finally, you can run the Fuel indexer as part of a project that uses other components of the Fuel ecosystem, such as Sway. The convention for a Fuel project layout including an indexer is as follows:
Expand Down
8 changes: 6 additions & 2 deletions docs/src/project-components/module.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,13 @@ mod indexer_mod {
}
```

The first line imports the [prelude](https://docs.rs/fuel-indexer-utils/latest/fuel_indexer_utils/prelude/index.html) from `fuel_indexer_utils`; this allows you to quickly bootstrap an indexer by using common types and traits. Then, we have a module decorated with the `#[indexer]` macro. This macro processes a manifest at the supplied file path, parses your schema and Sway contract ABI (if supplied), and generates code that is combined with handler functions in order to create a complete indexer module.
## What's going on here?

Finally, we have an example handler function. You can define which functions handle different events by using the function parameters. If you add a function parameter of a certain type `T`, the function will be triggered whenever that type is found as part of a block, transaction, or receipt. In this example, let's say that you have a Sway contract with a function that logs a `Greeting` struct. When that function executes as part of a transaction, the logged struct will be included in the data that is processed from the Fuel blockchain. Your indexer module will see the struct and execute `log_the_greeting`.
- The first line imports the [prelude](https://docs.rs/fuel-indexer-utils/latest/fuel_indexer_utils/prelude/index.html) from `fuel_indexer_utils`; this allows you to quickly bootstrap an indexer by using common types and traits. Then, we have a module decorated with the `#[indexer]` macro.
- This macro processes a manifest at the supplied file path, parses your schema and Sway contract ABI (if supplied), and generates code that is combined with handler functions in order to create a complete indexer module.

- Finally, we have an example handler function. You can define which functions handle different events by using the function parameters. If you add a function parameter of a certain type `T`, the function will be triggered whenever that type is found as part of a block, transaction, or receipt.
- In this example, let's say that you have a Sway contract with a function that logs a `Greeting` struct. When that function executes as part of a transaction, the logged struct will be included in the data that is processed from the Fuel blockchain. Your indexer module will see the struct and execute `log_the_greeting`.

> You can learn more about what data can be indexed and find example handlers in the [Indexing Fuel Types](../indexing-fuel-types/index.md) and [Indexing Custom Types](../indexing-custom-types/index.md) sections.
Expand Down
2 changes: 1 addition & 1 deletion docs/src/project-components/schema.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# GraphQL Schema

The GraphQL schema is a required component of the Fuel indexer. When data is indexed into the database, the actual values that are persisted to the database will be values created using the data structures defined in the schema.
The GraphQL schema is a required component of the Fuel indexer. When data is indexed into the database, the actual values that are persisted to the database will be values created using the data structures defined in the GraphQL schema.

Below is a sample GraphQL schema for a Fuel indexer.

Expand Down
2 changes: 1 addition & 1 deletion examples/fuel-explorer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ query {
```

> IMPORTANT: Since this example uses a dockerized indexer service, with the GraphQL
> web API being bound at interface `0.0.0.0` your LAN IP might differ from the
> web server being bound at interface `0.0.0.0` your LAN IP might differ from the
> `192.168.1.34` mentioned above.
>
> On *nix platforms you can typically find your LAN IP via `ifconfig | grep inet`
2 changes: 1 addition & 1 deletion examples/hello-world-native/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ query {
```

> IMPORTANT: Since this example uses a dockerized indexer service, with the GraphQL
> web API being bound at interface `0.0.0.0` your LAN IP might differ from the
> web server being bound at interface `0.0.0.0` your LAN IP might differ from the
> `192.168.1.34` mentioned above.
>
> On *nix platforms you can typically find your LAN IP via `ifconfig | grep inet`
2 changes: 1 addition & 1 deletion examples/hello-world/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ query {
```

> IMPORTANT: Since this example uses a dockerized indexer service, with the GraphQL
> web API being bound at interface `0.0.0.0` your LAN IP might differ from the
> web server being bound at interface `0.0.0.0` your LAN IP might differ from the
> `192.168.1.34` mentioned above.
>
> On *nix platforms you can typically find your LAN IP via `ifconfig | grep inet`
4 changes: 2 additions & 2 deletions packages/fuel-indexer-api-server/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ use tower_http::{
};
use tracing::{error, Level};

/// Result type returned by web API operations.
/// Result type returned by web server operations.
pub type ApiResult<T> = core::result::Result<T, ApiError>;

/// Size of the buffer for reqeusts being passed to the `RateLimitLayer`.
Expand Down Expand Up @@ -70,7 +70,7 @@ impl From<http::Error> for HttpError {
}
}

/// Error type returned by web API operations.
/// Error type returned by web server operations.
#[derive(Debug, Error)]
pub enum ApiError {
#[error("Query builder error {0:?}")]
Expand Down
4 changes: 2 additions & 2 deletions packages/fuel-indexer-api-server/src/models.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ pub struct VerifySignatureRequest {
pub message: String,
}

/// GraphQL web API response.
/// GraphQL web server response.
#[derive(Serialize)]
pub(crate) struct QueryResponse {
/// Arbitrarily sized JSON response.
Expand Down Expand Up @@ -74,7 +74,7 @@ impl Claims {
}
}

/// A SQL query posted to the web API.
/// A SQL query posted to the web server.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct SqlQuery {
/// The literal raw SQL query.
Expand Down
2 changes: 1 addition & 1 deletion packages/fuel-indexer-api-server/src/sql.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ pub enum SqlValidatorError {

/// A validator for SQL queries.
///
/// Intended to ensure that users posting raw SQL queries to web API endpoints
/// Intended to ensure that users posting raw SQL queries to web server endpoints
/// are not attempting to do anything malicious.
pub struct SqlQueryValidator;

Expand Down
Loading

0 comments on commit 17bbba0

Please sign in to comment.