Skip to content

Commit

Permalink
docs: Add initial doc for system tests (#22002)
Browse files Browse the repository at this point in the history
(cherry picked from commit 5c4f4ac)
  • Loading branch information
alpe authored and mergify[bot] committed Oct 2, 2024
1 parent d6ec71f commit 15f86d6
Show file tree
Hide file tree
Showing 2 changed files with 67 additions and 21 deletions.
58 changes: 58 additions & 0 deletions docs/build/building-apps/06-system-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
sidebar_position: 1
---

# System Tests

System tests provide a framework to write and execute black box tests against a running chain. This adds another level
of confidence on top of unit, integration, and simulations tests, ensuring that business-critical scenarios
(like double signing prevention) or scenarios that can't be tested otherwise (like a chain upgrade) are covered.

## Vanilla Go for Flow Control

System tests are vanilla Go tests that interact with the compiled chain binary. The `test runner` component starts a
local testnet of 4 nodes (by default) and provides convenient helper methods for accessing the
`system under test (SUT)`.
A `CLI wrapper` makes it easy to access keys, submit transactions, or execute operations. Together, these components
enable the replication and validation of complex business scenarios.

Here's an example of a double signing test, where a new node is added with the same key as the first validator:
[double signing test example](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/fraud_test.go)

The [getting started tutorial](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/getting_started.md)
contains a step-by-step guide to building and running your first system test. It covers setting chain state via genesis
or
transactions and validation via transaction response or queries.

## Design Principles and Guidelines

System tests are slower compared to unit or integration tests as they interact with a running chain. Therefore, certain
principles can guide their usage:

- **Perspective:** Tests should mimic a human interacting with the chain from the outside. Initial states can be set via
genesis or transactions to support a test scenario.
- **Roles:** The user can have multiple roles such as validator, delegator, granter, or group admin.
- **Focus:** Tests should concentrate on happy paths or business-critical workflows. Unit and integration tests are
better suited for more fine-grained testing.
- **Workflows:** Test workflows and scenarios, not individual units. Given the high setup costs, it is reasonable to
combine multiple steps and assertions in a single test method.
- **Genesis Mods:** Genesis modifications can incur additional time costs for resetting dirty states. Reuse existing
accounts (node0..n) whenever possible.
- **Framework:** Continuously improve the framework for better readability and reusability.

## Errors and Debugging

All output is logged to `systemtests/testnet/node{0..n}.out`. Usually, `node0.out` is very noisy as it receives the CLI
connections. Prefer any other node's log to find stack traces or error messages.

Using system tests for state setup during debugging has become very handy:

- Start the test with one node only and verbose output:

```sh
go test -v -tags=system_test ./ --run TestAccountCreation --verbose --nodes-count=1
```

- Copy the CLI command for the transaction and modify the test to stop before the command
- Start the node with `--home=<project-home>/tests/systemtests/testnet/node0/<binary-name>/` in debug mode
- Execute CLI command from shell and enter breakpoints
30 changes: 9 additions & 21 deletions docs/build/building-modules/16-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,38 +86,26 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/tests/integration/bank

## Simulations

Simulations uses as well a minimal application, built with [`depinject`](../packages/01-depinject.md):
Simulations fuzz tests for deterministic message execution. They use a minimal application, built with [`depinject`](../packages/01-depinject.md):

:::note
You can as well use the `AppConfig` `configurator` for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/slashing/app_test.go#L54-L62). There is no difference between those two ways, use whichever you prefer.
Simulations have been refactored to message factories
:::

Following is an example for `x/gov/` simulations:
An example for `x/bank/` simulations:

```go reference
https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L406-L430
https://github.com/cosmos/cosmos-sdk/blob/release/v0.52.x/x/bank/simulation/msg_factory.go#L13-L20
```

```go reference
https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L90-L132
```

## End-to-end Tests
## System Tests

End-to-end tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
They must test the whole application flow, from the user perspective (for instance, CLI tests). They are located under [`/tests/e2e`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e).
System tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
They test the whole application flow as black box, from the user perspective. They are located under [`/tests/systemtests`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/systemtests).

<!-- @julienrbrt: makes more sense to use an app wired app to have 0 simapp dependencies -->
For that, the SDK is using `simapp` but you should use your own application (`appd`).
Here are some examples:
For that, the SDK is using the `simapp` binary, but you should use your own binary.
More details about system test can be found in [building-apps](https://docs.cosmos.network/main/build/building-apps/06-system-tests.md)

* SDK E2E tests: <https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e>.
* Cosmos Hub E2E tests: <https://github.com/cosmos/gaia/tree/main/tests/e2e>.
* Osmosis E2E tests: <https://github.com/osmosis-labs/osmosis/tree/main/tests/e2e>.

:::note warning
The SDK is in the process of creating its E2E tests, as defined in [ADR-59](https://docs.cosmos.network/main/build/architecture/adr-059-test-scopes). This page will eventually be updated with better examples.
:::

## Learn More

Expand Down

0 comments on commit 15f86d6

Please sign in to comment.