Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add initial doc for system tests #22002

Merged
merged 5 commits into from
Oct 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions docs/build/building-apps/06-system-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
sidebar_position: 1
---

# System Tests

System tests provide a framework to write and execute black box tests against a running chain. This adds another level
of confidence on top of unit, integration, and simulations tests, ensuring that business-critical scenarios
(like double signing prevention) or scenarios that can't be tested otherwise (like a chain upgrade) are covered.

## Vanilla Go for Flow Control

System tests are vanilla Go tests that interact with the compiled chain binary. The `test runner` component starts a
local testnet of 4 nodes (by default) and provides convenient helper methods for accessing the
`system under test (SUT)`.
A `CLI wrapper` makes it easy to access keys, submit transactions, or execute operations. Together, these components
enable the replication and validation of complex business scenarios.

Here's an example of a double signing test, where a new node is added with the same key as the first validator:
[double signing test example](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/fraud_test.go)

The [getting started tutorial](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/getting_started.md)
contains a step-by-step guide to building and running your first system test. It covers setting chain state via genesis
or
transactions and validation via transaction response or queries.

## Design Principles and Guidelines

System tests are slower compared to unit or integration tests as they interact with a running chain. Therefore, certain
principles can guide their usage:

- **Perspective:** Tests should mimic a human interacting with the chain from the outside. Initial states can be set via
genesis or transactions to support a test scenario.
- **Roles:** The user can have multiple roles such as validator, delegator, granter, or group admin.
- **Focus:** Tests should concentrate on happy paths or business-critical workflows. Unit and integration tests are
better suited for more fine-grained testing.
- **Workflows:** Test workflows and scenarios, not individual units. Given the high setup costs, it is reasonable to
combine multiple steps and assertions in a single test method.
- **Genesis Mods:** Genesis modifications can incur additional time costs for resetting dirty states. Reuse existing
accounts (node0..n) whenever possible.
- **Framework:** Continuously improve the framework for better readability and reusability.

## Errors and Debugging

All output is logged to `systemtests/testnet/node{0..n}.out`. Usually, `node0.out` is very noisy as it receives the CLI
connections. Prefer any other node's log to find stack traces or error messages.

Using system tests for state setup during debugging has become very handy:

- Start the test with one node only and verbose output:

```sh
go test -v -tags=system_test ./ --run TestAccountCreation --verbose --nodes-count=1
```

- Copy the CLI command for the transaction and modify the test to stop before the command
- Start the node with `--home=<project-home>/tests/systemtests/testnet/node0/<binary-name>/` in debug mode
- Execute CLI command from shell and enter breakpoints
30 changes: 9 additions & 21 deletions docs/build/building-modules/16-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,38 +86,26 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/tests/integration/bank

## Simulations

Simulations uses as well a minimal application, built with [`depinject`](../packages/01-depinject.md):
Simulations fuzz tests for deterministic message execution. They use a minimal application, built with [`depinject`](../packages/01-depinject.md):

:::note
You can as well use the `AppConfig` `configurator` for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/slashing/app_test.go#L54-L62). There is no difference between those two ways, use whichever you prefer.
Simulations have been refactored to message factories
:::

Following is an example for `x/gov/` simulations:
An example for `x/bank/` simulations:

```go reference
https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L406-L430
https://github.com/cosmos/cosmos-sdk/blob/release/v0.52.x/x/bank/simulation/msg_factory.go#L13-L20
```

```go reference
https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L90-L132
```

## End-to-end Tests
## System Tests

End-to-end tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
They must test the whole application flow, from the user perspective (for instance, CLI tests). They are located under [`/tests/e2e`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e).
System tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
They test the whole application flow as black box, from the user perspective. They are located under [`/tests/systemtests`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/systemtests).

<!-- @julienrbrt: makes more sense to use an app wired app to have 0 simapp dependencies -->
For that, the SDK is using `simapp` but you should use your own application (`appd`).
Here are some examples:
For that, the SDK is using the `simapp` binary, but you should use your own binary.
More details about system test can be found in [building-apps](https://docs.cosmos.network/main/build/building-apps/06-system-tests.md)

* SDK E2E tests: <https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e>.
* Cosmos Hub E2E tests: <https://github.com/cosmos/gaia/tree/main/tests/e2e>.
* Osmosis E2E tests: <https://github.com/osmosis-labs/osmosis/tree/main/tests/e2e>.

:::note warning
The SDK is in the process of creating its E2E tests, as defined in [ADR-59](https://docs.cosmos.network/main/build/architecture/adr-059-test-scopes). This page will eventually be updated with better examples.
:::

## Learn More

Expand Down
Loading