diff --git a/integration-tests/README.md b/integration-tests/README.md index 5647a62316e..75fa57985ae 100644 --- a/integration-tests/README.md +++ b/integration-tests/README.md @@ -1,137 +1,97 @@ # Integration Tests -Here lives the integration tests for chainlink, utilizing our [chainlink-testing-framework](https://github.com/smartcontractkit/chainlink-testing-framework). +- [Integration Tests](#integration-tests) + - [Summary](#summary) + - [Guidelines](#guidelines) + - [Pre-requisites](#pre-requisites) + - [Test and node configuration](#test-and-node-configuration) + - [Run Tests](#run-tests) + - [Locally](#locally) + - [All tests in a suite](#all-tests-in-a-suite) + - [A single test](#a-single-test) + - [In CI](#in-ci) + - [In Kubernetes](#in-kubernetes) -## NOTE: Move to Testcontainers +## Summary -If you have previously run these smoke tests using GitHub Actions or some sort of Kubernetes setup, that method is no longer necessary. We have moved the majority of our tests to utilize plain Docker containers (with the help of [Testcontainers](https://golang.testcontainers.org/)). This should make tests faster, more stable, and enable you to run them on your local machine without much hassle. +This directory represent a place for different types of integration and system level tests. It utilizes [Chainlink Testing Framework (CTF)](https://github.com/smartcontractkit/chainlink-testing-framework). -## Requirements +> [!TIP] +> **Testcontainers (Dockerized tests)** +> If you want to have faster, locally running, more stable tests, utilize plain Docker containers (with the help of [Testcontainers](https://golang.testcontainers.org/)) instead of using GitHub Actions or Kubernetes. -1. [Go](https://go.dev/) -2. [Docker](https://www.docker.com/) -3. You'll probably want to [increase the resources available to Docker](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container) as most tests require quite a few containers (e.g. OCR requires 6 Chainlink nodes, 6 databases, a simulated blockchain, and a mock server). +## Guidelines -## Configure +### Pre-requisites -We have finished the first pass at moving all test configuration from env vars to TOML files. All product-related configuration is already in TOML files, but env vars are still used to control the log level, Slack notifications, and Kubernetes-related settings. See the [example.env](./example.env) file for how to set these environment variables. - -We have defined some sensible defaults for all products, you can find them in `./testconfig//.toml` files. Each product folder contains an `example.toml` file that describes all options. If you wish to override these values, you can do so by creating a `./testconfig/overrides.toml`. A detailed description of TOML configuration can be found in the [testconfig README](./testconfig/README.md), but if you want to run some tests using default values all you need to do is provide the Chainlink image and version you want to run tests on: -```toml -# ./testconfig/overrides.toml - -[ChainlinkImage] -image = "your image name" -version = "your tag" -``` - -The `./testconfig/overrides.toml` file **should never be committed** and has been added to the [.gitignore](../.gitignore) file as it can often contain secrets like private keys and RPC URLs. - -For more information on how to configure the tests, see the [testconfig README](./testconfig/README.md). - -## Build - -If you'd like to run the tests on a local build of Chainlink, you can point to your own docker image, or build a fresh one with `make`. - -`make build_docker_image image= tag=` - -e.g. - -`make build_docker_image image=chainlink tag=test-tag` - -## Run - -Ensure you have created a `./testconfig/overrides.toml` file with your desired Chainlink image and version. - -`go test ./smoke/_test.go` - -Most test files have a couple of tests, it's recommended to look into the file and focus on a specific one if possible. 90% of the time this will probably be the `Basic` test. See [ocr_test.go](./smoke/ocr_test.go) for example, which contains the `TestOCRBasic` test. - -`go test ./smoke/ocr_test.go -run TestOCRBasic` - -It's generally recommended to run only one test at a time on a local machine as it needs a lot of docker containers and can peg your resources otherwise. You will see docker containers spin up on your machine for each component of the test where you can inspect logs. - -### Configure Seth - -Our new evm client is Seth. Detailed instructions on how to configure it can be found in the [Seth README](./README_SETH.md) in this repo as well as in [Seth repository](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/seth). - -## Analyze - -You can see the results of each test in the terminal with normal `go test` output. If a test fails, logs of each Chainlink container will dump into the `smoke/logs/` folder for later analysis. You can also see these logs in CI uploaded as GitHub artifacts. - -## Running Soak, Performance, Benchmark, and Chaos Tests - -These tests remain bound to a Kubernetes run environment, and require more complex setup and running instructions not documented here. We endeavor to make these easier to run and configure, but for the time being please seek a member of the QA/Test Tooling team if you want to run these. - -### How to run reorg tests -Run soak/ocr_test.go with reorg below finality and `FinalityTagEnabled=false` +1. [Installed Go](https://go.dev/) +2. For local testing, [Installed Docker](https://www.docker.com/). Consider [increasing resources limits needed by Docker](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container) as most tests require building several containers for a Decentralized Oracle Network (e.g. OCR requires 6 nodes and DBs, a simulated blockchain, and a mock server). +3. For remote testing, access to Kubernetes cluster/AWS Docker registry (if you are pulling images from private links). +4. Docker image. If there is no image to pull from a registry, you may run tests against a custom build. Run the following command to build the image: ```bash -make test_soak_ocr_reorg_1 +make build_docker_image image= tag= ``` -Run soak/ocr_test.go with reorg below finality and `FinalityTagEnabled=true`: +Example: `make build_docker_image image=chainlink tag=test-tag` -```bash -make test_soak_ocr_reorg_2 -``` +#### Test and node configuration -Run reorg/automation_reorg_test.go with reorg settings: +1. Setup `.env` file in the root of `integration-tests` directory. See [example.env](./example.env) for how to set test-runner log level (not a node's log level), Slack notifications, and Kubernetes-related settings. -1. Use Simulated Geth network and put GethReorgConfig in overrides.toml + 1. Ensure to **update you environment** with the following commands: + 1. `cd integration-tests` + 2. `source .env` - ```toml - [Network] - selected_networks=["simulated"] - [Network.GethReorgConfig] - enabled = true - depth = 10 - delay_create = "3s" - ``` +2. Setup test secrets. See "how-to" details in the [CTF config README](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#test-secrets). -2. Then run the test: - ```bash - make test_reorg_automation - ``` +3. Provide test and node configuration (for more details refer to [testconfig README](./testconfig/README.md) and `example.toml` files): + 1. **Defaults** for all products are defined in `./testconfig//.toml` files. + 2. To **override default values**, create a `./testconfig/overrides.toml` file (yes, in the root of `testconfig`, not a product directory) specifying the values to override by your test (see some examples in [./testconfig/ocr2/overrides](./testconfig/ocr2/overrides)). + > [!IMPORTANT] + > **Image version and node configs** + > 1. Do not forget to set `[ChainlinkImage].version` to test against the necessary remotely accessible version or [custom build](#optional-build-docker-image). + > 2. When running OCR-related tests, pay attention to which version of OCR you enable/override in your `overrides.toml`. + > 3. Pay attention to not committing any sensitive data. -Run reorg above finality docker test: +4. [Optional] Configure Seth (or use defaults), an evm client used by tests. Detailed instructions on how to configure it can be found in the [Seth README](./README_SETH.md) and [Seth repository](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/seth). -```bash -go test -v -run ^TestReorgAboveFinality_FinalityTagDisabled$ ./smoke -``` +### Run Tests -### How to run gas simulation tests +#### Locally -Run soak/ocr_test.go with gas spike: +> [!NOTE] +> **Resources utilization** +> It's recommended to run only one test at a time (run tests sequentially) on a local machine as it needs a lot of docker containers and can peg your resources otherwise. You will see docker containers spin up on your machine for each component of the test where you can inspect logs. -```bash -make test_soak_ocr_gas_spike -``` +##### All tests in a suite -Run soak/ocr_test.go with changing gas limit creating block congestion: +1. Run CLI command: `go test ./smoke/_test.go` + Example: `go test ./smoke/ocr_test.go` +2. Logs of each Chainlink container will dump into the `smoke/logs/`. +3. To enable debugging of HTTP and RPC clients set the following env vars: ```bash -make test_soak_ocr_gas_limit_change +export SETH_LOG_LEVEL=debug +export RESTY_DEBUG=true ``` -Note: you can update gas simulation params for the tests below in in testconfig/ocr.toml +##### A single test -### How to run tests with RPC node failure +1. Run CLI command: `go test ./smoke/_test.go -run ` + Example: `go test ./smoke/ocr_test.go -run TestOCRBasic` -Run soak/ocr_test.go with RPC network chaos by bringing down network to RPC node for all Chainlink Nodes: +#### In CI -```bash -make test_soak_ocr_rpc_down_all_cl_nodes -``` +1. Refer [Tests Run Books](./run-books/) for more details. +2. Logs in CI uploaded as GitHub artifacts. -Run soak/ocr_test.go with RPC network chaos by bringing down network to RPC node for 50 percent of Chainlink Nodes: +#### In Kubernetes -```bash -make test_soak_ocr_rpc_down_half_cl_nodes -``` +Such tests as Soak, Performance, Benchmark, and Chaos Tests remain bound to a Kubernetes run environment. -### Debugging HTTP and RPC clients -```bash -export SETH_LOG_LEVEL=debug -export RESTY_DEBUG=true -``` +1. Ensure all necessary configuration is provided (see [Test Configuration](#test-configuration)) +2. You are logged in to your Kubernetes cluster (with `aws sso login`) +3. Run CLI command: `make test_` (see commands in [Makefile .PHONY lines](./Makefile) for more details) + Example: `make test_soak_ocr`, `make test_soak_ocr2`, `test_node_migrations`, etc. +4. Navigate to Grafana dashboards to see test and node logs, and results. diff --git a/integration-tests/run-books/VRF.md b/integration-tests/run-books/VRF.md new file mode 100644 index 00000000000..ed0b7775e76 --- /dev/null +++ b/integration-tests/run-books/VRF.md @@ -0,0 +1,65 @@ +# VRF Tests Run Book +* All test configs should be placed in the [integration-tests/testconfig](../testconfig) folder +* All test configs for running tests in live testnets should be under [integration-tests/testconfig/vrfv2plus/overrides](../testconfig/vrfv2plus/overrides) folder + + +## Functional Tests +### In CI - using On Demand Workflows +```bash +gh workflow run "on-demand-vrfv2plus-smoke-tests.yml" \ +--ref develop \ +-f=test_secrets_override_key= \ +-f test_config_override_path= \ +-f test_suite="Selected Tests" \ # Optional, Options - "All Tests", "Selected Tests". Default is "All Tests". If "Selected Tests" is selected, then `test_list_regex` should be provided +-f test_list_regex="" \ # Optional, default is "TestVRFv2Plus$/(Link_Billing|Native_Billing|Direct_Funding)|TestVRFV2PlusWithBHS" which are P0 tests +-f chainlink_version="<>" # Optional, default is image created from develop branch. Not needed if you run tests against existing environment +-f notify_user_id_on_failure= # Optional, default is empty. If provided, will notify the user on slack if the tests fail +``` + +#### Examples: +Run P0 tests against existing environment (Staging) on Arbitrum Sepolia +```bash +gh workflow run "on-demand-vrfv2plus-smoke-tests.yml" \ +--ref develop \ +-f=test_secrets_override_key= \ +-f test_config_override_path=integration-tests/testconfig/vrfv2plus/overrides/staging/arbitrum_sepolia_staging_test_config.toml \ +-f test_suite="Selected Tests" +``` + +Run all tests deploying all contracts, CL nodes with `2.15.0` version on Base Sepolia +```bash +gh workflow run "on-demand-vrfv2plus-smoke-tests.yml" \ +--ref develop \ +-f=test_secrets_override_key= \ +-f test_config_override_path=integration-tests/testconfig/vrfv2plus/overrides/new_env/base_sepolia_new_env_test_config.toml \ +-f test_suite="All Tests" \ +-f chainlink_version="2.15.0" +``` + +### Locally +```bash +cd integration-tests +TEST_LOG_LEVEL=debug \ +BASE64_CONFIG_OVERRIDE=$(cat | base64) \ +go test -v -timeout 15m -run "" ./smoke +``` + +## Performance Tests +```bash +gh workflow run "on-demand-vrfv2plus-performance-test.yml" \ +--ref develop \ +-f=test_secrets_override_key= \ +-f test_config_override_path= \ +-f performanceTestType=“Smoke” # Options - "Smoke", "Soak", "Stress", "Load". +-f test_list_regex="" # Optional, default is "TestVRFV2PlusPerformance" +``` + +#### Examples: +Run SOAK tests against existing environment (Staging) on Base Sepolia +```bash +gh workflow run "on-demand-vrfv2plus-performance-test.yml" \ +--ref develop \ +-f=test_secrets_override_key= \ +-f test_config_override_path=integration-tests/testconfig/vrfv2plus/overrides/staging/base_sepolia_staging_test_config.toml \ +-f performanceTestType=“Soak” +``` diff --git a/integration-tests/smoke/ocr_test.go b/integration-tests/smoke/ocr_test.go index a19adb5c022..6556591b6d5 100644 --- a/integration-tests/smoke/ocr_test.go +++ b/integration-tests/smoke/ocr_test.go @@ -105,9 +105,6 @@ func prepareORCv1SmokeTestEnv(t *testing.T, l zerolog.Logger, firstRoundResult i nodeClients := env.ClCluster.NodeAPIs() bootstrapNode, workerNodes := nodeClients[0], nodeClients[1:] - err = actions.FundChainlinkNodesFromRootAddress(l, sethClient, contracts.ChainlinkClientToChainlinkNodeWithKeysAndAddress(workerNodes), big.NewFloat(*config.Common.ChainlinkNodeFunding)) - require.NoError(t, err, "Error funding Chainlink nodes") - t.Cleanup(func() { // ignore error, we will see failures in the logs anyway _ = actions.ReturnFundsFromNodes(l, sethClient, contracts.ChainlinkClientToChainlinkNodeWithKeysAndAddress(env.ClCluster.NodeAPIs())) @@ -122,6 +119,10 @@ func prepareORCv1SmokeTestEnv(t *testing.T, l zerolog.Logger, firstRoundResult i err = actions.CreateOCRJobsLocal(ocrInstances, bootstrapNode, workerNodes, 5, env.MockAdapter, big.NewInt(sethClient.ChainID)) require.NoError(t, err, "Error creating OCR jobs") + // there is no need to fund the nodes unless the OCR contract and job are configured + err = actions.FundChainlinkNodesFromRootAddress(l, sethClient, contracts.ChainlinkClientToChainlinkNodeWithKeysAndAddress(workerNodes), big.NewFloat(*config.Common.ChainlinkNodeFunding)) + require.NoError(t, err, "Error funding Chainlink nodes") + err = actions.WatchNewOCRRound(l, sethClient, 1, contracts.V1OffChainAgrregatorToOffChainAggregatorWithRounds(ocrInstances), time.Duration(3*time.Minute)) require.NoError(t, err, "Error watching for new OCR round")