Update CI workflows for E2E tests #13740

lukaszcl · 2024-07-02T12:40:12Z

This PR introduces a Reusable GitHub Workflow for E2E tests, centralizing all the necessary logic to run various types of tests—such as Docker and K8s remote runner—within a single workflow. This approach allows other E2E test workflows to utilize this central workflow at different stages of the software development process, whether it's for every commit, within the merge queue, nightly, or on-demand.

The workflow sources the tests to run from the .github/e2e-tests.yml file, where each test or group of tests is defined with specific parameters: test-cmd to dictate how to execute the test, runs-on to specify the required GitHub runner, and a workflows parameter that outlines when each test should be executed (e.g., on every pull request or nightly).

  # Example of 1 runner for all tests in integration-tests/smoke/ocr_test.go
  - id: smoke/ocr_test.go:* 
    path: integration-tests/smoke/ocr_test.go
    test-type: docker
    test-inputs:
       chainlinkImage: '{{ env.QA_CHAINLINK_IMAGE }}'
       .. # other test inputs if required by a test
    runs-on: ubuntu-latest
    workflows:
      - Run PR E2E Tests
      - Run Nightly E2E Tests
    test-cmd: cd integration-tests/ && go test smoke/ocr_test.go -timeout 30m -count=1 -json
    pyroscope-env: ci-smoke-ocr-evm-simulated-nightly

The run-e2e-tests-reusable-workflow.yml workflow can be utilized in various ways:

Specify Test IDs: Provide comma-separated test IDs in the test_ids input to run specific tests. For example, see the run-selected-e2e-tests.yml workflow.
Run Tests by Workflow: Use the test_workflow input to run all tests associated with a particular workflow. For example, see the run-nightly-e2e-tests.yml workflow.
Custom Test List: Input a list of tests with custom settings using the test_list input. For example, see the run-automation-ondemand-e2e-tests.yml workflow.

To learn more about new CI Workflows for E2E Tests, please visit https://smartcontract-it.atlassian.net/wiki/spaces/TT/pages/815497220/CI+Workflows+for+E2E+Tests

Note: This update does not affect existing workflows for now. Changes to integrate existing workflows with this new setup will be addressed in separate PRs. More details are provided below.

Full scope of this PR:

Add reusable workflow to run E2E tests: run-e2e-tests-reusable-workflow.yml

Add a workflow to run any E2E test from .github/e2e-tests.yml on demand: run-selected-e2e-tests.yml (example build)

gh workflow run run-selected-e2e-tests.yml --ref ${branch_name} -f chainlink_version=develop -f test_ids="smoke/ocr_test.go:*,soak/ocr_test.go:^TestOCRv1Soak$"

Add a workflow to run tests nightly: run-nightly-e2e-tests.yml. Tests to run nightly have "Run Nightly E2E Tests" in workflows field in .github/e2e-tests.yml file. Example build.
Add run-automation-ondemand-e2e-tests.yml that works the same as automation-ondemand-tests.yml but uses the reusable workflow (example build). Once it's tested we can remove the old workflow. run-automation-ondemand-e2e-tests.yml is an example how to run tests with custom test inputs
Add go tool to detect dead-tests (tests that are not part of any CI workflow). Learn more here. This is disabled by default for now
Add go tool to create default test config override (./.github/actions/create-default-e2e-config-override) and use it instead of the old bash script
Add jobs to reusable workflow to easily build required Chainlink Node image and Chainlink Plugins image. Required versions can be set using require_chainlink_image_versions_in_qa_ecr and require_chainlink_plugin_versions_in_qa_ecr workflow inputs

Not in the scope:

Refactoring existing workflows to utilise the new Reusable E2E Tests Workflow will be addressed in separate PRs
Allow to pass full test_config_base64_override as workflow input. This will be enabled once TestConfig does not have any secrets. Related ticket https://smartcontract-it.atlassian.net/browse/TT-1283
Add Slack notifications for workflows that use the E2E tests reusable workflow. This will be addressed in a separate PR

add nightly pipeline add check tool that will be executed in PRs if a test from integration-tests is added or modified. It will report a comment or block pipeline requiring test to be added to the nightly workflow

.github/workflows/run-automation-ondemand-e2e-tests.yml

Tofel · 2024-07-04T10:42:01Z

.github/workflows/run-automation-ondemand-e2e-tests.yml

+          else
+            echo "Error: Not supported image source '${{ github.event.inputs.chainlinkImageUpdate }}'"
+            exit 1
+          fi            


maybe this could be defined as bash function?

Tofel · 2024-07-04T10:44:37Z

.github/workflows/run-e2e-tests-reusable-workflow.yml

+            exit 1
+          fi
+      - name: Install jq
+        run: sudo apt-get install jq          


are you sure it's needed? I'm using jq in a lot of workflows without ever installing it explicitly

.github/workflows/run-e2e-tests-reusable-workflow.yml

integration-tests/e2e_tests_ci_tool/cmd/check_tests_cmd.go

Tofel · 2024-07-04T11:02:37Z

integration-tests/e2e_tests_ci_tool/cmd/check_tests_cmd_test.go

+		{"go test -test.run TestA|^TestB$|TestC", "TestB", true},
+		{"go test -test.run ^TestA|^TestB|TestC$", "TestA", true},
+		{"go test -test.run ^TestA|^TestB|TestC$", "TestC", true},
+		{"go test -test.run ^TestA|^TestB|TestC$", "TestD", false},


same as before, what if I do this: go test -test.run my_tests.go or go test -test.run ^Test[A-C]$" or just go test -test.run '.*'?

@Tofel will go test -test.run my_tests.go work? Note that -test.run is used to specify test name inside a file.

integration-tests/e2e_tests_ci_tool/cmd/create_test_config_cmd.go

Tofel · 2024-07-04T11:09:23Z

@lukaszcl don't we wanna place these go tools you added in the CTF/tools? I think CCIP and other repos should be able to use them too

integration-tests/e2e_tests_ci_tool/cmd/mask_test_config_cmd.go

Tofel · 2024-07-04T11:15:15Z

integration-tests/e2e_tests_ci_tool/cmd/test_config_cmd_test.go

+		},
+		{
+			name: "PrivateEthereumNetworkExecutionLayerFlag",
+			args: []string{"create", "--private-ethereum-network-execution-layer=geth", "--private-ethereum-network-ethereum-version=1.10.0"},


this doesn't look okay. there are 2 ethereum version supported: eth1 and eth2, roughly pow and pos. It has nothing to do with image version :-)

…2e-tests-in-core

lukaszcl · 2024-07-09T17:40:26Z

I'm closing it for now in order to stop running all the tests and workflows for every commit in this PR. I will address all the comments you guys made once I reopen it. cc @Tofel @kalverra

cl-sonarqube-production · 2024-07-09T17:49:17Z

Quality Gate passed

Issues
23 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

lukaszcl added 30 commits June 21, 2024 17:48

wip nightly pipeline

8e21ca5

add nightly pipeline add check tool that will be executed in PRs if a test from integration-tests is added or modified. It will report a comment or block pipeline requiring test to be added to the nightly workflow

add reusable workflow

01d71a3

on push

08eb8d0

echo tests

3b5a6e7

fix matrix

2bf79bc

fix

dfbdecf

add job name

397ba14

run docker tests

6603e9b

add reusable workflow

e17499e

fix

f255221

fail if no test found

1f3b5ca

fix run selected workflow

5901261

Rename reusable workflow

d550f1d

Add nighly workflow

04f2b32

try running k8s tests

06f5922

cleanup

099d654

add permissions

53858e0

fix

7db6612

test

5611e17

Fix

f54a25b

fix

a6ca1ff

test remote runner

c2b411a

Add test suite param

2c9ab53

Run all_tests_in_ocr_tests_go in parallel

ada4954

Update e2e-tests.yml

3d587e8

Add check-tests cmd

fb95890

Unify tools into single cmd

a82ad06

Add check tests step to workflow

79d3146

show more info in annotation

0e6d314

fix

cad8ebd