[TT-748] TOML config for integration tests (#11588)

* WIP#1 * WIP#2 * WIP#3 * WIP#4 * WIP#5 * WIP#6 * wIP#7 - overrides and validations for vrfv2 * wIP#8 - overrides and validations for vrfv2lus * WIP#9 - adjust vrfv2/plus default configs * fix test config test, add logger to it * smoke tests should work now * fix replace, fix workflow * try to remove newlines from base64 * update ctf version, fix line wrap * use newer version of ctf, fix log poller and workflow issues * expand selected networks to array * fix vrfv2 smoke test, mask baseb4 override * define base64 override before masking it * a couple of masks more * set some content for BASE64_CONFIG_OVERRIDE so that it's masked * remove masks that break toml config * experiment with base64 #1 * fix ocr2 smoke test * fix lints * baseb4 experiment #2 * base64 experiment #3 * hardcode some image * exp#5 * fail config test on purpose * fix lints * run all smoke tests * add missing test config to ocr2 test * add missing loki_url * remove old config files, set config override for remote runner * fix compile errors * fix lint * make the vrfv2plus name in test matrix unique * fix loki env config, adjust vrfv2 and vrfv2plus tests and configs * add TOML-specific overriding of CL image/version in k8s * fixed automation load test * couple small fixes to how helm is created * remove multiple config files, store all config versions in single product-specific toml file * go.mod * restore deleted test, fix vrfv2 smoke tests * fix vrfv2plus config validation * properly fix vrfv2plus smoke test * fix loki basic auth passing * add add-mask again to base64 * remove obsolete configs, use latest ctf, adjust tomls * try another base64 hiding * go.sum * fix loki compile issue * try base64 env var with _SECRET suffix * try yet another variable name * restore old env var name * another obscuring attempt * another attempt * use newer CTF version, check what env vars are forwaded to k8s * go.sum * updated to latest ctf * try on demand autonation test with base64 input * modify all other test workflows so that they use TOML config * update reamdes, example envs and some more workflows * uncomment solana tests * update to latest ctf * update to latest ctf * go.mod * add MustCopy() method to TestConfig, fix failing vrvf2plus test * fix duration usages * fix lints * added examples of full configs for each product * use latest ctf * simplify some examples, correct client compatibility workflow * go.mod * copy TOML configs before executing test binary in compatibility job * use bash not sh to run the script, add missing patterns * chmod script and run it * add debug * print vars before checking their number * another debug attempt * try double quotes * store script and run that * fix script formatting * fix correct script 🤦 * fix zip command * fix upload artifact path * add some debug * update tracing readme, use latest ctf * latest ctf * lastest ctf * check what is the config missing * add missing pyrscope enabled var * remove debug * don't notify Adam if eth client compatibility test fail * latest ctf, faster eth2 chain by default * latest ctf, slots per epoch = 2 * go.sum * move convert_to_toml_array() bash function to a file and import it in GHA workflows instead of constantly defining before using * provide default configs to live testnet tests that use test binary * pass network config between jobs correctly * set .root_dir for live testnet job * try passing * try with defined output * define correct output for build-tests job * fix value name * debug * 2x base64 * try with shorter variable * add network config to dynamic config * include shared function * embed toml configs when build tag is present * test passing shared functions as outputs * update action version * remove newline from base64 * newer gha * fix log poller path, adjust client compatibility test to not copy tomls * debug * once more * do not use shared function * use correct network toml key for sepolia * fix dynamic toml * fix tomls * fix sepolia endpoint, uncomment slack notifications * remove outputs shared function from clients compatibility tests * fix alfajores once and for all * use newer gha to build tests * don't fail if there's no embedded config * remove tests we don't need anymore * use commit from main version of GHA repository instead of branch commit for build tests action * simplify config overriding * fix go.mod * remove ApplyOverride() and fix compile issue * fix lint * try base64 value extraction * try anothe approach * yet another attempt * yet another attempt 2 * fix base64 from input * one more time with final config * last test * make automation-ondemand test more secure, remove debug from on-demand-vrfv2-performance-test.yml * uncommet test execution in vrfv2 performance workflow * use private eth networks instead of geth in vrf load tests * update TOML keys related to Loki and Grafana * by default use pow, not pos * go.mod * latest ctf * remove references to deleted performance test suite * don't require Loki config when using remote runner * latest ctf * add Load config and figure out overrides * append loki config to ocr soak test on demand job * embed test configs and create .root_dir, when building test image * define an interface for global test config * move interface to testconfig * set pyroscope env for cron * fix check for remote running * bump CTF * use interfaces in public methods instead of TestConfig * small adjustments * fix test env builder * fix automation config type cast * use latest ctf * add special handling for slice override for Automation * update examples * slightly adjust vrfv2 configs, remove default empty values * move creation of base64 config override env var in smoke tests to a GHA action * fix compile error * debug for ocr test, camelcase for base64 action, two new actions to reduce loc * checkout repo before accessing actions * remove unnecessary test config instance from ocr test * mask before using input, more ocr debug * remove debug, fix reading named configurations from base64 override * mask input before using merge base64 action * fix integration tests workflow * update default.toml, add action for live testnet tests * download test binary after creating base64 env var * fix missing log targets * comment out reporting part * reuse live testnets specific action * fix network names * use dynamic network key * fix automation tests (#11815) * fix automation benchmark test action * update default toml * try newer run tests action * another try * look for .root_dir even deeper * Update integration-tests/benchmark/keeper_test.go * Update integration-tests/test.Dockerfile --------- Co-authored-by: Bartek Tofel <[email protected]> * fix chain config for eth2 compatibility tests * replace in-workflow toml generation for CL version upgrade tests with GH action * make test timeout configurable in client compatbility tests * prepare base64 config for log poller matrix tests --------- Co-authored-by: Ilja Pavlovs <[email protected]> Co-authored-by: skudasov <[email protected]> Co-authored-by: Gheorghe Strimtu <[email protected]> Co-authored-by: Anirudh Warrier <[email protected]>
smartcontractkit · Jan 24, 2024 · ccb3e99 · ccb3e99
1 parent aa05727
commit ccb3e99
Show file tree

Hide file tree

Showing 139 changed files with 6,166 additions and 4,635 deletions.
diff --git a/.github/actions/build-test-image/action.yml b/.github/actions/build-test-image/action.yml
@@ -15,7 +15,7 @@ inputs:
     required: false
   suites:
     description: The test suites to build into the image
-    default: chaos migration performance reorg smoke soak benchmark load/automationv2_1
+    default: chaos migration reorg smoke soak benchmark load/automationv2_1
     required: false
   QA_AWS_ROLE_TO_ASSUME:
     description: The AWS role to assume as the CD user, if any. Used in configuring the docker/login-action

diff --git a/.github/actions/setup-create-base64-config-live-testnets/action.yml b/.github/actions/setup-create-base64-config-live-testnets/action.yml
@@ -0,0 +1,130 @@
+name: Create Base64 Config
+description: A composite action that creates a base64-encoded config to be used by integration tests
+
+inputs:
+  runId:
+    description: The run id
+  testLogCollect:
+    description: Whether to always collect logs, even for passing tests
+    default: "false"
+  chainlinkImage:
+    description: The chainlink image to use
+    default: "public.ecr.aws/chainlink/chainlink"
+  chainlinkVersion:
+    description: The git commit sha to use for the image tag
+  pyroscopeServer:
+    description: URL of Pyroscope server
+  pyroscopeEnvironment: 
+    description: Name of Pyroscope environment
+  pyroscopeKey:
+    description: Pyroscope server key
+  lokiEndpoint:
+    description: Loki push endpoint
+  lokiTenantId:
+    description: Loki tenant id
+  lokiBasicAuth:
+    description: Loki basic auth
+  logstreamLogTargets:
+    description: Where to send logs (e.g. file, loki)
+  grafanaUrl:
+    description: Grafana URL
+  grafanaDashboardUrl:
+    description: Grafana dashboard URL
+  network:
+    description: Network to run tests on
+  httpEndpoints:
+    description: HTTP endpoints to use for network
+  wsEndpoints:
+    description: WS endpoints to use for network
+  fundingKeys:
+    description: Funding keys to use for network
+
+runs:
+  using: composite
+  steps:
+    - name: Prepare Base64 TOML override
+      shell: bash
+      id: base64-config-override
+      env:
+        RUN_ID: ${{ inputs.runId }}
+        PYROSCOPE_SERVER: ${{ inputs.pyroscopeServer }} 
+        PYROSCOPE_ENVIRONMENT: ${{ inputs.pyroscopeEnvironment }} 
+        PYROSCOPE_KEY: ${{ inputs.pyroscopeKey }} 
+        CHAINLINK_IMAGE: ${{ inputs.chainlinkImage }}
+        CHAINLINK_VERSION: ${{ inputs.chainlinkVersion }}
+        LOKI_ENDPOINT: ${{ inputs.lokiEndpoint }}
+        LOKI_TENANT_ID: ${{ inputs.lokiTenantId }}
+        LOKI_BASIC_AUTH: ${{ inputs.lokiBasicAuth }}
+        LOGSTREAM_LOG_TARGETS: ${{ inputs.logstreamLogTargets }}
+        GRAFANA_URL: ${{ inputs.grafanaUrl }}
+        GRAFANA_DASHBOARD_URL: ${{ inputs.grafanaDashboardUrl }}
+        NETWORK: ${{ inputs.network }}
+        HTTP_ENDPOINTS: ${{ inputs.httpEndpoints }}
+        WS_ENDPOINTS: ${{ inputs.wsEndpoints }}
+        FUNDING_KEYS: ${{ inputs.fundingKeys }}
+      run: |
+        convert_to_toml_array() {
+            local IFS=','
+            local input_array=($1)
+            local toml_array_format="["
+
+            for element in "${input_array[@]}"; do
+                toml_array_format+="\"$element\","
+            done
+
+            toml_array_format="${toml_array_format%,}]"
+            echo "$toml_array_format"
+        }
+
+        if [ -n "$PYROSCOPE_SERVER" ]; then
+          pyroscope_enabled=true
+        else
+          pyroscope_enabled=false
+        fi
+        
+        cat << EOF > config.toml
+        [Common]
+        chainlink_node_funding=0.5        
+        
+        [ChainlinkImage]
+        image="$CHAINLINK_IMAGE"
+        version="$CHAINLINK_VERSION"
+        
+        [Pyroscope]
+        enabled=$pyroscope_enabled
+        server_url="$PYROSCOPE_SERVER"
+        environment="$PYROSCOPE_ENVIRONMENT"
+        key="$PYROSCOPE_KEY"
+
+        [Logging]
+        run_id="$RUN_ID"
+
+        [Logging.LogStream]
+        log_targets=$(convert_to_toml_array "$LOGSTREAM_LOG_TARGETS")
+
+        [Logging.Loki]
+        tenant_id="$LOKI_TENANT_ID"
+        endpoint="$LOKI_URL"
+        basic_auth="$LOKI_BASIC_AUTH"
+
+        [Logging.Grafana]
+        base_url="$GRAFANA_URL"
+        dasboard_url="$GRAFANA_DASHBOARD_URL"
+
+        [Network]
+        selected_networks=["$NETWORK"]
+
+        [Network.RpcHttpUrls]
+        "$NETWORK" = $(convert_to_toml_array "$HTTP_ENDPOINTS")
+
+        [Network.RpcWsUrls]
+        "$NETWORK" = $(convert_to_toml_array "$WS_ENDPOINTS")
+
+        [Network.WalletKeys]
+        "$NETWORK" = $(convert_to_toml_array "$FUNDING_KEYS")
+        EOF
+        
+        BASE64_CONFIG_OVERRIDE=$(cat config.toml | base64 -w 0)
+        echo ::add-mask::$BASE64_CONFIG_OVERRIDE
+        echo "BASE64_CONFIG_OVERRIDE=$BASE64_CONFIG_OVERRIDE" >> $GITHUB_ENV
+        touch .root_dir
diff --git a/.github/actions/setup-create-base64-config/action.yml b/.github/actions/setup-create-base64-config/action.yml
@@ -0,0 +1,122 @@
+name: Create Base64 Config
+description: A composite action that creates a base64-encoded config to be used by integration tests
+
+inputs:
+  runId:
+    description: The run id
+  testLogCollect:
+    description: Whether to always collect logs, even for passing tests
+    default: "false"
+  selectedNetworks:
+    description: The networks to run tests against 
+  chainlinkImage:
+    description: The chainlink image to use
+    default: "public.ecr.aws/chainlink/chainlink"
+  chainlinkVersion:
+    description: The git commit sha to use for the image tag
+  pyroscopeServer:
+    description: URL of Pyroscope server
+  pyroscopeEnvironment: 
+    description: Name of Pyroscope environment
+  pyroscopeKey:
+    description: Pyroscope server key
+  lokiEndpoint:
+    description: Loki push endpoint
+  lokiTenantId:
+    description: Loki tenant id
+  lokiBasicAuth:
+    description: Loki basic auth
+  logstreamLogTargets:
+    description: Where to send logs (e.g. file, loki)
+  grafanaUrl:
+    description: Grafana URL
+  grafanaDashboardUrl:
+    description: Grafana dashboard URL
+
+runs:
+  using: composite
+  steps:
+    - name: Prepare Base64 TOML override
+      shell: bash
+      id: base64-config-override
+      env:
+        RUN_ID: ${{ inputs.runId }}
+        TEST_LOG_COLLECT: ${{ inputs.testLogCollect }}
+        SELECTED_NETWORKS: ${{ inputs.selectedNetworks }}
+        PYROSCOPE_SERVER: ${{ inputs.pyroscopeServer }} 
+        PYROSCOPE_ENVIRONMENT: ${{ inputs.pyroscopeEnvironment }} 
+        PYROSCOPE_KEY: ${{ inputs.pyroscopeKey }} 
+        CHAINLINK_IMAGE: ${{ inputs.chainlinkImage }}
+        CHAINLINK_VERSION: ${{ inputs.chainlinkVersion }}
+        LOKI_ENDPOINT: ${{ inputs.lokiEndpoint }}
+        LOKI_TENANT_ID: ${{ inputs.lokiTenantId }}
+        LOKI_BASIC_AUTH: ${{ inputs.lokiBasicAuth }}
+        LOGSTREAM_LOG_TARGETS: ${{ inputs.logstreamLogTargets }}
+        GRAFANA_URL: ${{ inputs.grafanaUrl }}
+        GRAFANA_DASHBOARD_URL: ${{ inputs.grafanaDashboardUrl }}
+      run: |
+        echo ::add-mask::$CHAINLINK_IMAGE        
+        function convert_to_toml_array() {
+          local IFS=','
+          local input_array=($1)
+          local toml_array_format="["
+
+          for element in "${input_array[@]}"; do
+              toml_array_format+="\"$element\","
+          done
+
+          toml_array_format="${toml_array_format%,}]"
+          echo "$toml_array_format"
+        }
+        
+        selected_networks=$(convert_to_toml_array "$SELECTED_NETWORKS")      
+        log_targets=$(convert_to_toml_array "$LOGSTREAM_LOG_TARGETS")
+        
+        if [ -n "$PYROSCOPE_SERVER" ]; then
+          pyroscope_enabled=true
+        else
+          pyroscope_enabled=false
+        fi
+
+        if [ -n "$TEST_LOG_COLLECT" ]; then
+          test_log_collect=true
+        else
+          test_log_collect=false
+        fi              
+
+        cat << EOF > config.toml
+        [Network]
+        selected_networks=$selected_networks
+        
+        [ChainlinkImage]
+        image="$CHAINLINK_IMAGE"
+        version="$CHAINLINK_VERSION"
+        
+        [Pyroscope]
+        enabled=$pyroscope_enabled
+        server_url="$PYROSCOPE_SERVER"
+        environment="$PYROSCOPE_ENVIRONMENT"
+        key="$PYROSCOPE_KEY"
+
+        [Logging]
+        test_log_collect=$test_log_collect
+        run_id="$RUN_ID"
+
+        [Logging.LogStream]
+        log_targets=$log_targets
+
+        [Logging.Loki]
+        tenant_id="$LOKI_TENANT_ID"
+        endpoint="$LOKI_ENDPOINT"
+        basic_auth="$LOKI_BASIC_AUTH"
+        # legacy, you only need this to access the cloud version
+        # bearer_token="bearer_token"
+
+        [Logging.Grafana]
+        base_url="$GRAFANA_URL"
+        dasboard_url="$GRAFANA_DASHBOARD_URL"
+        EOF
+        
+        BASE64_CONFIG_OVERRIDE=$(cat config.toml | base64 -w 0)
+        echo ::add-mask::$BASE64_CONFIG_OVERRIDE
+        echo "BASE64_CONFIG_OVERRIDE=$BASE64_CONFIG_OVERRIDE" >> $GITHUB_ENV
diff --git a/.github/actions/setup-create-base64-upgrade-config /action.yml b/.github/actions/setup-create-base64-upgrade-config /action.yml
@@ -0,0 +1,61 @@
+name: Create Base64 Upgrade Config
+description: A composite action that creates a base64-encoded config to be used by Chainlink version upgrade tests
+
+inputs:
+  selectedNetworks:
+    description: The networks to run tests against 
+  chainlinkImage:
+    description: The chainlink image to upgrade from
+    default: "public.ecr.aws/chainlink/chainlink"
+  chainlinkVersion:
+    description: The git commit sha to use for the image tag
+  upgradeImage:
+    description: The chainlink image to upgrade to
+    default: "public.ecr.aws/chainlink/chainlink"
+  upgradeVersion:
+    description: The git commit sha to use for the image tag
+
+runs:
+  using: composite
+  steps:
+    - name: Prepare Base64 TOML override
+      shell: bash
+      id: base64-config-override
+      env:
+        SELECTED_NETWORKS: ${{ inputs.selectedNetworks }}
+        CHAINLINK_IMAGE: ${{ inputs.chainlinkImage }}
+        CHAINLINK_VERSION: ${{ inputs.chainlinkVersion }}
+        UPGRADE_IMAGE: ${{ inputs.upgradeImage }}
+        UPGRADE_VERSION: ${{ inputs.upgradeVersion }}
+      run: |
+        function convert_to_toml_array() {
+          local IFS=','
+          local input_array=($1)
+          local toml_array_format="["
+
+          for element in "${input_array[@]}"; do
+              toml_array_format+="\"$element\","
+          done
+
+          toml_array_format="${toml_array_format%,}]"
+          echo "$toml_array_format"
+        }
+        
+        selected_networks=$(convert_to_toml_array "$SELECTED_NETWORKS")      
+
+        cat << EOF > config.toml
+        [Network]
+        selected_networks=$selected_networks
+        
+        [ChainlinkImage]
+        image="$CHAINLINK_IMAGE"
+        version="$CHAINLINK_VERSION"
+
+        [ChainlinkUpgradeImage]
+        image="$UPGRADE_IMAGE"
+        version="$UPGRADE_VERSION"
+        EOF
+        
+        BASE64_CONFIG_OVERRIDE=$(cat config.toml | base64 -w 0)
+        echo ::add-mask::$BASE64_CONFIG_OVERRIDE
+        echo "BASE64_CONFIG_OVERRIDE=$BASE64_CONFIG_OVERRIDE" >> $GITHUB_ENV
diff --git a/.github/actions/setup-merge-base64-config/action.yml b/.github/actions/setup-merge-base64-config/action.yml
@@ -0,0 +1,54 @@
+name: Merge Base64 Config
+description: A composite action that merges user-provided Base64-encoded config with repository's secrets
+
+inputs:
+  base64Config:
+    description: Base64-encoded config to decode
+
+runs:
+  using: composite  
+  steps:
+    - name: Add masks and export base64 config
+      shell: bash
+      run: |
+        BASE64_CONFIG_OVERRIDE=$(jq -r '.inputs.base64Config' $GITHUB_EVENT_PATH)
+        echo ::add-mask::$BASE64_CONFIG_OVERRIDE
+        echo "BASE64_CONFIG_OVERRIDE=$BASE64_CONFIG_OVERRIDE" >> $GITHUB_ENV
+
+        decoded_toml=$(echo $BASE64_CONFIG_OVERRIDE | base64 -d)
+        CHAINLINK_IMAGE=$(echo "$decoded_toml" | awk -F'=' '/^[[:space:]]*image[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2}' 2>/dev/null)
+        echo ::add-mask::$CHAINLINK_IMAGE
+        CHAINLINK_VERSION=$(echo "$decoded_toml" | awk -F'=' '/^[[:space:]]*version[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2}' 2>/dev/null)          
+        NETWORKS=$(echo "$decoded_toml" | awk -F'=' '/^[[:space:]]*selected_networks[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2}' 2>/dev/null)
+
+        if [ -n "$CHAINLINK_IMAGE" ]; then
+          echo "CHAINLINK_IMAGE=$CHAINLINK_IMAGE" >> $GITHUB_ENV
+        else
+          echo "No Chainlink Image found in base64-ed config. Exiting"
+          exit 1
+        fi
+        if [ -n "$CHAINLINK_VERSION" ]; then
+          echo "CHAINLINK_VERSION=$CHAINLINK_VERSION" >> $GITHUB_ENV
+        else
+          echo "No Chainlink Version found in base64-ed config. Exiting"
+          exit 1            
+        fi   
+        if [ -n "$NETWORKS" ]; then
+          echo "NETWORKS=$NETWORKS" >> $GITHUB_ENV           
+        fi    
+
+        # use Loki config from GH secrets and merge it with base64 input
+        cat << EOF > config.toml
+        [Logging.Loki]
+        tenant_id="$LOKI_TENANT_ID"
+        endpoint="$LOKI_URL"
+        basic_auth="$LOKI_BASIC_AUTH"
+        # legacy, you only need this to access the cloud version
+        # bearer_token="bearer_token"
+        EOF
+
+        echo "$decoded_toml" >> final_config.toml
+        cat config.toml >> final_config.toml
+        BASE64_CONFIG_OVERRIDE=$(cat final_config.toml | base64 -w 0)
+        echo ::add-mask::$BASE64_CONFIG_OVERRIDE
+        echo "BASE64_CONFIG_OVERRIDE=$BASE64_CONFIG_OVERRIDE" >> $GITHUB_ENV