[TT-1218] try modified compatibility pipeline (#13443)

* try modified compatibility pipeline

* don't wait with matrix generation for image to be built

* fix outputs names

* use images instead of tags

* fix array population

* construct array differently

* fix & debug

* fix echos

* try running tests

* fix path

* fix paths and run command

* more tests + change names + slack notification fix

* test ignored tags, add more tests

* fix slack + more tests

* fix script

* fix ignored tags array

* add vrfv2plus tests, commenout out vrfv2 tests that always fail

* disable vrfv2plus tests for besu and nethermind, modify pipeline triggers

* adjust triggers

* another trigger update

* not trigger should work as expected

* fix again

* yet another fix + add keeper test

* debug 1

* debug 2

* fix output target

* try different condition

* here comes another one...

* fix da condition again, maybe forevah?

* another attempt

* fix yaml

* add go script to check whether new releases created N days ago, update pipeline

* a bit of debug

* fix bash expansion

* return latest tag from go script + get correct ref when building chainlink image depending on the trigger

* fix workflow add depedency

* fix chainlink version that's passed to build

* fix ... mistake

* use correct chainlink image tag in tests

* allow to pass evm implementations and image count in workflow dispatch

* support for custom test/evm implementation/versions input

* fix custom input parsing

* use go script to get latest images, another fix of input decoding

* fix workflow conditions

* further fix the workflow

* remove one more invalid comment

* remove profile from latest_images script

* set aws region from secret

* modify latest images script, so that it ignored tags don't count towards latest images number

* add tests to compatibility script

* refactor latest images and add tests

* add tests for new_release script

* run only Link Billing vrfv2plus test

* fix latest images script

* Add one more test

* fix vrfv2plus regex

* use semver matching when fetching latest images

* fix constraints + warning message

* do not fail slack notification if not all products run

* more tests + variable to force check if image exists

* fix workflow lints

* remove scripts, use CLI tools from CTF

* add go/bin to path

* fix path

* bump go-ethereum to test the pipeline

* bump correct go.mod

* restore go-ethereum version, remove bash script, update compatibility pipeline

* fix pipeline

* remove evm compatiblity json

* add changeset

* fix changeset

* try success() instead of always()

* fix commit selection for workflow_dispatch

* run tests only if build-chainlink succeeds

* WIP: print aggregated results

* try result parsing scripts

* fix needs + script cmd

* run small test

* run small test 2

* fix typos

* do not save empty results

* fix file location + early exit

* do not panic if key is not present

* add concurrency guard

* fix output file name

* fix passing of table

* fix thread link

* fix result printing

* exit early if no results are found

* try wrapping summary table in markdown

* try what happens if there are no results

* fix slack notification part

* remove invalid character

* get results of current workflow

* print results of cron, runlog and flux tests as well + test newer tool version

* do not panic if there no results

* fix aggregated results printing condition

* do not output null, if there are no results

* early exit for asciitable

* remove debug from the pipeline

* check if base64 input is valid

* fix the json validity check

* provide valid json example

* try again with valid example

* fix indentation

* remove whitespaces before and after EOF

* use standalone tools in compat pipeline, modify notify action to also use them

* fix tool name + var name

* fix syntax + use newer asctiitable version

* fix tool name

* save results to input.json

* fix regex passed to notify job

* another slack notification fix

* do not post anything for skipped tests

* do not post, if there are no results

* remove incorrect line

* bump asciitable version

* remove comment

* remove unused action

* adjust live testnets workflow's regex to match new format expected by notify slack job

* bump down go-ethereum in tests

* bump down go-ethereum in core/scripts

.changeset/pretty-dancers-pull.md

@@ -0,0 +1,5 @@
+---
+"chainlink": patch
+---
+
+#changed Expand EVM implementation compatibility pipeline

.github/actions/build-chainlink-image/action.yml

@@ -18,12 +18,16 @@ inputs:
   dep_evm_sha:
     description: The chainlink-evm commit sha to use in go deps
     required: false
+  check_image_exists:
+    description: "Check if the image exists in ECR before building"
+    required: false
+    default: 'false'
 
 runs:
   using: composite
   steps:
     - name: Check if image exists
-      if: ${{ inputs.dep_evm_sha != '' }}
+      if: ${{ inputs.dep_evm_sha != '' || inputs.check_image_exists == 'true'}}
       id: check-image
       uses: smartcontractkit/chainlink-github-actions/docker/image-exists@fc3e0df622521019f50d772726d6bf8dc919dd38 # v2.3.19
       with:

.github/actions/notify-slack-jobs-result/action.yml

@@ -25,6 +25,9 @@ inputs:
   slack_thread_ts:
     description: "The Slack thread timestamp to post the message to, handy for keeping multiple related results in a single thread"
     required: false
+  base64_parsed_results:
+    description: "Base64 encoded parsed results to use"
+    required: false
 
 runs:
   using: composite
@@ -33,43 +36,54 @@ runs:
       shell: bash
       id: test-results
       run: |
-        # I feel like there's some clever, fully jq way to do this, but I ain't got the motivation to figure it out
-        echo "Querying test results at https://api.github.com/repos/${{inputs.github_repository}}/actions/runs/${{ inputs.workflow_run_id }}/jobs"
-
-        # we can get a maximum of 100 jobs per page, after that we need to start using pagination
-        PARSED_RESULTS=$(curl \
-        -H "Authorization: Bearer ${{ inputs.github_token }}" \
-        'https://api.github.com/repos/${{inputs.github_repository}}/actions/runs/${{ inputs.workflow_run_id }}/jobs?per_page=100' \
-        | jq -r --arg pattern "${{ inputs.github_job_name_regex }}" '.jobs[]
-        | select(.name | test($pattern)) as $job
-        | $job.steps[]
-        | select(.name == "Run Tests")
-        | { conclusion: (if .conclusion == "success" then ":white_check_mark:" else ":x:" end), cap: ("*" + ($job.name | capture($pattern).cap) + "*"), html_url: $job.html_url }')
-
+        if [ -n "${{ inputs.base64_parsed_results }}" ]; then
+          echo "Using base64 parsed results"
+          PARSED_RESULTS=$(echo "${{ inputs.base64_parsed_results }}" | base64 -d)
+        else
+          go install github.com/smartcontractkit/chainlink-testing-framework/tools/[email protected]
+          PATH=$PATH:$(go env GOPATH)/bin
+          export PATH
+        
+          workflowresultparser -workflowRunID ${{ inputs.workflow_run_id }} -githubToken ${{ inputs.github_token }} -githubRepo "${{ inputs.github_repository }}" -jobNameRegex "${{ inputs.github_job_name_regex }}" -outputFile=output.json
+        
+          if [ ! -f output.json ]; then
+            PARSED_RESULTS='""'
+          else 
+            PARSED_RESULTS=$(cat output.json | jq -c "select(has(\"results\")) | .results[]")
+          fi
+        
+        fi
+        
         echo "Parsed Results:"
         echo $PARSED_RESULTS
-
+        
         ALL_SUCCESS=true
-        echo "Checking for failures"
-        echo "$PARSED_RESULTS" | jq -s | jq -r '.[] | select(.conclusion != ":white_check_mark:")'
-        for row in $(echo "$PARSED_RESULTS" | jq -s | jq -r '.[] | select(.conclusion != ":white_check_mark:")'); do
-            ALL_SUCCESS=false
-            break
-        done
-        echo "Success: $ALL_SUCCESS"
-
-        echo all_success=$ALL_SUCCESS >> $GITHUB_OUTPUT
-
-        FORMATTED_RESULTS=$(echo $PARSED_RESULTS | jq -s '[.[]
-        | {
-          conclusion: .conclusion,
-          cap: .cap,
-          html_url: .html_url
-          }
-        ]
-        | map("{\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": \"<\(.html_url)|\(.cap)>: \(.conclusion)\"}}")
-        | join(",")')
-
+        
+        if [ "$PARSED_RESULTS" != '""' ]; then
+          echo "Checking for failures"
+          echo "$PARSED_RESULTS" | jq -s | jq -r '.[] | select(.conclusion != ":white_check_mark:")'
+          for row in $(echo "$PARSED_RESULTS" | jq -s | jq -r '.[] | select(.conclusion != ":white_check_mark:")'); do
+              ALL_SUCCESS=false
+              break
+          done
+          echo "Success: $ALL_SUCCESS"
+  
+          echo all_success=$ALL_SUCCESS >> $GITHUB_OUTPUT
+
+          FORMATTED_RESULTS=$(echo $PARSED_RESULTS | jq -s '[.[]
+          | {
+            conclusion: .conclusion,
+            cap: .cap,
+            html_url: .html_url
+            }
+          ]
+          | map("{\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": \"<\(.html_url)|\(.cap)>: \(.conclusion)\"}}")
+          | join(",")')
+        else
+          echo "Nothing to post, no results found"
+          exit 0
+        fi
+
         echo "Formatted Results:"
         echo $FORMATTED_RESULTS
 
@@ -82,6 +96,7 @@ runs:
         echo results=$CLEAN_RESULTS >> $GITHUB_OUTPUT
     - name: Post Results
       uses: slackapi/slack-github-action@6c661ce58804a1a20f6dc5fbee7f0381b469e001 # v1.25.0
+      if: steps.test-results.outputs.results != ''
       env:
         SLACK_BOT_TOKEN: ${{ inputs.slack_bot_token }}
       with:
@@ -108,4 +123,4 @@ runs:
                 ]
               }
             ]
-            }
+          }