[PH] Create workflow for manually dispatching performance harness runs #1378

oschwaldp-oci · 2023-07-07T16:00:24Z

Initial attempt of configurable ph run in cicd.

`platforms.yaml`

Split out Discover Platforms, Build Platforms, and Platform Matrix out into one reusable workflow. Platform Matrix is a new job that dynamically creates the matrix: platform: array from the discovered platforms or can be overridden by the input override-build-matrix. This reduces the maintenance overhead of having to update the matrix: platform arrays in the workflows anytime a platform is added or dropped.

`build_base.yaml`

Removes steps now contained in platforms.yaml workflow but takes as input p, the discovered platforms, and platform-matrix. Typically will be the output of the platforms.yaml workflow.

`build.yaml`

Updated to make use of platforms.yaml and build_base.yaml workflows.

`ph_backward_compatibility.yaml`

Updated to make use of platforms.yaml and build_base.yaml workflows.

`performance_harness_run.yaml`

A manual workflow to configure and run the Performance Harness in CICD. This workflow will attempt to reuse a build for the current head commit of the branch selected to run on, if it exists, otherwise it will build it from scratch. It has the following inputs:

Use workflow from
- Select branch to run on
Override build platform
- Drop down selection allows ubuntu20 or ubuntu22 as the build platform
Override perf harness params
- Provide any special command line arguments to pass to performance_test.py
- Default (if input field left blank): testBpOpMode
- e.g.
  - testBpOpMode --max-tps-to-test 50 --test-iteration-min-step 10 --test-iteration-duration-sec 10 --final-iterations-duration-sec 10 overrideBasicTestConfig -v --tps-limit-per-generator 25 --chain-state-db-size-mb 200
  - testBpOpMode --test-iteration-duration-sec 600 --final-iterations-duration-sec 600

spoonincode · 2023-07-10T16:03:19Z

.github/workflows/performance_harness_run.yaml

+          uses: AntelopeIO/upload-artifact-large-chunks-action@v1
+          with:
+            name: ${{github.event.inputs.platform-choice}}-build
+            path: build.tar.zst


Similar comments as #1356 (comment): we shouldn't be copy pasting all of this, we need to come up with a better approach.

@spoonincode - I have refactored the build to be a reusable workflow in PR#1356. Hoping to have that PR go into main first, then I will pull those changes into this branch and refactor the workflow to use build-base as well.

spoonincode · 2023-07-10T16:04:55Z

.github/workflows/performance_harness_run.yaml

+      - name: Run Performance Test
+        run: |
+          # https://github.com/actions/runner/issues/2033  -- need this because of full version label test looking at git revs
+          chown -R $(id -u):$(id -g) $PWD


Is the 2033 workaround really needed in this case? Unless there is a git command being run as part of the test, probably not

Addressed: 6b4f745

spoonincode · 2023-07-10T16:05:41Z

.github/workflows/performance_harness_run.yaml

        run: |
-          echo "Workflow Stub"
+          # https://github.com/actions/runner/issues/2033
+          chown -R $(id -u):$(id -g) $PWD


Similar to above -- is 2033 workaround needed here? Seems unlikely since there is just a tar

Addressed: 6b4f745

spoonincode · 2023-07-10T16:06:46Z

.github/workflows/performance_harness_run.yaml

    runs-on: ubuntu-latest
+    container:
+      image: ${{fromJSON(needs.d.outputs.p)[github.event.inputs.platform-choice].image}}
+      options: --security-opt seccomp=unconfined


Does performance harness make use of unshare? (does it come along indirectly via TestHarness or something?)

Addressed: 6b4f745

spoonincode · 2023-07-10T16:07:16Z

.github/workflows/performance_harness_run.yaml

+    needs: [v, d, Build]
+    if: always() && needs.Build.result == 'success'
+    strategy:
+      fail-fast: false


Without being a matrix, this is a noop

Addressed: 6b4f745

spoonincode · 2023-07-10T16:13:57Z

.github/workflows/performance_harness_run.yaml

+    needs: [v, d, Build]
+    if: always() && needs.Build.result == 'success'
+    strategy:
+      fail-fast: false
    runs-on: ubuntu-latest


This means it runs on a github owned runner, is that what you wanted?

@spoonincode - Correct. I was using the github owned runner for initial testing and verification of the workflow. Now that this is closer to complete, do you have a suggestion for a runner to use (enf-x86-beefy-long)? I know that most of the enf runners have time limits on them which could be problematic for performance runs depending on how a user configures the run.

…dy completed.

spoonincode · 2023-07-18T20:01:36Z

.github/workflows/build_base.yaml

    runs-on: ["self-hosted", "enf-x86-beefy"]
    container: ${{fromJSON(needs.d.outputs.p)[matrix.platform].image}}
    steps:
        - uses: actions/checkout@v3
          with:
+            ref: '${{inputs.leap-ref}}'


This seems very problematic since on first look I don't believe the workflow+platform used to build leap will match up with the version of leap checked out. You will need to do more effort in at least 2 places to get this to work correctly, I would recommend against the added complexity.

I am not sure I really had this working correctly, as you observed. I have removed this feature.

This reverts commit 47ddeb6.

This reverts commit 6c2758f.

…f available.

Use platforms in conjunction with build-base such that platforms can be run separately from build-base. Together they accomplish the same as build-base used to, but now can separate concerns between discovering and building platform knowledge from doing the actual software build. Update previous usages of build-base to now use platforms and build-base together.

These reusable workflows are intended to be called from other workflows, not necessarily dispatched individually in their own capacity.

spoonincode · 2023-07-31T14:27:25Z

.github/workflows/build.yaml

+    name: Run Platforms Workflow
+    uses: ./.github/workflows/platforms.yaml
+    with:
+      override-build-matrix: ${{github.event.inputs.platform-choice}}


There doesn't look to be a platform-choice input as part of this workflow.

I thought I had removed that. Somehow lost in transition. It is unnecessary in the build.yaml workflow as that workflow should always be expecting to use the full input from platforms.json to determine the build platforms to support. Removed here: bf95111

…n input.

spoonincode · 2023-08-02T14:54:44Z

.github/workflows/performance_harness_run.yaml

@@ -2,6 +2,16 @@ name: "Performance Harness Run"

 on:
  workflow_dispatch:
+    inputs:
+      platform-choice:
+        description: 'Override build platform'


I guess this isn't really overriding the platform but rather picking which platform

Correct. Here it is a platform selection, however down in platforms.yaml it serves to override the auto-generated platform matrix (from platforms.json). I'll update the verbiage here. Addressed: 8c47cce

spoonincode · 2023-08-02T14:55:14Z

.github/workflows/performance_harness_run.yaml

+      - name: Setup Input Params
+        id: overrides
+        env:
+          GH_TOKEN: ${{secrets.GITHUB_TOKEN}}


this used any where?

Addressed: 8c47cce

spoonincode · 2023-08-02T15:57:25Z

.github/workflows/platforms.yaml

+        id: check-matrix
+        run: |
+          echo '${{ steps.pm-results.outputs.platform-matrix }}'
+          echo '${{ steps.parse-pm.outputs.result }}'


quite a few echos in this workflow that look kind of like debugging, did you mean to leave them in?

Good catch. Thought I had caught all of these and removed them. Thanks.

Addressed: cd872b3

spoonincode · 2023-08-02T17:47:58Z

.github/workflows/performance_harness_run.yaml

+            target: ${{github.sha}}
+            artifact-name: ${{github.event.inputs.platform-choice}}-build
+            fail-on-missing-target: false
+            token: ${{github.token}}


with v3 you don't need this line, if you want to save a line

Done: d50381b

spoonincode · 2023-08-02T17:48:55Z

.github/workflows/performance_harness_run.yaml

+  tests:
+    name: Tests
+    needs: [v, platforms, reuse-build, build-base]
+    if: always() && needs.platforms.result == 'success' && (needs.build-base.result == 'success' || needs.reuse-build.result == 'success')


we really need to find a way to get #283 in good shape to avoid these tricky always() chains

Initial attempt of configurable ph run in cicd.

f7f246a

oschwaldp-oci self-assigned this Jul 7, 2023

oschwaldp-oci added the OCI Work exclusive to OCI team label Jul 7, 2023

oschwaldp-oci added 7 commits July 7, 2023 11:13

Fix yaml file syntax. Minor cleanup.

78564fa

Fix yaml file syntax.

7518f78

Fix yaml file syntax 'uses' already defined.

1f2ca01

Add dependency needs.

975426a

Pre-calculate the report location for use in upload step.

947770e

Use normal upload-artifact action.

8f01104

Try finding and uploading report via wildcard.

e3ae3a3

oschwaldp-oci marked this pull request as ready for review July 10, 2023 16:00

oschwaldp-oci requested review from heifner and spoonincode July 10, 2023 16:01

spoonincode requested changes Jul 10, 2023

View reviewed changes

oschwaldp-oci added 16 commits July 10, 2023 14:20

Merge branch 'main' into ph-manual-workflow

dc92ab5

Some cleanup.

6b4f745

Merge branch 'main' into ph-manual-workflow

8919364

Try to automatically parse platform matrix in script.

feffc03

Attempt to use the json object directly.

be9452b

Attempt to fix github variable reference in script.

754638d

Attempt to use constructed platform matrix in build job.

cb73a84

Put matrix-out into github output for use later.

b0075c0

Try quotes.

a414b8c

Try using fromJSON parsing.

3f13595

Debugging output.

a83266b

Don't needs backslashes to break lines up.

158ad55

Add input to override build platform matrix.

ba738b8

Use fromJSON.

282cfe0

Try again.

1de5698

Try again.

c7695c0

oschwaldp-oci added 3 commits July 18, 2023 12:24

Try to make build step optional if build for current commit has alrea…

1b9f64a

…dy completed.

Pipe leap-ref selection/override through workflows.

6c2758f

Fix spacing.

47ddeb6

spoonincode reviewed Jul 18, 2023

View reviewed changes

oschwaldp-oci added 12 commits July 18, 2023 15:16

Revert "Fix spacing."

4f3c5ef

This reverts commit 47ddeb6.

Revert "Pipe leap-ref selection/override through workflows."

f49f8c9

This reverts commit 6c2758f.

Remove attempt to override leap ref for this workflow.

c597816

Merge branch 'main' into ph-manual-workflow

7670690

Try using asset-artifact-download-action@v3 to pull down past build i…

e5139d6

…f available.

Fix file name in asset-artifact-download-action

1a96850

Remove workflow_dispatch from reusable workflows.

13424e2

These reusable workflows are intended to be called from other workflows, not necessarily dispatched individually in their own capacity.

Fix call to build-base with proper new inputs.

78b16c5

Merge branch 'main' into ph-manual-workflow

b30be9c

Try using something other than empty string.

4cb312e

Get the output directly from the step output.

45b6c77

oschwaldp-oci requested a review from spoonincode July 28, 2023 21:09

spoonincode reviewed Jul 31, 2023

View reviewed changes

Remove override input as build will always use the full platforms.jso…

bf95111

…n input.

spoonincode reviewed Aug 2, 2023

View reviewed changes

Address peer review comments.

8c47cce

spoonincode reviewed Aug 2, 2023

View reviewed changes

Remove debugging step and unnecessary echo.

cd872b3

spoonincode approved these changes Aug 2, 2023

View reviewed changes

heifner approved these changes Aug 2, 2023

View reviewed changes

oschwaldp-oci added 2 commits August 2, 2023 13:19

Remove unnecessary token input.

d50381b

Merge branch 'main' into ph-manual-workflow

b710831

oschwaldp-oci merged commit 1e632c9 into main Aug 2, 2023

oschwaldp-oci deleted the ph-manual-workflow branch August 2, 2023 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PH] Create workflow for manually dispatching performance harness runs #1378

[PH] Create workflow for manually dispatching performance harness runs #1378

oschwaldp-oci commented Jul 7, 2023 •

edited

Loading

spoonincode Jul 10, 2023

oschwaldp-oci Jul 12, 2023

spoonincode Jul 10, 2023

oschwaldp-oci Jul 10, 2023

spoonincode Jul 10, 2023

oschwaldp-oci Jul 10, 2023

spoonincode Jul 10, 2023

oschwaldp-oci Jul 10, 2023

spoonincode Jul 10, 2023

oschwaldp-oci Jul 10, 2023

spoonincode Jul 10, 2023

oschwaldp-oci Jul 11, 2023

spoonincode Jul 18, 2023

oschwaldp-oci Jul 18, 2023

spoonincode Jul 31, 2023

oschwaldp-oci Jul 31, 2023

spoonincode Aug 2, 2023

oschwaldp-oci Aug 2, 2023

spoonincode Aug 2, 2023

oschwaldp-oci Aug 2, 2023

spoonincode Aug 2, 2023

oschwaldp-oci Aug 2, 2023

spoonincode Aug 2, 2023

oschwaldp-oci Aug 2, 2023

spoonincode Aug 2, 2023

[PH] Create workflow for manually dispatching performance harness runs #1378

[PH] Create workflow for manually dispatching performance harness runs #1378

Conversation

oschwaldp-oci commented Jul 7, 2023 • edited Loading

platforms.yaml

build_base.yaml

build.yaml

ph_backward_compatibility.yaml

performance_harness_run.yaml

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oschwaldp-oci commented Jul 7, 2023 •

edited

Loading

`platforms.yaml`

`build_base.yaml`

`build.yaml`

`ph_backward_compatibility.yaml`

`performance_harness_run.yaml`