Avoid running functional tests for PRs whenever it makes sense

[iholder101]

Currently in Kubevirt, every PR needs to run and pass the complete test suite in order to merge.

However, for some PRs (e.g. non-code PRs) there's no value in running any functional tests at all, therefore CI resources are being wasted with no value whatsoever. As an example, a PR making someone an approver [1], which only requires changing a text file, should be tested. This leads to a waste of costly CI resources and should be avoided, which becomes even worse when tests are flakie and need to be re-tested for multiple times.

One possible approach to solving this is to trigger CI only if files under certain paths are being changed.

An even better situation would be to avoid running CI on more sophisticated scenarios, like changing a variable name, comments, file names, and other non-logical code refactoring.

[1] kubevirt/kubevirt#9900

[iholder101]

/cc @dhiller @jean-edouard @fossedihelm

[dhiller]

/assign

[dhiller]

/kind enhancement

[dhiller]

/area prow

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[iholder101]

/remove-lifecycle stale

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dhiller]

/reopen
/lifecycle frozen

[kubevirt-bot]

@dhiller: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dhiller]

Short update: we are still investigating how to tackle this. @oshoval contributed the phased plugin recently, which enables us holding back on testing until the PR has reached merge queue state. We are still investigating how to find a proper solution wrt run_if_changed/skip_if_only_changed, since it currently interferes with how the phased plugin works. Instead, a pragmatic solution has been merged, aborting functests directly if only certain files have been changed.

[iholder101]

Short update: we are still investigating how to tackle this. @oshoval contributed the phased plugin recently, which enables us holding back on testing until the PR has reached merge queue state. We are still investigating how to find a proper solution wrt run_if_changed/skip_if_only_changed, since it currently interferes with how the phased plugin works. Instead, a pragmatic solution has been merged, aborting functests directly if only certain files have been changed.

Great news! Thanks for the update and for pushing this subject forward!

Can we abort functests for files with certain suffixes? e.g. .md, .txt, etc?