Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mantle/kola/harness.go: add snooze support for tests #2307

Merged
merged 2 commits into from
Jul 22, 2021

Conversation

saqibali-2k
Copy link
Member

Add snooze support for tests as described here.

@saqibali-2k saqibali-2k linked an issue Jul 20, 2021 that may be closed by this pull request
Copy link
Member

@cgwalters cgwalters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks sane to me! Just one minor comment.

mantle/kola/harness.go Outdated Show resolved Hide resolved
@dustymabe
Copy link
Member

nice work! Is there any way possible we can add some extra logging (not debug) to the console when a test gets snoozed? Since the code now has differing behavior based on time (i.e. tests ran fine yesterday, and today they are failing, but nothing changed!) I think it's important that we mention very loudly when we don't run a test because of the policy/configuration we've set.

@saqibali-2k
Copy link
Member Author

saqibali-2k commented Jul 21, 2021

Is there any way possible we can add some extra logging (not debug) to the console when a test gets snoozed?

We can move the if obj.SnoozeDate != "" clause to the bottom of the loop, and add a print statement at the end of the clause. So it should look something like this:

		if len(obj.Arches) > 0 && !hasString(arch, obj.Arches) {
			continue
		}
		if len(stream) > 0 && len(obj.Streams) > 0 && !hasString(stream, obj.Streams) {
			continue
		}

		if obj.SnoozeDate != "" {
			snoozeDate, err := time.Parse(snoozeFormat, obj.SnoozeDate)
			if err != nil {
				return err
			} else if today.After(snoozeDate) {
				continue
			}
                        fmt.Printf("🕒  Snoozing test until %s \n", snoozeDate.Format("Jan 02 2006"))
		}

		fmt.Printf("⚠️  Skipping kola test pattern \"%s\":\n", obj.Pattern)
		fmt.Printf("  👉 %s\n", obj.Tracker)
		DenylistedTests = append(DenylistedTests, obj.Pattern)

And the output will look like:

🕒  Snoozing test until Jun 07 2022
⚠️  Skipping kola test pattern "fcos.internet":
  👉 https://github.com/coreos/coreos-assembler/pull/1478

@dustymabe
Copy link
Member

🕒  Snoozing test until Jun 07 2022
⚠️  Skipping kola test pattern "fcos.internet":
  👉 https://github.com/coreos/coreos-assembler/pull/1478

Looks mostly good. Maybe just add the test name in there too just to
make it completely obvious:

🕒  Snoozing test pattern "fcos.internet" until Jun 07 2022
⚠️  Skipping kola test pattern "fcos.internet":
 👉 https://github.com/coreos/coreos-assembler/pull/1478

Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool stuff! Some minor comments, but LGTM overall.

docs/kola.md Outdated Show resolved Hide resolved
docs/kola.md Outdated Show resolved Hide resolved
mantle/kola/harness.go Outdated Show resolved Hide resolved
mantle/kola/harness.go Outdated Show resolved Hide resolved
Copy link
Member

@dustymabe dustymabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@saqibali-2k saqibali-2k force-pushed the snooze-support branch 3 times, most recently from 731974d to ffff0d7 Compare July 22, 2021 18:14
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Optional nits only, happy to defer.

mantle/kola/harness.go Outdated Show resolved Hide resolved
mantle/kola/harness.go Outdated Show resolved Hide resolved
Added an option to snooze tests until a given date. This snooze date can be specified in kola-denylist.yaml with key "snooze" and format YYYY-MM-DD.
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Will let someone else have a final look and press the button.

Copy link
Member

@dustymabe dustymabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Nice work @saqibali-2k

@dustymabe dustymabe enabled auto-merge (rebase) July 22, 2021 20:45
@dustymabe dustymabe merged commit f5e6842 into coreos:main Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add support for snoozing tests
4 participants