Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DX Streamline] LOTUS_RUN_EXPENSIVE_TESTS periodically outside of PR workflow #12136

Open
BigLep opened this issue Jun 22, 2024 · 3 comments · May be fixed by #12234
Open

[DX Streamline] LOTUS_RUN_EXPENSIVE_TESTS periodically outside of PR workflow #12136

BigLep opened this issue Jun 22, 2024 · 3 comments · May be fixed by #12234
Milestone
DX-Streamline

Comments

@BigLep
Copy link
Member

BigLep commented Jun 22, 2024

Done Criteria

  1. LOTUS_RUN_EXPENSIVE_TESTS are run periodically (e.g., nightly) against master automatically. They aren't run as part of the normal PR flow because of how long they take.
  2. There is notification via github issue or Slack if there are any failures so that maintainers can try and identify the culprit change.
  3. There is a self-service URL to find the previous results of LOTUS_RUN_EXPENSIVE_TESTS.

Why Important

There are "expensive" tests that test key functionality. This is getting exasperated by NI-PoRep tests which can take 10+ minutes on CI. We want to avoid compromising on test quality because of long CI runs. We want the test coverage but not to be slowed down by it for every CI run.

User/Customer

Contributors and maintainers

Notes

  1. This was discussed in #fil-lotus-dev on 2024-06-20 in https://filecoinproject.slack.com/archives/CP50PPW2X/p1718859677050149
@BigLep BigLep added this to the DX-Streamline milestone Jun 22, 2024
@BigLep BigLep added this to FilOz Jun 22, 2024
@BigLep BigLep moved this to 📌 Triage in FilOz Jun 22, 2024
@rvagg
Copy link
Member

rvagg commented Jun 22, 2024

Currently LOTUS_RUN_EXPENSIVE_TESTS=1 is the same as CI=true, which is why I proposed that LOTUS_RUN_VERY_EXPENSIVE_TESTS introduces a new level that we would run periodically.

@rvagg rvagg mentioned this issue Jun 25, 2024
Merged
10 tasks
@rvagg
Copy link
Member

rvagg commented Jun 25, 2024 •
edited
Loading

master now includes one test that's behind LOTUS_RUN_VERY_EXPENSIVE_TESTS, so this has become a slightly higher priority now

@galargh
Copy link
Contributor

galargh commented Jun 28, 2024

I think it'd be good if there was also a way to request the run of very expensive tests on a PR (e.g. through a label) and if they run when the test itself was modified

@laurentsenta laurentsenta linked a pull request Jul 15, 2024 that will close this issue
Open
14 tasks
@laurentsenta laurentsenta mentioned this issue Jul 23, 2024
Merged
9 tasks
@rjan90 rjan90 removed the status in FilOz Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
FilOz
Status: No status
Milestone
DX-Streamline
Development

Successfully merging a pull request may close this issue.

ci: tests: adds tooling to run very expensive tests filecoin-project/lotus
3 participants
@BigLep @rvagg @galargh