Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it easier to handle asynchronously declared tests when running as a standalone executable #2253

Open
natebosch opened this issue Jul 9, 2024 · 0 comments
Labels
P3 A lower priority bug or feature request status-blocked Blocked from making progress by another (referenced) issue type-enhancement A request for a change that isn't a bug type-ux A user experience or user interface related issue

Comments

@natebosch
Copy link
Member

The test package supports running tests through the test runner with full feature support, and it also supports running a _test.dart library directly as an executable, what we call a "standalone" test suite. When running as a standalone suite none of the test runner features are available. Configuration is ignored (whether in yaml, or annotations), and no arguments can be passed (so no features that rely on arguments are available).

There are also some behavior differences. One is that with the full test runner tests can be declared in a main() async { method, and declarer will wait for the returned Future to complete before running tests. When running as a standalone file the first call to group or test marks the beginning of a synchronous block in which all tests must be declared. After that synchronous execution the tests start running and it becomes an error to declare another test. (There seems to be some looseness here, it's not strictly synchronous, but it doesn't allow arbitrary async delays)

We could investigate the feasibility of removing the restriction that tests are all declared before any starts running. I suspect this is would not be worth the effort, since the impact would cascade outside of the test runner. IDEs which use the JSON reporter output would also likely need work to handle this case, since we currently report metadata about all the tests before we run them.

We could disallow asynchronous test declaration in the full test runner so that the standalone suites don't behave differently. I think the pain this would cause users of the test runner (the vast majority of test users) much more aggregate pain than the sharp edges for standalone suites (a small minority of test users).

We could introduce an API to mark that tests will be declared asynchronously, and then an API to explicitly start running the suite. Or something like Future<void> declareTests(Future<void> Function()). I'd prefer not to have to add this to the API surface area, but if there are test suites where it's difficult to refactor and there becomes a need for them to run as standalone suites this might be our only reasonable escape hatch.

Once macros have landed we could implement some macro that is applied on main() and injects some behavior that surfaces the Future returned from an async main to the global declarer. If we land the above API the macro could use it.

See discussion in dart-lang/dart_style#1499

@natebosch natebosch added type-enhancement A request for a change that isn't a bug status-blocked Blocked from making progress by another (referenced) issue P3 A lower priority bug or feature request type-ux A user experience or user interface related issue labels Jul 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 A lower priority bug or feature request status-blocked Blocked from making progress by another (referenced) issue type-enhancement A request for a change that isn't a bug type-ux A user experience or user interface related issue
Projects
None yet
Development

No branches or pull requests

1 participant