Make it easier to handle asynchronously declared tests when running as a standalone executable #2253
Labels
P3
A lower priority bug or feature request
status-blocked
Blocked from making progress by another (referenced) issue
type-enhancement
A request for a change that isn't a bug
type-ux
A user experience or user interface related issue
The test package supports running tests through the test runner with full feature support, and it also supports running a
_test.dart
library directly as an executable, what we call a "standalone" test suite. When running as a standalone suite none of the test runner features are available. Configuration is ignored (whether in yaml, or annotations), and no arguments can be passed (so no features that rely on arguments are available).There are also some behavior differences. One is that with the full test runner tests can be declared in a
main() async {
method, and declarer will wait for the returned Future to complete before running tests. When running as a standalone file the first call togroup
ortest
marks the beginning of a synchronous block in which all tests must be declared. After that synchronous execution the tests start running and it becomes an error to declare another test. (There seems to be some looseness here, it's not strictly synchronous, but it doesn't allow arbitrary async delays)We could investigate the feasibility of removing the restriction that tests are all declared before any starts running. I suspect this is would not be worth the effort, since the impact would cascade outside of the test runner. IDEs which use the JSON reporter output would also likely need work to handle this case, since we currently report metadata about all the tests before we run them.
We could disallow asynchronous test declaration in the full test runner so that the standalone suites don't behave differently. I think the pain this would cause users of the test runner (the vast majority of test users) much more aggregate pain than the sharp edges for standalone suites (a small minority of test users).
We could introduce an API to mark that tests will be declared asynchronously, and then an API to explicitly start running the suite. Or something like
Future<void> declareTests(Future<void> Function())
. I'd prefer not to have to add this to the API surface area, but if there are test suites where it's difficult to refactor and there becomes a need for them to run as standalone suites this might be our only reasonable escape hatch.Once macros have landed we could implement some macro that is applied on
main()
and injects some behavior that surfaces the Future returned from an asyncmain
to the global declarer. If we land the above API the macro could use it.See discussion in dart-lang/dart_style#1499
The text was updated successfully, but these errors were encountered: