Ability to uniquely identify a parameterized test

[stephencelis]

Description

Test is identifiable, which is handy to distinguish between tests and avoid letting "global" state to bleed between tests since they can be bucketed by Test.ID. Unfortunately, this breaks when it comes to parameterized tests, because each parameterized case is the same test and thus has the same identity.

Expected behavior

Ideally, Test.ID would incorporate the SPI-internal Test.Case.ID into its identity, or Test.Case.ID would be publicly accessible.

Actual behavior

Currently there's no way to access a parameterized test case's identity because it's SPI.

Steps to reproduce

No response

swift-testing version/commit hash

2e9df4f

Swift & OS version (output of swift --version && uname -a)

n/a

[grynspan]

A test is identified by its ID, but a test may have multiple test cases, which have their own unique IDs derived from the test's ID and their arguments.

So the test's ID, as currently defined, is correct—but we do need to expose a way to uniquely identify a specific case from that test. We've left that out of the API surface for now because it's a hard problem to solve given that parameters to a test can be of nearly any type. I believe @stmontgomery has some thoughts on how we might resolve that issue.

[stephencelis]

Yup, I explored the SPI internals and it seems to failably depend on encodability (or a custom Testing protocol that encodes), this seems fine, but of course if you're not using a type that can be encoded then you lose that identity.

It may be a silly question, but are there times when source location doesn't suffice for identity? And in the case of parameterized tests, could the parameter index be used to identify a case?

[grynspan]

The source location is sufficient to identify a test uniquely in a given test run; we include the fully-qualified name of the test function or suite type in its ID to improve readability for humans and to avoid ambiguity if source code is moved around over time.

However, test cases may not map back to source locations that are visible at compile time. Consider these contrived, but valid, test functions:

@Test(arguments: [a, b, c, d, e].map(\.description))
func testDescription(_ desc: String) { ... }

@Test(arguments: SomeEnum.allCases)
func testEnumCase(_ ec: SomeEnum) { ... }

@Test(arguments: await downloadCasesFromInternet())
func testDownloadedValue(_ value: Value) { ... }

In these cases, there is no source location information we could use to identify the cases. In fact, the only source location information we might be able to derive would be from bare array or dictionary literals. That's something we want to build, mind you.

Indices are tempting but indices are not stable values for many kinds of collection. They're stable for arrays, but not stable for dictionaries, sets, AllCases (in practice stable, but not guaranteed), data acquired asynchronously…

So, for test cases with identity (conforming to Identifiable, Codable, or a few other useful protocols) we know how to build test case IDs; for test cases without identity but with source location information, we can synthesize reasonable test case IDs; for test cases yielded from an array, we can use indices; and then there's the vast ocean of test cases that don't fall into either category, and that's where the hard part lies.

I hope that makes sense!

[stmontgomery]

Thanks for the issue @stephencelis! I definitely do plan to work more on this and the eventual goal is to make Test.Case.ID public and have Test.Case conform to Identifiable.

The biggest question to solve right now is how to model arguments which don't conform to one of the supported protocols (Codable, Identifiable, etc.) and attempt to provide some kind of unique identifier for them. We don't necessarily need the identifier for such values to be stable across successive runs/executions, and likely cannot ensure that. But we want them to at least be unique within a given run, so they can be distinguished when analyzing the results of that run.

One idea I've had so far is to fall back to a random identifier for these values, and make Test.Case.ID use an enum or otherwise represent which values have stable identity and which don't (and thus use a random identifier). Tools which integrate with Swift Testing would be able to key off this to disable certain features like the ability to re-run those arguments. They could additionally include the index of the argument in its collection, but the index would only be considered "advisory" and not part of the actual identifier, because (like @grynspan mentioned) index is not stable for all collection types.

[stephencelis]

Thanks to both of you for the insight! A random identifier makes sense, and I agree that it seems unlikely that you can guarantee uniqueness across runs. At the end of the day, default identifiable identity for objects is their object identifier, which will never be unique across runs, so stable identity seems like a desire that should simply degrade gracefully.

[stmontgomery]

Tracked internally as rdar://119522099