Introduce some tests of non-determinism #60
Conversation
|
I'll need to update my runner to account for this change. I can work on that tomorrow. |
It needs to be an array of arrays with at least two elements. This will help diagnose the likely mistake of using "results" instead of "result", since it should be fairly unusual for the results of a query to be an array of two or more arrays.
| ] | ||
| ], | ||
| "properties": { | ||
| "invalid_selector": false, |
There was a problem hiding this comment.
@gregsdennis I took the liberty of tightening this up, even though it's not needed for the current PR.
| ] | ||
| ], | ||
| "properties": { | ||
| "document": false, |
There was a problem hiding this comment.
@gregsdennis I took the liberty of tightening this up, even though it's not needed for the current PR.
|
Have we considered adding a flag (e.g. I might not understand the idea fully, but if we want to allow indeterministic order somewhere deep inside of the complex result object, we could maybe introduce a list of json pointers pointing to the particular elements of the result which are allowed to be indeterministic. |
|
The non-determinism is actually only for object selection. As you can see in the new test on line 480, while the values from the object properties are selected in any order, the values are always It gets weird when you have something like {
"a": [1, 2, 3],
"b": [4, 5, 6]
}Here, you must always have 1, 2, and 3 together and in order, as well as 4, 5, and 6 together and in order, but 4, 5, and 6 could come before or after 1, 2, and 3. So it's not "in any order," it's "one of these specific orders." |
|
That said, if you'd like to propose an alternative, I'm happy to consider it. |
Yes, that's expected since your implementation is more deterministic than the spec.
Agreed. If we had a full implementation of the RFC which was deliberately non-deterministic in the way it enumerated objects, then we'd have a better chance of automatically discovering tests that should be made non-deterministic (since those tests would fail, at least some of the time). As it is, there is a risk that we'll miss some tests which should be made non-deterministics. In a sense, that's ok because we can always increase the amount of non-determinism in the CTS, so long as we don't go beyond the spec. |
I might be able to set up the tests in a way to search for objects and scramble them so that the lib pulls the properties out of order. That would introduce some non-determinism. It would only be the test runner and wouldn't affect the lib at all. (I kinda want to play with that now.) |
That would produce some of the non-determinism allowed by the RFC, but not necessarily all of it (e.g. it wouldn't reproduce the non-determinism of the query |
|
For what it's worth, the nondeterministic branch of JSON P3 adds some randomness to object iteration. After a clone (inc submodules) and After a few of runs, I've only seen one more case that needs updating, "basic, multiple selectors, wildcard and name", which is less than I was expecting. I might be missing something. |
That's really helpful, thanks for that! (BTW please would you consider adding JSON P3 to the JSONPath comparison project? A first step is to raise an issue there to add JSON P3.)
I wasn't expecting large numbers of the current tests to exhibit non-determinism as it requires a combination of factors:
(We could use these criteria to design further non-deterministic tests to plug any gaps.) |
|
@jg-rp thanks! I'll go ahead and merge this, then update that test in another PR. |
Yeah, I tried shuffling objects and it wasn't precise enough. I get false positives where it thinks a test that is deterministic shouldn't be. |
|
@glyn - thanks for this. I updated |
This is not necessarily an exhaustive set of tests of the non-deterministic features of RFC 9535. However, it establishes a convention for how to write such tests.
A non-deterministic test has a "results" key instead of a "result" key. The value corresponding to "results" is an array or possible results. If an implementation produces any one of these, the test should pass. If an implementation produces any other result, the test should fail.
Ref #9