Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include DRAFT 2020-12 to serde benchmark #63

Merged
merged 4 commits into from
Nov 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,23 +22,26 @@ Adding a new validator implementation is relatively straight forward and very we
1. First, take a look at the [micro-site][micro-site], as it gives some explanation of what is being tested.
2. Clone the repo and pull it down locally, creating your own branch to work in.
3. Add necessary dependencies to [build.gradle.kts](build.gradle.kts).
4. Add a new implementation of [Implementation](src/main/java/org/creekservice/kafka/test/perf/implementations/Implementation.java)
4. Ensure GitHub's Dependabot will update the new dependency version.
This will ensure the site updates when new versions are released.
See the [Dependabot version updates docs](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates).
5. Add a new implementation of [Implementation](src/main/java/org/creekservice/kafka/test/perf/implementations/Implementation.java)
to the [main implementations](src/main/java/org/creekservice/kafka/test/perf/implementations) package for the new validator library.
See JavaDocs and other implementations for help.
5. Add a unit test class for your new implementation to the [test implementations](src/test/java/org/creekservice/kafka/test/perf/implementations) package.
6. Add a unit test class for your new implementation to the [test implementations](src/test/java/org/creekservice/kafka/test/perf/implementations) package.
This should subtype [ImplementationTest.java](src/test/java/org/creekservice/kafka/test/perf/implementations/ImplementationTest.java).
The unit test class needs to content. See other implementations for examples.
Ensure tests pass!
6. Register your new Implementation type in [Implementations.java](src/main/java/org/creekservice/kafka/test/perf/implementations/Implementations.java).
7. Register your new Implementation type in [Implementations.java](src/main/java/org/creekservice/kafka/test/perf/implementations/Implementations.java).
This will ensure the new implementation is included in the docs and included in the functional test
7. Manually add appropriate benchmark methods to [JsonSerdeBenchmark.java](src/main/java/org/creekservice/kafka/test/perf/performance/JsonSerdeBenchmark.java)
8. Manually add appropriate benchmark methods to [JsonSerdeBenchmark.java](src/main/java/org/creekservice/kafka/test/perf/performance/JsonSerdeBenchmark.java)
and [JsonValidateBenchmark.java](src/main/java/org/creekservice/kafka/test/perf/performance/JsonValidateBenchmark.java).
This is currently manual as JMH library does provide a way to generate these automatically.
There should be one test per supported draft version. See JavaDocs and the other methods in these classes for examples.
8. Run `./gradlew` to format your code, perform static analysis and run the tests.
9. Run `./gradlew` to format your code, perform static analysis and run the tests.
Ensure this passes!
9. Follow [these instructions](docs) to build and view the website, and ensure your new implementation data is included in tables and charts.
10. Raise a PR with your changes.
10. Follow [these instructions](docs) to build and view the website, and ensure your new implementation data is included in tables and charts.
11. Raise a PR with your changes.

### Running things locally

Expand Down
6 changes: 6 additions & 0 deletions config/spotbugs/suppressions.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,10 @@
<Bug pattern="PI_DO_NOT_REUSE_PUBLIC_IDENTIFIERS_CLASS_NAMES"/>
</Match>

<Match>
<!-- Disable this check as throwing from a constructor is a common pattern and totally fine as long as no finalizer is used. -->
<!-- Finalizers are a dead pattern -->
<Bug pattern="CT_CONSTRUCTOR_THROW"/>
</Match>

</FindBugsFilter>
15 changes: 9 additions & 6 deletions docs/_docs/3. performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ The first of these benchmark covers a wide range of JSON schema functionality, w
real-world example, using a small common subset of functionality, in the context of using schema validated JSON
as a serialization format. Combined, these should give a good comparison of performance.

**Note:**
The benchmarks are run on GitHub's own infrastructure. These may not be dedicated machines, which can influence performance results.
{: .notice--warning}

### JSON schema test suite benchmark

This benchmark measures the average time taken to run through all _positive_ test cases in the standard
Expand Down Expand Up @@ -92,18 +96,17 @@ for comparison.

The serialized form is roughly 1KB of JSON, and the schema is roughly 2KB.

The preferred draft specification for this benchmark is `DRAFT 7`. However, not all implementations support this.
Where an implementation does not support `DRAFT 7`, it is tested with `DRAFT 2020-12`.
[Task 59](https://github.com/creek-service/json-schema-validation-comparison/issues/59) will change this to output results for both.
Rather than test every supported schema version, the benchmark covers `DRAFT 7` and `DRAFT 2020-12`, which covers
all currently implementations, at least once.

The schema file for `DRAFT 2020-12` can be found [here][2020-schema], and for `DRAFT 7` [here][7-schema].

Each of the following graphs compares the average time it took each implementation to serialize & validate,
then validate & deserialize the simple Java object, with the following caveats:

**Note:**
As different implementations are tested using different versions of the schema specification,
which may be more or less rich than other versions, comparison across specification versions may be misleading.
Newer schema versions are more feature rich, and this can come at a cost.
Comparison of different implementations across specification versions may be misleading.
{: .notice--warning}

<div>
Expand Down Expand Up @@ -154,7 +157,7 @@ which may be more or less rich than other versions, comparison across specificat
plugins: {
title: {
display: true,
text: draft + ' ' + title + ' (less is better)'
text: draft + ' ' + title + ' (lower is better)'
},
legend: {
display: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,10 @@ private static void runBenchmarks(final String[] suppliedArgs) throws IOExceptio
"json",
// To a named file
"-rff",
JSON_RESULTS.toString()
JSON_RESULTS.toString(),
// Fail on Error
"-foe",
"true"
};

final String[] allArgs = new String[suppliedArgs.length + additionalArgs.length];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,4 @@ private Validator validator(final SchemaSpec spec, final AdditionalSchemas addit
throw new RuntimeException("Unsupported Spec:" + spec);
}
}

// Final, empty finalize method stops spotbugs CT_CONSTRUCTOR_THROW
// Can be moved to base type after https://github.com/spotbugs/spotbugs/issues/2665
@Override
@SuppressWarnings({"deprecation", "Finalize"})
protected final void finalize() {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -179,12 +179,6 @@ public Set<SchemaSpec> supported() {
public String color() {
return "rgb(" + color.getRed() + "," + color.getGreen() + "," + color.getBlue() + ")";
}

// Final, empty finalize method stops spotbugs CT_CONSTRUCTOR_THROW
// Can be moved to base type after https://github.com/spotbugs/spotbugs/issues/2665
@Override
@SuppressWarnings({"deprecation", "Finalize"})
protected final void finalize() {}
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ public class JacksonImplementation implements Implementation {
"Jackson",
Language.Java,
Licence.Apache_v2_0,
Set.of(SchemaSpec.DRAFT_07),
Set.of(SchemaSpec.DRAFT_07, SchemaSpec.DRAFT_2020_12),
"https://github.com/FasterXML/jackson-core",
new Color(20, 84, 166));

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,10 +149,4 @@ private SpecVersion schemaVersion(final SchemaSpec spec) {
}
return ver;
}

// Final, empty finalize method stops spotbugs CT_CONSTRUCTOR_THROW
// Can be moved to base type after https://github.com/spotbugs/spotbugs/issues/2665
@Override
@SuppressWarnings({"deprecation", "Finalize"})
protected final void finalize() {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -143,10 +143,4 @@ private Schema parseSchema(
throw new RuntimeException(e);
}
}

// Final, empty finalize method stops spotbugs CT_CONSTRUCTOR_THROW
// Can be moved to base type after https://github.com/spotbugs/spotbugs/issues/2665
@Override
@SuppressWarnings({"deprecation", "Finalize"})
protected final void finalize() {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@
* use the basic JSON schema features: primitives, enums, arrays, polymorphic types and length
* assertions. This can be extended in the future it needed.
*
* <p>The preferred Schema draft is Draft_7. Draft_2020_12 will be used where implementations do not
* support 7.
* <p>Benchmark methods should be added for Draft_7 and Draft_2020_12 for each implementation that
* supports them.
*/
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(MICROSECONDS)
Expand All @@ -80,7 +80,12 @@ public JacksonState() {

@Benchmark
public TestModel measureDraft_07_Jackson(final JacksonState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

@Benchmark
public TestModel measureDraft_2020_12_Jackson(final JacksonState impl, final ModelState model) {
return impl.roundTrip(model, SchemaSpec.DRAFT_2020_12);
}

public static class MedeiaState extends ImplementationState {
Expand All @@ -91,7 +96,7 @@ public MedeiaState() {

@Benchmark
public TestModel measureDraft_07_Medeia(final MedeiaState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

public static class EveritState extends ImplementationState {
Expand All @@ -102,7 +107,7 @@ public EveritState() {

@Benchmark
public TestModel measureDraft_07_Everit(final EveritState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

public static class SkemaState extends ImplementationState {
Expand All @@ -113,7 +118,7 @@ public SkemaState() {

@Benchmark
public TestModel measureDraft_2020_12_Skema(final SkemaState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_2020_12);
}

public static class VertxState extends ImplementationState {
Expand All @@ -124,7 +129,12 @@ public VertxState() {

@Benchmark
public TestModel measureDraft_07_Vertx(final VertxState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

@Benchmark
public TestModel measureDraft_2020_12_Vertx(final VertxState impl, final ModelState model) {
return impl.roundTrip(model, SchemaSpec.DRAFT_2020_12);
}

public static class SchemaFriendState extends ImplementationState {
Expand All @@ -136,7 +146,13 @@ public SchemaFriendState() {
@Benchmark
public TestModel measureDraft_07_SchemaFriend(
final SchemaFriendState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

@Benchmark
public TestModel measureDraft_2020_12_SchemaFriend(
final SchemaFriendState impl, final ModelState model) {
return impl.roundTrip(model, SchemaSpec.DRAFT_2020_12);
}

public static class NetworkNtState extends ImplementationState {
Expand All @@ -147,7 +163,13 @@ public NetworkNtState() {

@Benchmark
public TestModel measureDraft_07_NetworkNt(final NetworkNtState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

@Benchmark
public TestModel measureDraft_2020_12_NetworkNt(
final NetworkNtState impl, final ModelState model) {
return impl.roundTrip(model, SchemaSpec.DRAFT_2020_12);
}

public static class SnowState extends ImplementationState {
Expand All @@ -158,7 +180,7 @@ public SnowState() {

@Benchmark
public TestModel measureDraft_07_Snow(final SnowState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

public static class JustifyState extends ImplementationState {
Expand All @@ -169,7 +191,7 @@ public JustifyState() {

@Benchmark
public TestModel measureDraft_07_Justify(final JustifyState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_07);
}

public static class DevHarrelState extends ImplementationState {
Expand All @@ -181,40 +203,54 @@ public DevHarrelState() {
@Benchmark
public TestModel measureDraft_2020_12_DevHarrel(
final DevHarrelState impl, final ModelState model) {
return impl.roundTrip(model);
return impl.roundTrip(model, SchemaSpec.DRAFT_2020_12);
}

@State(Scope.Thread)
private static class ImplementationState {

private final Implementation.JsonValidator validator;
private final Implementation.JsonValidator validator07;
private final Implementation.JsonValidator validator2020;

ImplementationState(final Implementation impl) {
this.validator = buildValidator(impl);
this.validator07 =
impl.supports(SchemaSpec.DRAFT_07)
? impl.prepare(
TestSchemas.DRAFT_7_SCHEMA,
SchemaSpec.DRAFT_07,
new AdditionalSchemas(Map.of(), Path.of("")))
: null;

this.validator2020 =
impl.supports(SchemaSpec.DRAFT_2020_12)
? impl.prepare(
TestSchemas.DRAFT_2020_SCHEMA,
SchemaSpec.DRAFT_2020_12,
new AdditionalSchemas(Map.of(), Path.of("")))
: null;

if (validator07 == null && validator2020 == null) {
throw new UnsupportedOperationException(
"Benchmark code needs enhancing to cover this case.");
}
}

public TestModel roundTrip(final ModelState model) {
public TestModel roundTrip(final ModelState model, final SchemaSpec version) {
final Implementation.JsonValidator validator = validator(version);
final byte[] serialized = validator.serialize(model.model, true);
return validator.deserialize(serialized);
}

private static Implementation.JsonValidator buildValidator(final Implementation impl) {
if (impl.supports(SchemaSpec.DRAFT_07)) {
return impl.prepare(
TestSchemas.DRAFT_7_SCHEMA,
SchemaSpec.DRAFT_07,
new AdditionalSchemas(Map.of(), Path.of("")));
}

if (impl.supports(SchemaSpec.DRAFT_2020_12)) {
return impl.prepare(
TestSchemas.DRAFT_2020_SCHEMA,
SchemaSpec.DRAFT_2020_12,
new AdditionalSchemas(Map.of(), Path.of("")));
private Implementation.JsonValidator validator(final SchemaSpec version) {
switch (version) {
case DRAFT_07:
return validator07;
case DRAFT_2020_12:
return validator2020;
default:
throw new UnsupportedOperationException(
"Benchmark code needs enhancing to cover this case.");
}

throw new UnsupportedOperationException(
"Benchmark code needs enhancing to cover this case.");
}
}
}