[WIP] Automatically generate supported integrations #6322

bouwkast · 2024-11-19T22:11:52Z

Summary of changes

Adds functionality for us to automatically generate a markdown table of what integrations we support and their minimum and maximum versions.

Reason for change

We'd like to automate this process away.
We'd like to provide more transparency in the versions that we actually test with respect to the minimum version and the maximum version.

Implementation details

Test coverage

Other details

andrewlock · 2024-11-19T22:45:34Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6322) - mean (72ms)  : 64, 80
     .   : milestone, 72,
    master - mean (72ms)  : 63, 82
     .   : milestone, 72,

    section CallTarget+Inlining+NGEN
    This PR (6322) - mean (1,118ms)  : 1095, 1140
     .   : milestone, 1118,
    master - mean (1,108ms)  : 1092, 1123
     .   : milestone, 1108,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6322) - mean (119ms)  : 111, 126
     .   : milestone, 119,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6322) - mean (783ms)  : 710, 856
     .   : milestone, 783,
    master - mean (767ms)  : 751, 783
     .   : milestone, 767,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6322) - mean (92ms)  : 90, 94
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (6322) - mean (721ms)  : 702, 740
     .   : milestone, 721,
    master - mean (729ms)  : 714, 743
     .   : milestone, 729,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6322) - mean (191ms)  : 187, 196
     .   : milestone, 191,
    master - mean (191ms)  : 186, 196
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6322) - mean (1,215ms)  : 1190, 1239
     .   : milestone, 1215,
    master - mean (1,209ms)  : 1187, 1231
     .   : milestone, 1209,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6322) - mean (276ms)  : 272, 281
     .   : milestone, 276,
    master - mean (276ms)  : 272, 279
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6322) - mean (944ms)  : 925, 962
     .   : milestone, 944,
    master - mean (945ms)  : 923, 966
     .   : milestone, 945,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6322) - mean (265ms)  : 261, 270
     .   : milestone, 265,
    master - mean (267ms)  : 264, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6322) - mean (926ms)  : 910, 942
     .   : milestone, 926,
    master - mean (931ms)  : 912, 951
     .   : milestone, 931,

datadog-ddstaging · 2024-11-19T22:51:24Z

Datadog Report

Branch report: steven/auto-doc
Commit report: d1241ef
Test service: dd-trace-dotnet

❌ 48 Failed (0 Known Flaky), 424757 Passed, 1512 Skipped, 12h 58m 38.46s Total Time

❌ Failed Tests (48)

This report shows up to 5 failed tests.

SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Details
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```

andrewlock · 2024-11-19T23:14:13Z

Benchmarks Report for tracer 🐌

Benchmarks for #6322 compared to master:

1 benchmarks are slower, with geometric mean 1.155
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.1μs	44.9ns	277ns	0.0229	0.0115	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	54.8ns	338ns	0.0194	0.0097	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	56.8ns	220ns	1.05	0.318	0.0979	6.21 KB
#6322	`StartStopWithChild`	net6.0	8.16μs	45.1ns	274ns	0.0173	0.00863	0	5.61 KB
#6322	`StartStopWithChild`	netcoreapp3.1	10.3μs	54.4ns	278ns	0.0159	0.00529	0	5.81 KB
#6322	`StartStopWithChild`	net472	16.4μs	69.1ns	268ns	1.03	0.285	0.0814	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	486μs	162ns	561ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	653μs	374ns	1.45μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	867μs	465ns	1.74μs	0.428	3.3 KB
#6322	`WriteAndFlushEnrichedTraces`	net6.0	486μs	308ns	1.19μs	0	2.7 KB
#6322	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	666μs	361ns	1.35μs	0	2.7 KB
#6322	`WriteAndFlushEnrichedTraces`	net472	835μs	552ns	2.07μs	0.417	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	198μs	1.11μs	9.32μs	0.21	18.73 KB
master	`SendRequest`	netcoreapp3.1	223μs	1.21μs	6.93μs	0.238	20.89 KB
master	`SendRequest`	net472	0.00114ns	0.000479ns	0.00185ns	0	0 b
#6322	`SendRequest`	net6.0	202μs	1.13μs	7.94μs	0.193	18.73 KB
#6322	`SendRequest`	netcoreapp3.1	226μs	1.32μs	11.7μs	0.241	20.89 KB
#6322	`SendRequest`	net472	0.000111ns	0.000111ns	0.000416ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6322

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.82 KB	41.48 KB	-341 B	-0.82%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	593μs	3.14μs	17.2μs	0.558	0	0	41.82 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	684μs	3.84μs	27.2μs	0.361	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	2.88μs	10.8μs	8.56	2.57	0.428	53.33 KB
#6322	`WriteAndFlushEnrichedTraces`	net6.0	545μs	2.9μs	14.8μs	0.541	0	0	41.48 KB
#6322	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	3.03μs	12.9μs	0.336	0	0	41.81 KB
#6322	`WriteAndFlushEnrichedTraces`	net472	867μs	4.06μs	15.7μs	8.62	2.59	0.431	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.4μs	1.55ns	5.8ns	0.0138	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	1.52ns	5.9ns	0.0131	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.02μs	1.81ns	7.01ns	0.157	0.000997	987 B
#6322	`ExecuteNonQuery`	net6.0	1.32μs	1.24ns	4.81ns	0.0146	0	1.02 KB
#6322	`ExecuteNonQuery`	netcoreapp3.1	1.74μs	1.53ns	5.74ns	0.0133	0	1.02 KB
#6322	`ExecuteNonQuery`	net472	2.16μs	1.54ns	5.95ns	0.156	0.00109	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.28μs	0.661ns	2.47ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.5μs	1.02ns	3.81ns	0.0128	976 B
master	`CallElasticsearch`	net472	2.59μs	1.8ns	6.97ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	0.772ns	2.99ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.71μs	0.736ns	2.75ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.72μs	1.39ns	5.38ns	0.167	1.05 KB
#6322	`CallElasticsearch`	net6.0	1.2μs	0.998ns	3.87ns	0.0138	976 B
#6322	`CallElasticsearch`	netcoreapp3.1	1.52μs	0.35ns	1.21ns	0.013	976 B
#6322	`CallElasticsearch`	net472	2.62μs	3.23ns	12.5ns	0.157	995 B
#6322	`CallElasticsearchAsync`	net6.0	1.32μs	0.663ns	2.48ns	0.0133	952 B
#6322	`CallElasticsearchAsync`	netcoreapp3.1	1.73μs	1.05ns	3.92ns	0.0138	1.02 KB
#6322	`CallElasticsearchAsync`	net472	2.71μs	2.52ns	9.42ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.22μs	0.459ns	1.72ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.7μs	0.999ns	3.74ns	0.0128	952 B
master	`ExecuteAsync`	net472	1.81μs	0.475ns	1.78ns	0.145	915 B
#6322	`ExecuteAsync`	net6.0	1.27μs	0.368ns	1.33ns	0.0135	952 B
#6322	`ExecuteAsync`	netcoreapp3.1	1.66μs	2.38ns	9.2ns	0.0124	952 B
#6322	`ExecuteAsync`	net472	1.85μs	0.457ns	1.77ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.48μs	1.24ns	4.46ns	0.0316	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.21μs	1.96ns	7.32ns	0.0391	2.85 KB
master	`SendAsync`	net472	7.24μs	1.21ns	4.67ns	0.493	3.12 KB
#6322	`SendAsync`	net6.0	4.31μs	2ns	7.49ns	0.0327	2.31 KB
#6322	`SendAsync`	netcoreapp3.1	5.38μs	1.4ns	5.24ns	0.0375	2.85 KB
#6322	`SendAsync`	net472	7.36μs	1.88ns	7.28ns	0.493	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.59μs	0.76ns	2.74ns	0.0232	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.17μs	0.937ns	3.38ns	0.0217	1.64 KB
master	`EnrichedLog`	net472	2.64μs	1.13ns	4.22ns	0.25	1.57 KB
#6322	`EnrichedLog`	net6.0	1.48μs	0.93ns	3.48ns	0.0229	1.64 KB
#6322	`EnrichedLog`	netcoreapp3.1	2.12μs	3.66ns	13.7ns	0.0226	1.64 KB
#6322	`EnrichedLog`	net472	2.6μs	2.39ns	9.25ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	182ns	704ns	0.0596	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	124μs	76.9ns	277ns	0	0	4.28 KB
master	`EnrichedLog`	net472	153μs	133ns	516ns	0.695	0.232	4.46 KB
#6322	`EnrichedLog`	net6.0	120μs	165ns	640ns	0.0602	0	4.28 KB
#6322	`EnrichedLog`	netcoreapp3.1	124μs	133ns	497ns	0	0	4.28 KB
#6322	`EnrichedLog`	net472	149μs	84.1ns	315ns	0.669	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.11μs	0.826ns	3.2ns	0.0308	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.28μs	2.29ns	8.86ns	0.0299	2.2 KB
master	`EnrichedLog`	net472	4.84μs	1.66ns	6.42ns	0.319	2.02 KB
#6322	`EnrichedLog`	net6.0	3.08μs	0.976ns	3.78ns	0.0308	2.2 KB
#6322	`EnrichedLog`	netcoreapp3.1	4.25μs	1.42ns	5.33ns	0.0298	2.2 KB
#6322	`EnrichedLog`	net472	4.74μs	1.34ns	5.18ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.711ns	2.75ns	0.0163	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.71μs	1.35ns	5.24ns	0.0155	1.14 KB
master	`SendReceive`	net472	2.13μs	1.31ns	4.89ns	0.183	1.16 KB
#6322	`SendReceive`	net6.0	1.38μs	0.69ns	2.58ns	0.0159	1.14 KB
#6322	`SendReceive`	netcoreapp3.1	1.79μs	0.845ns	3.27ns	0.015	1.14 KB
#6322	`SendReceive`	net472	2.17μs	1.93ns	7.47ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.71μs	1.3ns	5.02ns	0.0217	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.85μs	1.49ns	5.76ns	0.0213	1.65 KB
master	`EnrichedLog`	net472	4.48μs	3.6ns	13.9ns	0.322	2.04 KB
#6322	`EnrichedLog`	net6.0	2.63μs	1.13ns	4.39ns	0.0224	1.6 KB
#6322	`EnrichedLog`	netcoreapp3.1	3.9μs	2.69ns	10.4ns	0.0215	1.65 KB
#6322	`EnrichedLog`	net472	4.42μs	11.2ns	43.3ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6322

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.155	415.74	480.38

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	416ns	0.163ns	0.565ns	0.00797	576 B
master	`StartFinishSpan`	netcoreapp3.1	605ns	0.373ns	1.44ns	0.00768	576 B
master	`StartFinishSpan`	net472	672ns	0.328ns	1.27ns	0.0915	578 B
master	`StartFinishScope`	net6.0	494ns	0.202ns	0.783ns	0.00971	696 B
master	`StartFinishScope`	netcoreapp3.1	686ns	0.366ns	1.42ns	0.00927	696 B
master	`StartFinishScope`	net472	864ns	1.06ns	4.09ns	0.104	658 B
#6322	`StartFinishSpan`	net6.0	480ns	0.277ns	1.07ns	0.00817	576 B
#6322	`StartFinishSpan`	netcoreapp3.1	617ns	0.39ns	1.51ns	0.00772	576 B
#6322	`StartFinishSpan`	net472	664ns	0.669ns	2.59ns	0.0916	578 B
#6322	`StartFinishScope`	net6.0	532ns	0.235ns	0.91ns	0.00986	696 B
#6322	`StartFinishScope`	netcoreapp3.1	740ns	0.85ns	3.29ns	0.00925	696 B
#6322	`StartFinishScope`	net472	857ns	0.452ns	1.75ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	655ns	0.353ns	1.37ns	0.00994	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	923ns	0.503ns	1.95ns	0.00925	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.29ns	1.13ns	0.104	658 B
#6322	`RunOnMethodBegin`	net6.0	590ns	0.232ns	0.897ns	0.00976	696 B
#6322	`RunOnMethodBegin`	netcoreapp3.1	893ns	2.71ns	10.1ns	0.00938	696 B
#6322	`RunOnMethodBegin`	net472	1.15μs	0.366ns	1.42ns	0.104	658 B

Tried making an analyzer to read through the instrumented attributes but it didn't work.

Add basic automated markdown table generation

331956a

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Nov 19, 2024

bouwkast added 2 commits November 19, 2024 20:14

Add various samples of WIP tables

2a1d6fc

Remove failed table generations

d1241ef

Tried making an analyzer to read through the instrumented attributes but it didn't work.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Automatically generate supported integrations #6322

[WIP] Automatically generate supported integrations #6322

bouwkast commented Nov 19, 2024

andrewlock commented Nov 19, 2024 •

edited

Loading

datadog-ddstaging bot commented Nov 19, 2024 •

edited

Loading

andrewlock commented Nov 19, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6322

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6322

Raw results

Raw results

[WIP] Automatically generate supported integrations #6322

Are you sure you want to change the base?

[WIP] Automatically generate supported integrations #6322

Conversation

bouwkast commented Nov 19, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Nov 19, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Nov 19, 2024 • edited Loading

Datadog Report

❌ Failed Tests (48)

andrewlock commented Nov 19, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6322

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6322

Raw results

Raw results

andrewlock commented Nov 19, 2024 •

edited

Loading

datadog-ddstaging bot commented Nov 19, 2024 •

edited

Loading

andrewlock commented Nov 19, 2024 •

edited

Loading