Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ekump/add test agent to trace utils #498

Closed
wants to merge 7 commits into from

Conversation

ekump
Copy link
Contributor

@ekump ekump commented Jun 20, 2024

WIP: Not ready for review yet.

What does this PR do?

A brief description of the change being made with this pull request.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

@ekump ekump force-pushed the ekump/add-test-agent-to-trace-utils branch 6 times, most recently from a18e1dc to 4fb6d0a Compare June 25, 2024 19:54
@ekump ekump force-pushed the ekump/add-test-agent-to-trace-utils branch 4 times, most recently from 74fba59 to 122fa9b Compare June 28, 2024 19:55
@ekump ekump force-pushed the ekump/add-test-agent-to-trace-utils branch from 122fa9b to 1baeae7 Compare June 28, 2024 19:56
@pr-commenter
Copy link

pr-commenter bot commented Jun 28, 2024

Benchmarks

Comparison

Parameters

Baseline Candidate
config baseline candidate
git_branch main ekump/add-test-agent-to-trace-utils
git_commit_date 1719860752 1719930378
git_commit_sha 8a5d447 45cd1e6
iterations 716.0 677.0
See matching parameters
Baseline Candidate
ci_job_date 1719930629 1719930629
ci_job_id 559796829 559796829
ci_pipeline_id 38138949 38138949
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 0 metrics, 0 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:sql/obfuscate_sql_string worse
[+4.056µs; +4.071µs] or [+5.808%; +5.829%]
73.899µs 69.836µs

Candidate

Candidate benchmark details

Group 1

iterations config cpu_model ci_job_date ci_job_id ci_pipeline_id git_commit_sha git_commit_date git_branch
677.0 candidate Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1719930629 559796829 38138949 45cd1e6 1719930378 ekump/add-test-agent-to-trace-utils
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 73.840µs 73.899µs ± 0.024µs 73.898µs ± 0.015µs 73.913µs 73.944µs 73.957µs 73.960µs 0.08% 0.259 -0.025 0.03% 0.002µs 1 100
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [73.895µs; 73.904µs] or [-0.006%; +0.006%] None None None

Warnings

Scenario: sql/obfuscate_sql_string, Metric: execution_time

Sample size is 100, which is lower than 105.

The minimal sample size in case of normal distribution to achieve significance
level of 0.05 for difference of means with effect size Cohen's d = 0.5 must be at
least 105.

The conclusions from confidence intervals may be invalid.
---------------------------------------------------------------------------

Baseline

Baseline benchmark details

Group 1

iterations config cpu_model ci_job_date ci_job_id ci_pipeline_id git_commit_sha git_commit_date git_branch
716.0 baseline Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1719930629 559796829 38138949 8a5d447 1719860752 main
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 69.765µs 69.836µs ± 0.028µs 69.837µs ± 0.019µs 69.855µs 69.882µs 69.891µs 69.914µs 0.11% 0.057 -0.199 0.04% 0.003µs 1 100
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [69.830µs; 69.841µs] or [-0.008%; +0.008%] None None None

Warnings

Scenario: sql/obfuscate_sql_string, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------
Scenario: sql/obfuscate_sql_string, Metric: execution_time

Sample size is 100, which is lower than 105.

The minimal sample size in case of normal distribution to achieve significance
level of 0.05 for difference of means with effect size Cohen's d = 0.5 must be at
least 105.

The conclusions from confidence intervals may be invalid.
---------------------------------------------------------------------------

@ekump ekump force-pushed the ekump/add-test-agent-to-trace-utils branch from 35f07ad to e7565db Compare July 1, 2024 20:33
@codecov-commenter
Copy link

codecov-commenter commented Jul 1, 2024

Codecov Report

Attention: Patch coverage is 96.66667% with 3 lines in your changes missing coverage. Please review.

Project coverage is 70.46%. Comparing base (8a5d447) to head (45cd1e6).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #498      +/-   ##
==========================================
+ Coverage   70.38%   70.46%   +0.08%     
==========================================
  Files         204      205       +1     
  Lines       27835    27925      +90     
==========================================
+ Hits        19591    19678      +87     
- Misses       8244     8247       +3     
Components Coverage Δ
crashtracker 16.70% <ø> (ø)
datadog-alloc 98.76% <ø> (ø)
data-pipeline 51.15% <ø> (ø)
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 86.12% <ø> (ø)
ddcommon-ffi 74.15% <ø> (ø)
ddtelemetry 59.37% <ø> (ø)
ipc 84.66% <ø> (ø)
profiling 78.63% <ø> (ø)
profiling-ffi 58.19% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 36.19% <ø> (ø)
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 54.98% <ø> (ø)
trace-mini-agent 69.70% <ø> (ø)
trace-normalization 97.79% <ø> (ø)
trace-obfuscation 95.75% <ø> (ø)
trace-protobuf 77.16% <ø> (ø)
trace-utils 91.41% <96.66%> (+0.20%) ⬆️

@ekump ekump force-pushed the ekump/add-test-agent-to-trace-utils branch from e7565db to e2ac68e Compare July 1, 2024 20:51
@ekump ekump force-pushed the ekump/add-test-agent-to-trace-utils branch from d01616e to 5f39622 Compare July 2, 2024 13:30
@ekump ekump closed this Aug 5, 2024
@ekump ekump deleted the ekump/add-test-agent-to-trace-utils branch October 23, 2024 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants