Skip to content

Commit

Permalink
Tests: Clean test cases clouds
Browse files Browse the repository at this point in the history
  • Loading branch information
@perdelt
perdelt committed Dec 20, 2024
1 parent 4ebeb42 commit b826bbf
Show file tree
Hide file tree
Showing 40 changed files with 3,288 additions and 1,381 deletions.
463 changes: 322 additions & 141 deletions docs/Example-CloudDatabase.md
View file

Large diffs are not rendered by default.

264 changes: 186 additions & 78 deletions docs/Example-CockroachDB.md
View file

Large diffs are not rendered by default.

280 changes: 227 additions & 53 deletions docs/Example-YugaByteDB.md
View file

Large diffs are not rendered by default.

136 changes: 72 additions & 64 deletions logs_tests/doc_benchbase_cockroachdb_1.log
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ Data Directory : is running
Result Directory : is running
Dashboard : is running
Message Queue : is running
Experiment : has code 1734624013
Experiment : starts at 2024-12-19 17:00:12.861793 (4308814.268927681)
Experiment : has code 1734646253
Experiment : starts at 2024-12-19 23:10:53.358305 (4331054.765439166)
Experiment : This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
Benchbase data is generated and loaded using several threads.
Benchmark is 'tpcc'. Scaling factor (e.g., number of warehouses) is 16. Benchmarking runs for 5 minutes. Target is based on multiples of '1024'. Factors for benchmarking are [16].
Expand All @@ -23,22 +23,22 @@ done
CockroachDB-1-1-1024 : will start now
- waiting 30s - : done
CockroachDB-1-1-1024 : is not loaded yet
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 16:03:13 (that is in 120 secs)
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 22:13:54 (that is in 120 secs)
- waiting 30s - : done
CockroachDB-1-1-1024 : is not loaded yet
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 16:03:13
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 22:13:54
- waiting 30s - : done
CockroachDB-1-1-1024 : is not loaded yet
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 16:03:13
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 22:13:54
- waiting 30s - : done
CockroachDB-1-1-1024 : is not loaded yet
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 16:03:13
CockroachDB-1-1-1024 : will start loading but not before 2024-12-19 22:13:54
- waiting 30s - : done
CockroachDB-1-1-1024 : is not loaded yet
done
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734624013-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-2']
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734646253-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-2']
CockroachDB-1-1-1024 Workers 3 of 3
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734624013-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-2']
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734646253-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-2']
CockroachDB-1-1-1024 : start asynch loading scripts of type loaded
CockroachDB-1-1-1024 : is loading
- waiting 30s - : done
Expand All @@ -49,19 +49,27 @@ CockroachDB-1-1-1024 : is loading
CockroachDB-1-1-1024 : is loading
- waiting 30s - : done
CockroachDB-1-1-1024 : is loading
- waiting 30s - : done
CockroachDB-1-1-1024 : is loading
- waiting 30s - : done
CockroachDB-1-1-1024 : is loading
- waiting 30s - : done
CockroachDB-1-1-1024 : is loading
- waiting 30s - : done
CockroachDB-1-1-1024 : is loading
done
CockroachDB-1-1-1024 : showing loader times
CockroachDB-1-1-1024 : loader timespan (first to last [s]) = 142
CockroachDB-1-1-1024 : loader timespan (first to last [s]) = 267
CockroachDB-1-1-1024 : benchmarks done 0 of 1. This will be client 1
CockroachDB-1-1-1024 : we will change parameters of benchmark as {'PARALLEL': '1', 'SF': '16', 'BENCHBASE_BENCH': 'tpcc', 'BENCHBASE_PROFILE': 'cockroachdb', 'BEXHOMA_DATABASE': 'defaultdb', 'BENCHBASE_TARGET': 16384, 'BENCHBASE_TERMINALS': 16, 'BENCHBASE_TIME': 300, 'BENCHBASE_ISOLATION': 'TRANSACTION_READ_COMMITTED', 'BEXHOMA_USER': 'root', 'BEXHOMA_PASSWORD': ''}
CockroachDB-1-1-1024-1 : start benchmarking
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734624013-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-2']
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734624013-0
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734624013-1
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734624013-2
CockroachDB-1-1-1024 : benchmarking results in folder /home/perdelt/benchmarks/1734624013
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734646253-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-2']
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734646253-0
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734646253-1
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734646253-2
CockroachDB-1-1-1024 : benchmarking results in folder /home/perdelt/benchmarks/1734646253
- waiting 10s - : done
CockroachDB-1-1-1024 : benchmarking is waiting for job bexhoma-benchmarker-cockroachdb-1-1-1024-1734624013-1-1-68j7x: found
CockroachDB-1-1-1024 : benchmarking is waiting for job bexhoma-benchmarker-cockroachdb-1-1-1024-1734646253-1-1-vbl52: found
- waiting 30s - : done
CockroachDB-1-1-1024 : has running benchmarks
- waiting 30s - : done
Expand All @@ -83,20 +91,20 @@ CockroachDB-1-1-1024 : has running benchmarks
done
CockroachDB-1-1-1024 : has running benchmarks
CockroachDB-1-1-1024-1 : showing benchmarker times
CockroachDB-1-1-1024-1 : benchmarker timespan (start to end single container [s]) = 313
CockroachDB-1-1-1024-1 : benchmarker times (start/end per pod and container) = [(1734624363, 1734624667)]
CockroachDB-1-1-1024-1 : found and updated times {'benchmarker': [(1734624363, 1734624667)]}
CockroachDB-1-1-1024-1 : benchmarker timespan (start to end single container [s]) = 314
CockroachDB-1-1-1024-1 : benchmarker times (start/end per pod and container) = [(1734646723, 1734647027)]
CockroachDB-1-1-1024-1 : found and updated times {'benchmarker': [(1734646723, 1734647027)]}
done
CockroachDB-1-1-1024 : benchmarks done 0 of 1. This will be client 2
CockroachDB-1-1-1024 : we will change parameters of benchmark as {'PARALLEL': '2', 'SF': '16', 'BENCHBASE_BENCH': 'tpcc', 'BENCHBASE_PROFILE': 'cockroachdb', 'BEXHOMA_DATABASE': 'defaultdb', 'BENCHBASE_TARGET': 8192, 'BENCHBASE_TERMINALS': 8, 'BENCHBASE_TIME': 300, 'BENCHBASE_ISOLATION': 'TRANSACTION_READ_COMMITTED', 'BEXHOMA_USER': 'root', 'BEXHOMA_PASSWORD': ''}
CockroachDB-1-1-1024-2 : start benchmarking
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734624013-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734624013-2']
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734624013-0
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734624013-1
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734624013-2
CockroachDB-1-1-1024 : benchmarking results in folder /home/perdelt/benchmarks/1734624013
Worker pods found: ['bexhoma-worker-cockroachdb-1-1-1024-1734646253-0', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-1', 'bexhoma-worker-cockroachdb-1-1-1024-1734646253-2']
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734646253-0
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734646253-1
CockroachDB-1-1-1024 : distributed system - get host info for worker bexhoma-worker-cockroachdb-1-1-1024-1734646253-2
CockroachDB-1-1-1024 : benchmarking results in folder /home/perdelt/benchmarks/1734646253
- waiting 10s - : done
CockroachDB-1-1-1024 : benchmarking is waiting for job bexhoma-benchmarker-cockroachdb-1-1-1024-1734624013-1-2-8wmjv: found
CockroachDB-1-1-1024 : benchmarking is waiting for job bexhoma-benchmarker-cockroachdb-1-1-1024-1734646253-1-2-2vzvp: found
- waiting 30s - : done
CockroachDB-1-1-1024 : has running benchmarks
- waiting 30s - : done
Expand All @@ -118,24 +126,24 @@ CockroachDB-1-1-1024 : has running benchmarks
done
CockroachDB-1-1-1024 : has running benchmarks
CockroachDB-1-1-1024-2 : showing benchmarker times
CockroachDB-1-1-1024-2 : benchmarker timespan (start to end single container [s]) = 315
CockroachDB-1-1-1024-2 : benchmarker times (start/end per pod and container) = [(1734624714, 1734625018), (1734624714, 1734625019)]
CockroachDB-1-1-1024-2 : found and updated times {'benchmarker': [(1734624714, 1734625018), (1734624714, 1734625019)]}
CockroachDB-1-1-1024-2 : benchmarker timespan (start to end single container [s]) = 314
CockroachDB-1-1-1024-2 : benchmarker times (start/end per pod and container) = [(1734647075, 1734647379), (1734647075, 1734647379)]
CockroachDB-1-1-1024-2 : found and updated times {'benchmarker': [(1734647075, 1734647379), (1734647075, 1734647379)]}
done
CockroachDB-1-1-1024 : can be stopped
Experiment ends at 2024-12-19 17:17:39.472259 (4309860.879392857): 1046.6104651754722s total
Experiment ends at 2024-12-19 23:30:19.076842 (4332220.48397571): 1165.7185365445912s total
Experiment : downloading partial results
Experiment : uploading full results
Experiment : uploading workload file
Result workflow not complete
Result workflow complete

## Show Summary

### Workload
Benchbase Workload SF=16 (warehouses for TPC-C)
Type: benchbase
Duration: 1047s
Code: 1734624013
Duration: 1166s
Code: 1734646253
Benchbase runs the benchmark.
This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
Benchbase data is generated and loaded using several threads.
Expand All @@ -156,91 +164,91 @@ CockroachDB-1-1-1024-1 uses docker image cockroachdb/cockroach:v24.2.4
Cores:64
host:5.15.0-126-generic
node:cl-worker11
disk:249214372
disk:249215596
requests_cpu:4
requests_memory:16Gi
worker 0
RAM:1081966526464
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker28
disk:439691956
datadisk:116318604
disk:461657896
datadisk:116314488
volume_size:1000G
volume_used:109G
worker 1
RAM:1081751019520
Cores:128
host:5.15.0-126-generic
node:cl-worker29
disk:153273172
datadisk:116107360
volume_size:1000G
volume_used:109G
worker 2
RAM:1081965555712
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker27
disk:584549888
datadisk:116107116
disk:635102812
datadisk:116104180
volume_size:1000G
volume_used:109G
worker 2
RAM:540587499520
Cores:128
host:5.15.0-126-generic
node:cl-worker22
disk:123840188
datadisk:116091372
volume_size:1000G
volume_used:109G
CockroachDB-1-1-1024-2 uses docker image cockroachdb/cockroach:v24.2.4
RAM:541008576512
Cores:64
host:5.15.0-126-generic
node:cl-worker11
disk:249214372
disk:249215600
requests_cpu:4
requests_memory:16Gi
worker 0
RAM:1081966526464
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker28
disk:439986660
datadisk:116611832
disk:461867536
datadisk:116522308
volume_size:1000G
volume_used:109G
worker 1
RAM:1081751019520
Cores:128
host:5.15.0-126-generic
node:cl-worker29
disk:153570092
datadisk:116404276
volume_size:1000G
volume_used:109G
worker 2
RAM:1081965555712
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker27
disk:584845212
datadisk:116401204
disk:635488320
datadisk:116308436
volume_size:1000G
volume_used:109G
worker 2
RAM:540587499520
Cores:128
host:5.15.0-126-generic
node:cl-worker22
disk:124062476
datadisk:116312956
volume_size:1000G
volume_used:109G

### Execution
experiment_run terminals target pod_count time Throughput (requests/second) Latency Distribution.95th Percentile Latency (microseconds) Latency Distribution.Average Latency (microseconds)
CockroachDB-1-1-1024-1 1 16 16384 1 300.0 481.72 84008.0 33199.0
CockroachDB-1-1-1024-2 1 16 16384 2 300.0 448.16 95345.0 35721.0
CockroachDB-1-1-1024-1 1 16 16384 1 300.0 312.89 95381.0 51118.0
CockroachDB-1-1-1024-2 1 16 16384 2 300.0 241.61 142861.0 66206.0

Warehouses: 16

### Workflow

#### Actual
DBMS CockroachDB-1-1-1024 - Pods [[2, 1]]
DBMS CockroachDB-1-1-1024 - Pods [[1, 2]]

#### Planned
DBMS CockroachDB-1-1-1024 - Pods [[1, 2]]

### Loading
time_load terminals pods Imported warehouses [1/h]
CockroachDB-1-1-1024-1 142.0 1.0 1.0 405.633803
CockroachDB-1-1-1024-2 142.0 1.0 2.0 405.633803
CockroachDB-1-1-1024-1 267.0 1.0 1.0 215.730337
CockroachDB-1-1-1024-2 267.0 1.0 2.0 215.730337

### Tests
TEST passed: Throughput (requests/second) contains no 0 or NaN
Expand Down
70 changes: 35 additions & 35 deletions logs_tests/doc_benchbase_cockroachdb_1_summary.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
### Workload
Benchbase Workload SF=16 (warehouses for TPC-C)
Type: benchbase
Duration: 1047s
Code: 1734624013
Duration: 1166s
Code: 1734646253
Benchbase runs the benchmark.
This experiment compares run time and resource consumption of Benchbase queries in different DBMS.
Benchbase data is generated and loaded using several threads.
Expand All @@ -25,91 +25,91 @@ CockroachDB-1-1-1024-1 uses docker image cockroachdb/cockroach:v24.2.4
Cores:64
host:5.15.0-126-generic
node:cl-worker11
disk:249214372
disk:249215596
requests_cpu:4
requests_memory:16Gi
worker 0
RAM:1081966526464
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker28
disk:439691956
datadisk:116318604
disk:461657896
datadisk:116314488
volume_size:1000G
volume_used:109G
worker 1
RAM:1081751019520
Cores:128
host:5.15.0-126-generic
node:cl-worker29
disk:153273172
datadisk:116107360
volume_size:1000G
volume_used:109G
worker 2
RAM:1081965555712
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker27
disk:584549888
datadisk:116107116
disk:635102812
datadisk:116104180
volume_size:1000G
volume_used:109G
worker 2
RAM:540587499520
Cores:128
host:5.15.0-126-generic
node:cl-worker22
disk:123840188
datadisk:116091372
volume_size:1000G
volume_used:109G
CockroachDB-1-1-1024-2 uses docker image cockroachdb/cockroach:v24.2.4
RAM:541008576512
Cores:64
host:5.15.0-126-generic
node:cl-worker11
disk:249214372
disk:249215600
requests_cpu:4
requests_memory:16Gi
worker 0
RAM:1081966526464
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker28
disk:439986660
datadisk:116611832
disk:461867536
datadisk:116522308
volume_size:1000G
volume_used:109G
worker 1
RAM:1081751019520
Cores:128
host:5.15.0-126-generic
node:cl-worker29
disk:153570092
datadisk:116404276
volume_size:1000G
volume_used:109G
worker 2
RAM:1081965555712
Cores:256
host:5.15.0-1067-nvidia
node:cl-worker27
disk:584845212
datadisk:116401204
disk:635488320
datadisk:116308436
volume_size:1000G
volume_used:109G
worker 2
RAM:540587499520
Cores:128
host:5.15.0-126-generic
node:cl-worker22
disk:124062476
datadisk:116312956
volume_size:1000G
volume_used:109G

### Execution
experiment_run terminals target pod_count time Throughput (requests/second) Latency Distribution.95th Percentile Latency (microseconds) Latency Distribution.Average Latency (microseconds)
CockroachDB-1-1-1024-1 1 16 16384 1 300.0 481.72 84008.0 33199.0
CockroachDB-1-1-1024-2 1 16 16384 2 300.0 448.16 95345.0 35721.0
CockroachDB-1-1-1024-1 1 16 16384 1 300.0 312.89 95381.0 51118.0
CockroachDB-1-1-1024-2 1 16 16384 2 300.0 241.61 142861.0 66206.0

Warehouses: 16

### Workflow

#### Actual
DBMS CockroachDB-1-1-1024 - Pods [[2, 1]]
DBMS CockroachDB-1-1-1024 - Pods [[1, 2]]

#### Planned
DBMS CockroachDB-1-1-1024 - Pods [[1, 2]]

### Loading
time_load terminals pods Imported warehouses [1/h]
CockroachDB-1-1-1024-1 142.0 1.0 1.0 405.633803
CockroachDB-1-1-1024-2 142.0 1.0 2.0 405.633803
CockroachDB-1-1-1024-1 267.0 1.0 1.0 215.730337
CockroachDB-1-1-1024-2 267.0 1.0 2.0 215.730337

### Tests
TEST passed: Throughput (requests/second) contains no 0 or NaN
Expand Down
Loading

0 comments on commit b826bbf

Please sign in to comment.