diff --git a/README.md b/README.md index ded544b..8627bbc 100644 --- a/README.md +++ b/README.md @@ -90,16 +90,14 @@ Please see more details about this feature via [link](https://github.com/th2-net # Performance -The component provides a performance of 100K events per second if the events are packaged in batches of 20 or -more events(event size: 1.4KB, event status: SUCCEED, no attached messages). +The component provides a performance of 100K events per second if the events are packaged in batches of 50 - 300 events +where event body size is 1.1 KiB, status is SUCCEED and one message id attached. Processing speed (K events/sec) vs batch size for estore (under load of 100K events/s): -![performance chart](./perf_chart.png) - Note: for smaller batches (less than 100 events) higher mqRouter.prefetchCount value should be used (e.g. 1000) to achieve these results. -More details [here](perftest/perftest.md). +More details [here](doc/perftest-5.7.0-dev/perftest). # Changes diff --git a/doc/perftest-5.7.0-dev/cluster_schema.png b/doc/perftest-5.7.0-dev/cluster_schema.png new file mode 100644 index 0000000..cfff271 Binary files /dev/null and b/doc/perftest-5.7.0-dev/cluster_schema.png differ diff --git a/doc/perftest-5.7.0-dev/computation_resources_estore.png b/doc/perftest-5.7.0-dev/computation_resources_estore.png new file mode 100644 index 0000000..aaac11a Binary files /dev/null and b/doc/perftest-5.7.0-dev/computation_resources_estore.png differ diff --git a/doc/perftest-5.7.0-dev/computation_resources_rabbitmq.png b/doc/perftest-5.7.0-dev/computation_resources_rabbitmq.png new file mode 100644 index 0000000..50b2e79 Binary files /dev/null and b/doc/perftest-5.7.0-dev/computation_resources_rabbitmq.png differ diff --git a/doc/perftest-5.7.0-dev/estore_event_batch_content_size.png b/doc/perftest-5.7.0-dev/estore_event_batch_content_size.png new file mode 100644 index 0000000..0a0a997 Binary files /dev/null and b/doc/perftest-5.7.0-dev/estore_event_batch_content_size.png differ diff --git a/doc/perftest-5.7.0-dev/jvm_estore_gc.png b/doc/perftest-5.7.0-dev/jvm_estore_gc.png new file mode 100644 index 0000000..df707b7 Binary files /dev/null and b/doc/perftest-5.7.0-dev/jvm_estore_gc.png differ diff --git a/doc/perftest-5.7.0-dev/jvm_estore_memory.png b/doc/perftest-5.7.0-dev/jvm_estore_memory.png new file mode 100644 index 0000000..4d6224e Binary files /dev/null and b/doc/perftest-5.7.0-dev/jvm_estore_memory.png differ diff --git a/doc/perftest-5.7.0-dev/monitoring_estore_event_rate.png b/doc/perftest-5.7.0-dev/monitoring_estore_event_rate.png new file mode 100644 index 0000000..3b26bf7 Binary files /dev/null and b/doc/perftest-5.7.0-dev/monitoring_estore_event_rate.png differ diff --git a/doc/perftest-5.7.0-dev/monitoring_estore_event_size.png b/doc/perftest-5.7.0-dev/monitoring_estore_event_size.png new file mode 100644 index 0000000..065e309 Binary files /dev/null and b/doc/perftest-5.7.0-dev/monitoring_estore_event_size.png differ diff --git a/doc/perftest-5.7.0-dev/perftest.md b/doc/perftest-5.7.0-dev/perftest.md new file mode 100644 index 0000000..72f57e4 --- /dev/null +++ b/doc/perftest-5.7.0-dev/perftest.md @@ -0,0 +1,229 @@ +# Store events via th2-estore:5.7.0-dev (perftest cluster) + +## Hardware configuration + +### Cluster + ++ **CPU**: Intel Xeon Gold 5218 x 2 ++ **RAM**: 768 GB (32 GB RAM x 24) ++ **Disk**: SAMSUNG MZ7LM3T8HMLP SSD 3,5Tb x 5 (raid5 + Hotspare) + +### Virtual machines are deployed on the cluster: + ++ **cassandra** + + **CPU**: 4 core + + **RAM**: 8Gb ++ **cluster node** - x2 + + **CPU**: 11 core + + **RAM**: 30Gb + +## Software configuration + +### RabbitMQ ++ **VM**: cluster node ++ **Docker image**: docker.io/bitnami/rabbitmq:3.11.2-debian-11-r0 + +### Cassandra ++ **VM**: cassandra ++ **Version**: 4.0.5 ++ **Special java args**: + + -Xlog:gc=info,heap*=debug,age*=debug,safepoint=info,promotion*=debug... + + -Xms4G -Xmx4G + +## Cluster schema +![Cluster schema](cluster_schema.png) + +## th2 components configuration + +### woodpecker +```yaml +apiVersion: th2.exactpro.com/v2 +kind: Th2Box +metadata: + name: woodpecker1 +spec: + disabled: false + imageName: ghcr.io/th2-net/th2-woodpecker-template + imageVersion: 2.0.0-TH2-5204-estore-perf-9547964275-56d1114 + type: th2-conn + customConfig: + tickRate: 1 + maxBatchSize: 175 + useTransportMode: EVENT_MODE + generatorSettings: + messageGeneratorSettings: + bookName: test_book + sessionGroupPrefix: raw-group + sessionGroupNumber: 1 + sessionAliasPrefix: raw-session-alias + sessionAliasNumber: 1 + protocol: fix + oneOf: + directionToExamples: + SECOND: + messages: + - 'raw-message' + eventGeneratorSettings: + bookName: estore_perf_test_01 + scope: qa_session_01 + generationMode: SINGLE_ROOT + treeDepth: 0 + childCount: 0 + failureRate: 0 + descriptionLength: 0 + bodyDataLength: 200 + pins: + mq: + subscribers: + - name: in_events + attributes: [event, in] + - name: in_messages + attributes: [subscribe, in, protobuf-group] + publishers: + - name: out_messages + attributes: [publish, out, raw, store, protobuf-group] + grpc: + server: + - name: server + serviceClasses: + - com.exactpro.th2.woodpecker.grpc.WoodpeckerService + extendedSettings: + nodeSelector: + "kubernetes.io/hostname": node-2 + envVariables: + JAVA_TOOL_OPTIONS: > + -Djava.net.preferIPv4Stack=true + resources: + limits: + cpu: 2000m + memory: 1000Mi + requests: + cpu: 500m + memory: 200Mi +``` + +### estore +```yaml +apiVersion: th2.exactpro.com/v2 +kind: Th2Estore +metadata: + name: estore +spec: + imageName: ghcr.io/th2-net/th2-estore + imageVersion: 5.7.0-dev + cradleManager: + prepareStorage: false + timeout: 5000 + resultPageSize: 1000 + composingServiceThreads: 1 + counterPersistenceInterval: 15000 + maxUncompressedTestEventSize: 5120 + compressionType: LZ4 + bookRefreshIntervalMillis: 60000 + customConfig: + maxTaskCount : 128 + maxTaskDataSize : 536870912 + maxRetryCount : 3 + processingThreads: 1 + mqRouter: + prefetchCount: 160 # 125 % of maxTaskCount + extendedSettings: + nodeSelector: + "kubernetes.io/hostname": node-1 + envVariables: + JAVA_TOOL_OPTIONS: > + -XX:+ExitOnOutOfMemoryError + -XX:+UseContainerSupport + -Dlog4j2.shutdownHookEnabled=false + -XX:MaxRAMPercentage=84.2 + -XX:MaxMetaspaceSize=70M + -XX:CompressedClassSpaceSize=10M + -XX:ReservedCodeCacheSize=40M + -XX:MaxDirectMemorySize=50M + -Ddatastax-java-driver.advanced.connection.init-query-timeout="5000 milliseconds" + -Ddatastax-java-driver.basic.request.timeout="3 seconds" + resources: + limits: + cpu: 3000m + memory: 2000Mi + requests: + cpu: 1000m + memory: 100Mi +``` +## performance test parameters + +### cradle parameters +Only one page is used for the whole test because use case is uploading 100'000 events per second into usual namespace (one cradle page per day) during short period + +### events parameters +* tree structure: one root, several children in batches +

+* body: generated JSON with 1.1 kB size +* attached messages: 1 +* status: success + +### steps parameters for woodpeckers x2 +| step | maxBatchSize | duration | +|:-----|:-------------|:---------| +| 1 | 300 | 30 min | +| 2 | 400 | ~30 min | + +## metrics during performance tests execution + +### events received by estore (avg) + +| step | events in batch | batch size
(serialized protobuf) | event size
(serialized protobuf) | event rate | +|------|-----------------|--------------------------------------|--------------------------------------|--------------| +| 1 | 299 | 436 KiB | 1.46 KiB | 99K item/s | +| 2 | 400 | 582 KiB | 1.46 KiB | 80.6K item/s | + +![monitoring estore: event rate](monitoring_estore_event_rate.png) +![monitoring estore: event size](monitoring_estore_event_size.png) + +### cradle (avg) + +| step | raw rate | compressed rate | compression | +|------|-----------|-----------------|-------------| +| 1 | 127 MiB/s | 23.5 MiB/s | 5.39 times | +| 2 | 103 MiB/s | 19.1 MiB/s | 5.40 times | + +![estore: event batch content size](estore_event_batch_content_size.png) + +### computation resources by estore (max) + +| step | CPU | CPU throttling | RAM | +|------|-----|----------------|----------| +| 1 | 2.6 | 13.2 % | 1.9 GiB | +| 2 | 2.9 | 70.7 % | 1.91 GiB | + +![computation resources estore](computation_resources_estore.png) + +### JVM metrics of estore (max) + +| step | HEAP | old gen | yang gen | GC count
old gen | GC count
yang gen | +|------|----------|----------|----------|----------------------|-----------------------| +| 1 | 1.13 GiB | 881 MiB | 992 MiB | 0 | 218 times/s | +| 2 | 1.13 GiB | 1.03 GiB | 449 MiB | 0 | 815 times/s | + +![JVM estore: memory](jvm_estore_memory.png) +![JVM estore: GC](jvm_estore_gc.png) + +## Computation resources RabbitMQ (max) + +| step | CPU | RAM | +|------|------|----------| +| 1 | 0.73 | 594 MiB | +| 2 | 1.62 | 1.99 GiB | + +![computation resources rabbitmq](computation_resources_rabbitmq.png) + +## Computation resources Cassandra (max) + +| step | CPU | RAM | Network | +|------|------|---------|-------------| +| 1 | 1.35 | 5.8 GiB | 290.41 Mbps | +| 2 | 1.2 | 5.7 GiB | 273.12 Mbps | + +![zabbix cassandra: CPU](zabbix_cassandra_cpu.png) +![zabbix cassandra: RAM](zabbix_cassandra_ram.png) +![zabbix cassandra: Network](zabbix_cassandra_network.png) \ No newline at end of file diff --git a/doc/perftest-5.7.0-dev/zabbix_cassandra_cpu.png b/doc/perftest-5.7.0-dev/zabbix_cassandra_cpu.png new file mode 100644 index 0000000..4f3597e Binary files /dev/null and b/doc/perftest-5.7.0-dev/zabbix_cassandra_cpu.png differ diff --git a/doc/perftest-5.7.0-dev/zabbix_cassandra_network.png b/doc/perftest-5.7.0-dev/zabbix_cassandra_network.png new file mode 100644 index 0000000..96f301b Binary files /dev/null and b/doc/perftest-5.7.0-dev/zabbix_cassandra_network.png differ diff --git a/doc/perftest-5.7.0-dev/zabbix_cassandra_ram.png b/doc/perftest-5.7.0-dev/zabbix_cassandra_ram.png new file mode 100644 index 0000000..3e7bdb4 Binary files /dev/null and b/doc/perftest-5.7.0-dev/zabbix_cassandra_ram.png differ diff --git a/perf_chart.png b/perf_chart.png deleted file mode 100644 index 2f5ca91..0000000 Binary files a/perf_chart.png and /dev/null differ diff --git a/perftest/cassandra_disk_usage.png b/perftest/cassandra_disk_usage.png deleted file mode 100644 index 8682271..0000000 Binary files a/perftest/cassandra_disk_usage.png and /dev/null differ diff --git a/perftest/cluster_schema.png b/perftest/cluster_schema.png deleted file mode 100644 index aa74d0d..0000000 Binary files a/perftest/cluster_schema.png and /dev/null differ diff --git a/perftest/estore_heap.png b/perftest/estore_heap.png deleted file mode 100644 index de3ed1f..0000000 Binary files a/perftest/estore_heap.png and /dev/null differ diff --git a/perftest/estore_persist.png b/perftest/estore_persist.png deleted file mode 100644 index db8e8fa..0000000 Binary files a/perftest/estore_persist.png and /dev/null differ diff --git a/perftest/estore_res.png b/perftest/estore_res.png deleted file mode 100644 index d809781..0000000 Binary files a/perftest/estore_res.png and /dev/null differ diff --git a/perftest/perftest.md b/perftest/perftest.md deleted file mode 100644 index 17d7046..0000000 --- a/perftest/perftest.md +++ /dev/null @@ -1,124 +0,0 @@ -# Store events via th2-estore (perftest cluster) - -## Hardware configuration - -### Server - -+ **CPU**: Intel Xeon Gold 5218 x 2 -+ **RAM**: 768 GB (32 GB RAM x 24) -+ **Disk**: SAMSUNG MZ7LM3T8HMLP SSD 3,5Tb x 5 (raid5 + Hotspare) - -### Virtual machines are deployed on de-qa18 server: - -+ **perftest-cassandra01** - cassandra - + **CPU**: 4 core - + **RAM**: 8Gb -+ **kos-perftest-kuber-node01** - cluster node - + **CPU**: 11 core - + **RAM**: 30Gb -+ **kos-perftest-kuber-node02** - cluster node - + **CPU**: 11 core - + **RAM**: 30Gb - -## Software configuration (perftest cluster) - -### RabbitMQ -+ **VM**: kos-perftest-kuber-node01 -+ **Docker image**: docker.io/bitnami/rabbitmq:3.11.2-debian-11-r0 - -### Cassandra -+ **VM**: perftest-cassandra01 -+ **Version**: 4.0.5 -+ **Special java args**: - + -Xlog:gc=info,heap*=debug,age*=debug,safepoint=info,promotion*=debug... - + -Xms4G -Xmx4G - -## Cluster schema -![Cluster schema](cluster_schema.png) - -## th2 components configuration (perftest cluster) - -scheme configuration: -https://gitlab.exactpro.com/vivarium/th2/th2-internal-instances/th2-perf-schemas/-/commit/0b8dc314356aaa12816f0165408137ac24584781 - -### woodpecker -+ **Docker image**: ghcr.io/th2-net/th2-woodpecker-template:2.0.0-TH2-5204-estore-perf-9406747565-897a638 -+ **Data format**: protobuf - -### estore -```yaml -apiVersion: th2.exactpro.com/v2 -kind: Th2Estore -metadata: - name: estore -spec: - imageName: ghcr.io/th2-net/th2-estore - imageVersion: 5.7.0-dev - cradleManager: - prepareStorage: false - timeout: 5000 - pageSize: 1000 - composingServiceThreads: 4 - counterPersistenceInterval: 15000 - maxUncompressedTestEventSize: 0 - maxUncompressedMessageBatchSize: 0 - storeIndividualMessageSessions: false - compressionType: LZ4 - bookRefreshIntervalMillis: 60000 - customConfig: - maxTaskCount : 128 - maxTaskDataSize : 536870912 - maxRetryCount : 3 - mqRouter: - prefetchCount: 1000 - extendedSettings: - envVariables: - JAVA_TOOL_OPTIONS: > - -XX:+ExitOnOutOfMemoryError - -XX:+UseContainerSupport - -Dlog4j2.shutdownHookEnabled=false - -XX:MaxRAMPercentage=84.2 - -XX:MaxMetaspaceSize=70M - -XX:CompressedClassSpaceSize=10M - -XX:ReservedCodeCacheSize=40M - -XX:MaxDirectMemorySize=50M - -Ddatastax-java-driver.advanced.connection.init-query-timeout="5000 milliseconds" - -Ddatastax-java-driver.basic.request.timeout="3 seconds" - resources: - limits: - cpu: 2500m - memory: 2000Mi - requests: - cpu: 1000m - memory: 100Mi -``` - -## Test parameters -| woodpeckers (n) | batch size (events) | event size (KB) | attached messages (n) | load duration (min) | page duration (days) | -|-----------------|---------------------|-----------------|-----------------------|---------------------|----------------------| -| 2 | 700 | 1.37 | 1 | 40 | 1 | - -| component | rate (event/sec) | rate (MB/sec) | Total data size (GB) | Total events (n) | -|------------|------------------|---------------|----------------------|------------------| -| woodpecker | 50,000 | 67 | 157.0 | 90,000,000 | -| total x2 | 100,000 | 134 | 314.1 | 180,000,000 | - -## th2-estore metrics -![th2-estore metrics](estore_persist.png) - -## Cassandra metrics -![Cassandra metrics](cassandra_disk_usage.png) - -## Computation resources - -| | RabbitMQ | th2-estore | Cassandra | -|-------------|----------|------------|-----------| -| CPU (cores) | 0.6 | 2.3 | | -| RAM | 433 MB | 1.93 GB | 3.8 GB | - -## Computation resources th2-estore metrics -![th2-estore resources](estore_res.png)\ -![th2-estore heap usage](estore_heap.png) - -## Computation resources RabbitMQ metrics -![RabbitMQ resources](rabbitmq_res.png) \ No newline at end of file diff --git a/perftest/rabbitmq_cpu.png b/perftest/rabbitmq_cpu.png deleted file mode 100644 index cf5fd09..0000000 Binary files a/perftest/rabbitmq_cpu.png and /dev/null differ diff --git a/perftest/rabbitmq_ram.png b/perftest/rabbitmq_ram.png deleted file mode 100644 index a58e654..0000000 Binary files a/perftest/rabbitmq_ram.png and /dev/null differ diff --git a/perftest/rabbitmq_res.png b/perftest/rabbitmq_res.png deleted file mode 100644 index f36350c..0000000 Binary files a/perftest/rabbitmq_res.png and /dev/null differ