Skip to content

Commit

Permalink
test(jmeter): Add jmeter module (#30789) (#30842)
Browse files Browse the repository at this point in the history 
### Proposed Changes
* Add initial jmeter testing module that is not enabled in the CI builds
* See test-jmeter/README.md
* Must be enabled locally with -Djmeter.test.skip=false otherwise will
be skiped as a module in maven commands.
* Must test against a https server currently and is currently setup to
work with the new local docker-desktop k8s helm chart
* Should mainly test to ensure it does get skipped in the CI build and
is a WIP for further development and discussion.

### Additional Info
Related to #30789 (Provide basic mechanism to generate traffic and an).


### Screenshots
Original             |  Updated
:-------------------------:|:-------------------------:
** original screenshot **  |  ** updated screenshot **
  • Loading branch information
@spbolton
spbolton authored Dec 4, 2024
1 parent 3b866d6 commit 6bff5a0
Show file tree
Hide file tree
Showing 6 changed files with 1,900 additions and 0 deletions.
3 changes: 3 additions & 0 deletions justfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,9 @@ run-java-cli-native *ARGS:
tools/dotcms-cli/cli/target/dotcms-cli-1.0.0-SNAPSHOT-runner {{ARGS}}


run-jmeter-tests:
./mvnw verify -Djmeter.test.skip=false -pl :dotcms-test-jmeter

###########################################################
# Useful Maven Helper Commands
###########################################################
Expand Down
6 changes: 6 additions & 0 deletions parent/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@
<environment.properties.defaults>${environment.properties.folder}/environment.properties</environment.properties.defaults>
<spotless.skip>true</spotless.skip>
<glowroot.version>0.14.2</glowroot.version>
<jmeter.test.skip>true</jmeter.test.skip>
</properties>

<build>
Expand All @@ -223,6 +224,11 @@
<pluginManagement>

<plugins>
<plugin>
<groupId>com.lazerycode.jmeter</groupId>
<artifactId>jmeter-maven-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>flatten-maven-plugin</artifactId>
Expand Down
16 changes: 16 additions & 0 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,4 +102,20 @@
</extensions>
</build>

<profiles>
<profile>
<id>jmeter</id>
<activation>
<property>
<name>jmeter.test.skip</name>
<value>false</value>
</property>
</activation>
<modules>
<module>test-jmeter</module>
</modules>
</profile>
</profiles>


</project>
108 changes: 108 additions & 0 deletions test-jmeter/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# dotCMS JMeter Performance Tests

This module contains JMeter performance tests for dotCMS. The tests are configured to run against a dotCMS instance and measure various performance metrics.

This is a work in progress. It currently requires a https connection to the dotCMS instance to maintain session cookies. It will also not run in the CI environment and is only for local testing requiring the -Djmeter.test.skip=false flag to be set to enable


## Test Configuration

The JMeter tests are configured in `src/test/jmeter/sessions.jmx`. The default configuration includes:

- Host: dotcms.local
- Port: 443
- Ramp-up period: 0 seconds
- Startup delay: 5 seconds
- Test duration: 5 seconds

## Running the Tests

### Basic Execution

```bash
./mvnw install -Djmeter.test.skip=false -pl :dotcms-test-jmeter
```

you can also run the above with the justfile alias `run-jmeter-tests`:


```bash
just run-jmeter-tests
```

### Opening test script in JMeter GUI

```bash
cd test-jmeter
../mvnw jmeter:configure jmeter:gui -DguiTestFile=src/test/jmeter/sessions.jmx
````

### Overriding Test Parameters

You can override the default settings using command-line properties:

```bash
# Override host and port
./mvnw install -Djmeter.test.skip=false -pl :dotcms-test-jmeter \
-Djmeter.host=my-dotcms-instance.com \
-Djmeter.port=444 \ # default is 443
-Djmeter.thread.number=10 # The number of concurrent users to simulate
# Override test timing parameters
./mvnw install -Djmeter.test.skip=false -pl :dotcms-test-jmeter \
-Djmeter.rampup=10 \
-Djmeter.startup.delay=2 \
-Djmeter.test.duration=30
```

## Test Reports

HTML reports are generated in the `target/jmeter/reports` directory. The plugin is configured to:
- Delete existing results before new test runs

A csv is also generated in the `target/jmeter/results` directory. e.g. `20241203-sessions.csv` this contains
- Capture additional variables: JVM_HOSTNAME, SESSION_ID, X_DOT_SERVER
- The SESSION_ID and X_DOT_SERVER can be used in the csv to validate session propagation when there are multiple replicas and can be used to show behaviour when replicas are scaled up and down.
## Configuration Files

- Main JMeter test file: `src/test/jmeter/sessions.jmx`
- Maven configuration: `pom.xml`

## Properties Configuration

Default properties in pom.xml:
```xml
<properties>
<jmeter.host>dotcms.local</jmeter.host>
<jmeter.port>443</jmeter.port>
<jmeter.rampup>0</jmeter.rampup>
<jmeter.startup.delay>5</jmeter.startup.delay>
<jmeter.test.duration>60</jmeter.test.duration>
<jmeter.thread.number>2</jmeter.thread.number>
</properties>
```

## Profile Information

The tests run under the `jmeter-standalone` profile, which is active by default. This profile includes:
- Clean configuration for test reports
- JMeter test execution
- Results validation
- Report generation

## Troubleshooting

We have not yet validated the memory requirements. Eventually we should probably be explicit about the JVM memory settings. These can be added into the configuration block
in the pom.xml e.g.:
```xml
<jMeterProcessJVMSettings>
<arguments>
<argument>-XX:MaxMetaspaceSize=256m</argument>
<argument>-Xmx1024m</argument>
<argument>-Xms1024m</argument>
</arguments>
</jMeterProcessJVMSettings>
```
## High load testing
Currently this runs as a standalone service, for high load testing we would need to run this in a distributed mode with multiple jmeter instances and jmeter should not be running on the same server as DotCMS. This is not yet supported.
As such performance issues in adding too many threads may be due to local server limitations of resources and not the dotCMS instance itself.
132 changes: 132 additions & 0 deletions test-jmeter/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.dotcms</groupId>
<artifactId>dotcms-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
</parent>

<artifactId>dotcms-test-jmeter</artifactId>
<packaging>pom</packaging>

<name>test-jmeter</name>


<properties>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<jmeterScript>sessions.jmx</jmeterScript>
<jmeter.host>dotcms.local</jmeter.host>
<jmeter.port>443</jmeter.port>
<jmeter.rampup>0</jmeter.rampup>
<jmeter.startup.delay>5</jmeter.startup.delay>
<jmeter.test.duration>60</jmeter.test.duration>
<jmeter.thread.number>2</jmeter.thread.number>
</properties>

<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>

<profiles>
<profile>
<id>jmeter-standalone</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<build>
<plugins>

<plugin>
<artifactId>maven-clean-plugin</artifactId>
<executions>
<execution>
<id>clean-reports</id>
<phase>test</phase>
<goals>
<goal>clean</goal>
</goals>
<configuration>
<excludeDefaultDirectories>true</excludeDefaultDirectories>
<failOnError>false</failOnError>
<filesets>
<fileset>
<directory>target</directory>
<includes>
<include>jmeter/reports/**/*</include>
</includes>
</fileset>
</filesets>
</configuration>
</execution>
</executions>
</plugin>



<plugin>
<groupId>com.lazerycode.jmeter</groupId>
<artifactId>jmeter-maven-plugin</artifactId>
<executions>
<!-- Generate JMeter configuration -->
<execution>
<id>configuration</id>
<goals>
<goal>configure</goal>
</goals>
</execution>
<!-- Run JMeter tests -->
<execution>
<id>jmeter-tests</id>
<goals>
<goal>jmeter</goal>
</goals>
</execution>
<!-- Fail build on errors in test -->
<execution>
<id>jmeter-check-results</id>
<goals>
<goal>results</goal>
</goals>
</execution>

</executions>
<configuration>
<testFilesDirectory>${project.basedir}/src/test/jmeter</testFilesDirectory>
<testResultsTimestamp>true</testResultsTimestamp>

<propertiesUser>
<jmeter.save.saveservice.output_format>csv</jmeter.save.saveservice.output_format>
<CookieManager.save.cookies>true</CookieManager.save.cookies>
<sample_variables>JVM_HOSTNAME,SESSION_ID,X_DOT_SERVER</sample_variables>
<resultcollector.action_if_file_exists>DELETE</resultcollector.action_if_file_exists>
<host>${jmeter.host}</host>
<port>${jmeter.port}</port>
<rampup>${jmeter.rampup}</rampup>
<startup.delay>${jmeter.startup.delay}</startup.delay>
<test.duration>${jmeter.test.duration}</test.duration>
<thread.number>${jmeter.thread.number}</thread.number>
</propertiesUser>
<testFilesIncluded>
<jMeterTestFile>${jmeterScript}</jMeterTestFile>
</testFilesIncluded>
<generateReports>true</generateReports>


</configuration>
</plugin>

</plugins>
</build>
</profile>
</profiles>
</project>
Loading

0 comments on commit 6bff5a0

Please sign in to comment.