Skip to content

Commit

Permalink
KAFKA-13273: Add support for Java 17 (apache#11296)
Browse files Browse the repository at this point in the history
Java 17 is at release candidate stage and it will be a LTS release once
it's out (previous LTS release was Java 11).

Details:
* Replace Java 16 with Java 17 in Jenkins and Readme.
* Replace `--illegal-access=permit` (which was removed from Java 17)
   with  `--add-opens` for the packages we require internal access to.
   Filed KAFKA-13275 for updating the tests not to require `--add-opens`
   (where possible).
* Update `release.py` to use JDK8. and JDK 17 (instead of JDK 8 and JDK 15).
* Removed all but one Streams test from `testsToExclude`. The
   Connect test exclusion list remains the same.
* Add notable change to upgrade.html
* Upgrade to Gradle 7.2 as it's required for proper Java 17 support.
* Upgrade mockito to 3.12.4 for better Java 17 support.
* Adjusted `KafkaRaftClientTest` and `QuorumStateTest` not to require
   private access to `jdk.internal.util.random`.

Reviewers: Manikumar Reddy <[email protected]>, Chia-Ping Tsai <[email protected]>
  • Loading branch information
ijuma authored Sep 6, 2021
1 parent 81667e2 commit 0118330
Show file tree
Hide file tree
Showing 12 changed files with 125 additions and 79 deletions.
12 changes: 6 additions & 6 deletions Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -142,10 +142,10 @@ pipeline {
}
}

stage('JDK 16 and Scala 2.13') {
stage('JDK 17 and Scala 2.13') {
agent { label 'ubuntu' }
tools {
jdk 'jdk_16_latest'
jdk 'jdk_17_latest'
}
options {
timeout(time: 8, unit: 'HOURS')
Expand All @@ -157,7 +157,7 @@ pipeline {
steps {
doValidation()
doTest(env)
echo 'Skipping Kafka Streams archetype test for Java 16'
echo 'Skipping Kafka Streams archetype test for Java 17'
}
}

Expand Down Expand Up @@ -231,14 +231,14 @@ pipeline {
}
}

stage('JDK 16 and Scala 2.12') {
stage('JDK 17 and Scala 2.12') {
when {
not { changeRequest() }
beforeAgent true
}
agent { label 'ubuntu' }
tools {
jdk 'jdk_16_latest'
jdk 'jdk_17_latest'
}
options {
timeout(time: 8, unit: 'HOURS')
Expand All @@ -250,7 +250,7 @@ pipeline {
steps {
doValidation()
doTest(env)
echo 'Skipping Kafka Streams archetype test for Java 16'
echo 'Skipping Kafka Streams archetype test for Java 17'
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ See our [web site](https://kafka.apache.org) for details on the project.

You need to have [Java](http://www.oracle.com/technetwork/java/javase/downloads/index.html) installed.

We build and test Apache Kafka with Java 8, 11 and 16. We set the `release` parameter in javac and scalac
We build and test Apache Kafka with Java 8, 11 and 17. We set the `release` parameter in javac and scalac
to `8` to ensure the generated binaries are compatible with Java 8 or higher (independently of the Java version
used for compilation). Java 8 support has been deprecated since Apache Kafka 3.0 and will be removed in Apache
Kafka 4.0 (see [KIP-750](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223) for more details).
Expand Down
23 changes: 18 additions & 5 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,22 @@ ext {

defaultMaxHeapSize = "2g"
defaultJvmArgs = ["-Xss4m", "-XX:+UseParallelGC"]
if (JavaVersion.current() == JavaVersion.VERSION_16)
defaultJvmArgs.add("--illegal-access=permit")

// "JEP 403: Strongly Encapsulate JDK Internals" causes some tests to fail when they try
// to access internals (often via mocking libraries). We use `--add-opens` as a workaround
// for now and we'll fix it properly (where possible) via KAFKA-13275.
if (JavaVersion.current().isCompatibleWith(JavaVersion.VERSION_16))
defaultJvmArgs.addAll(
"--add-opens=java.base/java.io=ALL-UNNAMED",
"--add-opens=java.base/java.nio=ALL-UNNAMED",
"--add-opens=java.base/java.nio.file=ALL-UNNAMED",
"--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
"--add-opens=java.base/java.util.regex=ALL-UNNAMED",
"--add-opens=java.base/java.util.stream=ALL-UNNAMED",
"--add-opens=java.base/java.text=ALL-UNNAMED",
"--add-opens=java.base/java.time=ALL-UNNAMED",
"--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"
)

userMaxForks = project.hasProperty('maxParallelForks') ? maxParallelForks.toInteger() : null
userIgnoreFailures = project.hasProperty('ignoreFailures') ? ignoreFailures : false
Expand Down Expand Up @@ -359,7 +373,7 @@ subprojects {
// The suites are for running sets of tests in IDEs.
// Gradle will run each test class, so we exclude the suites to avoid redundantly running the tests twice.
def testsToExclude = ['**/*Suite.class']
// Exclude PowerMock tests when running with Java 16 until a version of PowerMock that supports Java 16 is released
// Exclude PowerMock tests when running with Java 16 or newer until a version of PowerMock that supports the relevant versions is released
// The relevant issues are https://github.com/powermock/powermock/issues/1094 and https://github.com/powermock/powermock/issues/1099
if (JavaVersion.current().isCompatibleWith(JavaVersion.VERSION_16)) {
testsToExclude.addAll([
Expand All @@ -372,8 +386,7 @@ subprojects {
"**/WorkerSinkTaskTest.*", "**/WorkerSinkTaskThreadedTest.*", "**/WorkerSourceTaskTest.*",
"**/WorkerTaskTest.*", "**/WorkerTest.*", "**/RestServerTest.*",
// streams tests
"**/KafkaStreamsTest.*", "**/RepartitionTopicsTest.*", "**/RocksDBMetricsRecorderTest.*",
"**/StreamsMetricsImplTest.*", "**/StateManagerUtilTest.*", "**/TableSourceNodeTest.*"
"**/KafkaStreamsTest.*"
])
}

Expand Down
7 changes: 6 additions & 1 deletion docs/upgrade.html
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,12 @@

<script id="upgrade-template" type="text/x-handlebars-template">

<h5><a id="upgrade_300_notable" href="#upgrade_300_notable">Notable changes in 3.0.0</a></h5>
<h5><a id="upgrade_310_notable" href="#upgrade_310_notable">Notable changes in 3.1.0</a></h5>
<ul>
<li>Apache Kafka supports Java 17.</li>
</ul>

<h5><a id="upgrade_310_notable" href="#upgrade_300_notable">Notable changes in 3.0.0</a></h5>
<ul>
<li>ZooKeeper has been upgraded to version 3.6.3.</li>
<li>A preview of KRaft mode is available, though upgrading to it from the 2.8 Early Access release is not possible. See
Expand Down
4 changes: 2 additions & 2 deletions gradle/dependencies.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ versions += [
checkstyle: "8.36.2",
commonsCli: "1.4",
dropwizardMetrics: "4.1.12.1",
gradle: "7.1.1",
gradle: "7.2",
grgit: "4.1.0",
httpclient: "4.5.13",
easymock: "4.3",
Expand Down Expand Up @@ -100,7 +100,7 @@ versions += [
lz4: "1.7.1",
mavenArtifact: "3.8.1",
metrics: "2.2.0",
mockito: "3.9.0",
mockito: "3.12.4",
netty: "4.1.62.Final",
powermock: "2.0.9",
reflections: "0.9.12",
Expand Down
4 changes: 2 additions & 2 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionSha256Sum=9bb8bc05f562f2d42bdf1ba8db62f6b6fa1c3bf6c392228802cc7cb0578fe7e0
distributionUrl=https\://services.gradle.org/distributions/gradle-7.1.1-all.zip
distributionSha256Sum=a8da5b02437a60819cad23e10fc7e9cf32bcb57029d9cb277e26eeff76ce014b
distributionUrl=https\://services.gradle.org/distributions/gradle-7.2-all.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
4 changes: 2 additions & 2 deletions gradlew
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ case "`uname`" in
Darwin* )
darwin=true
;;
MINGW* )
MSYS* | MINGW* )
msys=true
;;
NONSTOP* )
Expand All @@ -84,7 +84,7 @@ esac
# Loop in case we encounter an error.
for attempt in 1 2 3; do
if [ ! -e "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" ]; then
if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v7.1.1/gradle/wrapper/gradle-wrapper.jar"; then
if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v7.2.0/gradle/wrapper/gradle-wrapper.jar"; then
rm -f "$APP_HOME/gradle/wrapper/gradle-wrapper.jar"
# Pause for a bit before looping in case the server throttled us.
sleep 5
Expand Down
53 changes: 17 additions & 36 deletions raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,7 @@ public void testRejectVotesFromSameEpochAfterResigningLeadership() throws Except
int epoch = 2;

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withElectedLeader(epoch, localId)
.build();

Expand All @@ -126,9 +124,7 @@ public void testRejectVotesFromSameEpochAfterResigningCandidacy() throws Excepti
int epoch = 2;

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withVotedCandidate(epoch, localId)
.build();

Expand All @@ -151,11 +147,9 @@ public void testGrantVotesFromHigherEpochAfterResigningLeadership() throws Excep
int epoch = 2;

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.withElectedLeader(epoch, localId)
.build();
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withElectedLeader(epoch, localId)
.build();

// Resign from leader, will restart in resigned state
assertTrue(context.client.quorum().isResigned());
Expand All @@ -181,11 +175,9 @@ public void testGrantVotesFromHigherEpochAfterResigningCandidacy() throws Except
int epoch = 2;

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.withVotedCandidate(epoch, localId)
.build();
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withVotedCandidate(epoch, localId)
.build();

// Resign from candidate, will restart in candidate state
assertTrue(context.client.quorum().isCandidate());
Expand Down Expand Up @@ -235,11 +227,9 @@ public void testInitializeAsResignedAndBecomeCandidate() throws Exception {
int epoch = 2;

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.withElectedLeader(epoch, localId)
.build();
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withElectedLeader(epoch, localId)
.build();

// Resign from leader, will restart in resigned state
assertTrue(context.client.quorum().isResigned());
Expand All @@ -262,9 +252,7 @@ public void testInitializeAsResignedLeaderFromStateStore() throws Exception {
int epoch = 2;

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withElectedLeader(epoch, localId)
.build();

Expand Down Expand Up @@ -728,9 +716,7 @@ public void testEndQuorumIgnoredAsCandidateIfOlderEpoch() throws Exception {
Set<Integer> voters = Utils.mkSet(localId, otherNodeId);

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(jitterMs).when(random).nextInt(Mockito.anyInt());
})
.updateRandom(r -> r.mockNextInt(jitterMs))
.withUnknownLeader(epoch - 1)
.build();

Expand Down Expand Up @@ -1238,9 +1224,7 @@ public void testRetryElection() throws Exception {
Set<Integer> voters = Utils.mkSet(localId, otherNodeId);

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(exponentialFactor).when(random).nextInt(Mockito.anyInt());
})
.updateRandom(r -> r.mockNextInt(exponentialFactor))
.build();

context.assertUnknownLeader(0);
Expand Down Expand Up @@ -2184,9 +2168,7 @@ public void testFetchShouldBeTreatedAsLeaderAcknowledgement() throws Exception {
Set<Integer> voters = Utils.mkSet(localId, otherNodeId);

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withUnknownLeader(epoch - 1)
.build();

Expand Down Expand Up @@ -2395,9 +2377,7 @@ public void testClusterAuthorizationFailedInBeginQuorumEpoch() throws Exception
Set<Integer> voters = Utils.mkSet(localId, otherNodeId);

RaftClientTestContext context = new RaftClientTestContext.Builder(localId, voters)
.updateRandom(random -> {
Mockito.doReturn(0).when(random).nextInt(DEFAULT_ELECTION_TIMEOUT_MS);
})
.updateRandom(r -> r.mockNextInt(DEFAULT_ELECTION_TIMEOUT_MS, 0))
.withUnknownLeader(epoch - 1)
.build();

Expand Down Expand Up @@ -2799,4 +2779,5 @@ public void testObserverFetchWithNoLocalId() throws Exception {
private static KafkaMetric getMetric(final Metrics metrics, final String name) {
return metrics.metrics().get(metrics.metricName(name, "raft-metrics"));
}

}
51 changes: 51 additions & 0 deletions raft/src/test/java/org/apache/kafka/raft/MockableRandom.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kafka.raft;

import java.util.OptionalInt;
import java.util.Random;
import java.util.function.IntFunction;

/**
* A Random instance that makes it easy to modify the behavior of certain methods for test purposes.
*/
class MockableRandom extends Random {

private IntFunction<OptionalInt> nextIntFunction = __ -> OptionalInt.empty();

public MockableRandom(long seed) {
super(seed);
}

public void mockNextInt(int expectedBound, int returnValue) {
this.nextIntFunction = b -> {
if (b == expectedBound)
return OptionalInt.of(returnValue);
else
return OptionalInt.empty();
};
}

public void mockNextInt(int returnValue) {
this.nextIntFunction = __ -> OptionalInt.of(returnValue);
}

@Override
public int nextInt(int bound) {
return nextIntFunction.apply(bound).orElse(super.nextInt(bound));
}
}
Loading

0 comments on commit 0118330

Please sign in to comment.