Skip to content

Commit

Permalink
KAFKA-1714: Fix gradle wrapper bootstrapping (apache#6031)
Browse files Browse the repository at this point in the history
Given we need to follow the Apache rule of not checking
any binaries into the source code, Kafka has always had
a bit of a tricky Gradle bootstrap.
Using ./gradlew as users expect doesn’t work and a
local and compatible version of Gradle was required to
generate the wrapper first.

This patch changes the behavior of the wrapper task to
instead generate a gradlew script that can bootstrap the
jar itself. Additionally it adds a license, removes the bat
script, and handles retries.

The documentation in the readme was also updated.

Going forward patches that upgrade gradle should run
`gradle wrapper` before checking in the change.

With this change users using ./gradlew can be sure they
are always building with the correct version of Gradle.

Reviewers: Viktor Somogyi <[email protected]>, Ismael Juma <[email protected]
  • Loading branch information
granthenke authored and ijuma committed Nov 21, 2019
1 parent 4e43124 commit 09f700a
Show file tree
Hide file tree
Showing 5 changed files with 323 additions and 15 deletions.
3 changes: 1 addition & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,7 @@ Vagrantfile.local
config/server-*
config/zookeeper-*
core/data/*
gradle/wrapper/*
gradlew
gradle/wrapper/*.jar
gradlew.bat

results
Expand Down
10 changes: 1 addition & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,12 @@ Apache Kafka
=================
See our [web site](https://kafka.apache.org) for details on the project.

You need to have [Gradle](https://www.gradle.org/installation) and [Java](https://www.oracle.com/technetwork/java/javase/downloads/index.html) installed.

Kafka requires Gradle 5.0 or higher.
You need to have [Java](http://www.oracle.com/technetwork/java/javase/downloads/index.html) installed.

Java 8 should be used for building in order to support both Java 8 and Java 11 at runtime.

Scala 2.12 is used by default, see below for how to use a different Scala version or all of the supported Scala versions.

### First bootstrap and download the wrapper ###
cd kafka_source_dir
gradle

Now everything else will work.

### Build a jar and run it ###
./gradlew jar

Expand Down
19 changes: 19 additions & 0 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#
# Copyright 2017 the original author or authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-5.0-all.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
199 changes: 199 additions & 0 deletions gradlew
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
#!/usr/bin/env sh
#
# Copyright 2017 the original author or authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

##############################################################################
##
## Gradle start up script for UN*X
##
##############################################################################

# Attempt to set APP_HOME
# Resolve links: $0 may be a link
PRG="$0"
# Need this for relative symlinks.
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`"/$link"
fi
done
SAVED="`pwd`"
cd "`dirname \"$PRG\"`/" >/dev/null
APP_HOME="`pwd -P`"
cd "$SAVED" >/dev/null

APP_NAME="Gradle"
APP_BASE_NAME=`basename "$0"`

# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m"'

# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD="maximum"

warn () {
echo "$*"
}

die () {
echo
echo "$*"
echo
exit 1
}

# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "`uname`" in
CYGWIN* )
cygwin=true
;;
Darwin* )
darwin=true
;;
MINGW* )
msys=true
;;
NONSTOP* )
nonstop=true
;;
esac


# Loop in case we encounter an error.
for attempt in 1 2 3; do
if [ ! -e $APP_HOME/gradle/wrapper/gradle-wrapper.jar ]; then
if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v5.0.0/gradle/wrapper/gradle-wrapper.jar"; then
rm -f "$APP_HOME/gradle/wrapper/gradle-wrapper.jar"
# Pause for a bit before looping in case the server throttled us.
sleep 5
continue
fi
fi
done

CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar

# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
else
JAVACMD="$JAVA_HOME/bin/java"
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD="java"
which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi

# Increase the maximum file descriptors if we can.
if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
MAX_FD_LIMIT=`ulimit -H -n`
if [ $? -eq 0 ] ; then
if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
MAX_FD="$MAX_FD_LIMIT"
fi
ulimit -n $MAX_FD
if [ $? -ne 0 ] ; then
warn "Could not set maximum file descriptor limit: $MAX_FD"
fi
else
warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
fi
fi

# For Darwin, add options to specify how the application appears in the dock
if $darwin; then
GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
fi

# For Cygwin, switch paths to Windows format before running java
if $cygwin ; then
APP_HOME=`cygpath --path --mixed "$APP_HOME"`
CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
JAVACMD=`cygpath --unix "$JAVACMD"`

# We build the pattern for arguments to be converted via cygpath
ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
SEP=""
for dir in $ROOTDIRSRAW ; do
ROOTDIRS="$ROOTDIRS$SEP$dir"
SEP="|"
done
OURCYGPATTERN="(^($ROOTDIRS))"
# Add a user-defined pattern to the cygpath arguments
if [ "$GRADLE_CYGPATTERN" != "" ] ; then
OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
fi
# Now convert the arguments - kludge to limit ourselves to /bin/sh
i=0
for arg in "$@" ; do
CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option

if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition
eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
else
eval `echo args$i`="\"$arg\""
fi
i=$((i+1))
done
case $i in
(0) set -- ;;
(1) set -- "$args0" ;;
(2) set -- "$args0" "$args1" ;;
(3) set -- "$args0" "$args1" "$args2" ;;
(4) set -- "$args0" "$args1" "$args2" "$args3" ;;
(5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
(6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
(7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
(8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
(9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
esac
fi

# Escape application args
save () {
for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
echo " "
}
APP_ARGS=$(save "$@")

# Collect all arguments for the java command, following the shell quoting and substitution rules
eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"

# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
cd "$(dirname "$0")"
fi

exec "$JAVACMD" "$@"
107 changes: 103 additions & 4 deletions wrapper.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,108 @@
* under the License.
*/

defaultTasks 'downloadWrapper'
// This file contains tasks for the gradle wrapper generation.

task downloadWrapper(type: Wrapper) {
description = "Download the gradle wrapper and requisite files. Overwrites existing wrapper files."
// Ensure the wrapper script is generated based on the version defined in the project
// and not the version installed on the machine running the task.
// Read more about the wrapper here: https://docs.gradle.org/current/userguide/gradle_wrapper.html
wrapper {
gradleVersion = project.gradleVersion
}
distributionType = Wrapper.DistributionType.ALL
}

def licenseString = """#
# Copyright 2017 the original author or authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License."""

// Custom task to inject support for downloading the gradle wrapper jar if it doesn't exist.
// This allows us to avoid checking in the jar to our repository.
// Additionally adds a license header to the wrapper while editing the file contents.
task bootstrapWrapper() {
// In the doLast block so this runs when the task is called and not during project configuration.
doLast {
def wrapperBasePath = "\$APP_HOME/gradle/wrapper"
def wrapperJarPath = wrapperBasePath + "/gradle-wrapper.jar"

// Add a trailing zero to the version if needed.
def fullVersion = project.gradleVersion.count(".") == 1 ? "${project.gradleVersion}.0" : versions.gradle
// Leverages the wrapper jar checked into the gradle project on github because the jar isn't
// available elsewhere. Using raw.githubusercontent.com instead of github.com because
// github.com servers deprecated TLSv1/TLSv1.1 support some time ago, so older versions
// of curl (built against OpenSSL library that doesn't support TLSv1.2) would fail to
// fetch the jar.
def wrapperBaseUrl = "https://raw.githubusercontent.com/gradle/gradle/v$fullVersion/gradle/wrapper"
def wrapperJarUrl = wrapperBaseUrl + "/gradle-wrapper.jar"

def bootstrapString = """
# Loop in case we encounter an error.
for attempt in 1 2 3; do
if [ ! -e $wrapperJarPath ]; then
if ! curl -s -S --retry 3 -L -o "$wrapperJarPath" "$wrapperJarUrl"; then
rm -f "$wrapperJarPath"
# Pause for a bit before looping in case the server throttled us.
sleep 5
continue
fi
fi
done
""".stripIndent()

def wrapperScript = wrapper.scriptFile
def wrapperLines = wrapperScript.readLines()
wrapperScript.withPrintWriter { out ->
def licenseWritten = false
def bootstrapWritten = false
wrapperLines.each { line ->
// Print the wrapper bootstrap before the first usage of the wrapper jar.
if (!bootstrapWritten && line.contains("gradle-wrapper.jar")) {
out.println(bootstrapString)
bootstrapWritten = true
}
out.print(line)
// Print the licence after the shebang.
if(!licenseWritten && line.contains("#!/usr/bin/env sh")) {
out.println()
out.print(licenseString)
licenseWritten = true
}
out.println() // New Line
}
}
}
}
wrapper.finalizedBy bootstrapWrapper

// Custom task to add a license header to the gradle-wrapper.properties file.
task bootstrapWrapperProperties() {
// In the doLast block so this runs when the task is called and not during project configuration.
doLast {
def wrapperProperties = wrapper.propertiesFile
def wrapperLines = wrapperProperties.readLines()
wrapperProperties.withPrintWriter { out ->
// Print the license
out.println(licenseString)
wrapperLines.each { line ->
out.println(line)
}
}
}
}
wrapper.finalizedBy bootstrapWrapperProperties

// Remove the generated batch file since we don't test building in the Windows environment.
task removeWindowsScript(type: Delete) {
delete "$rootDir/gradlew.bat"
}
wrapper.finalizedBy removeWindowsScript

0 comments on commit 09f700a

Please sign in to comment.