How are you producing openjdk-8u222-jvmci-19.3-b03-darwin-amd64.tar.gz?

[chrisseaton]

I don't understand how you're producing for example openjdk-8u222-jvmci-19.3-b03-darwin-amd64.tar.gz.

I can't find a build script for it in this repo.

When I try to do the obvious thing and build graal-jvmci-8 with a bootstrap Java 8 I don't get the Contents/Home/jre/lib/*.a files that SVM needs now. That's trying with both AdoptOpenJDK and Oracle JDK (but that's 8u221, so I'm even more confused.)

Can you share how you're managing to build graal-jvmci-8 and get .a files?

Things I don't understand:

• what are your build commands and what JVM do you use?
• why do none of my JVM's include .a files and whatever your using does?
• isn't graal-jvmci-8 a full JVM? Why doesn't it produce the .a files?

[dougxc]

Building these JVMCI JDK8 binaries is a 2 step process:

1. Build a base JDK and the .a files for the base JDK.
2. Build a JVMCI JDK with mx --java-home=/path/to/base/JDK build in the graal-jvmci-8 repo.

The base JDKs are produced internally using an Oracle build farm that provides all the right machines with the right OSes to satisfy the build requirements for JDK 8.

The .a files are produced by applying a set of patches to JDK 8 and then using the same build farm. It should then be a matter of using the --enable-static-build=yes option to configure to produce the .a files.

[chrisseaton]

So what has changed is I now can't use an unmodified, already built, JDK as the base anymore? I was using the one in Ubuntu on Linux, and AdoptOpenJDK on macOS, for that before and it was working fine.

It doesn't work anymore because we now need the static libraries and I need to apply patches to the base JDK before building it myself in order to get those, is that right?

But I don't see this repo using those patches, or using a build from your farm, so how is it working here?

And isn't graal-jvmci-8 a full JDK source itself? Why can't we build it without copying things like static libraries from a base JDK?

Sorry for the continued questions but I can't reproduce an open source GraalVM build without binary blobs from Oracle anymore!

[dougxc]

In random order:

• graal-jvmci-8 is not a full JDK - it's just the hotspot/ tree from a JDK8 forest.
• This repo is only used to host the binaries. It is no longer an active build recipe.

[chrisseaton]

Right - it looks like this repo is set up to do the deploys, but then it actually fails in Travis https://travis-ci.org/graalvm/openjdk8-jvmci-builder/jobs/591209944#L1839.

[dougxc]

I've just pushed a change now that clarifies the current purpose of this repo.

[chrisseaton]

I wasn't able to apply those patches on top of http://hg.openjdk.java.net/jdk8u/jdk8u-dev/, tag jdk8u232-b05. patches/jdk.patch does not apply - there is no make/BuildJdk.gmk.

At the moment I'm not able to build GraalVM from source without using what is effectively a closed-source binary from Oracle.

Will the situation be better if I use Java 11 instead? Do we have a repository and set of patches from which I could build JDK JVMCI 11? Could that be documented if it isn't already?

CC @gilles-duboscq

[dougxc]

You need to apply jdk.patch against http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk.

[dougxc]

JDK JVMCI 11 is built from https://github.com/graalvm/labs-openjdk-11. You should be able to work out how to build it based on ci.jsonnet.

[helloguo]

Building these JVMCI JDK8 binaries is a 2 step process:

1. Build a base JDK and the .a files for the base JDK.
2. Build a JVMCI JDK with mx --java-home=/path/to/base/JDK build in the graal-jvmci-8 repo.

The base JDKs are produced internally using an Oracle build farm that provides all the right machines with the right OSes to satisfy the build requirements for JDK 8.

The .a files are produced by applying a set of patches to JDK 8 and then using the same build farm. It should then be a matter of using the --enable-static-build=yes option to configure to produce the .a files.

@dougxc Can you explain more about how to build a base JDK and the .a files for the base JDK? Here is what I did for building them from source code:

• hg clone --verbose https://hg.openjdk.java.net/jdk8u/jdk8u-dev/ -r aa1559aa1f70 jdk8u232-b06
• cd jdk8u232-b06
• sh ./get_source.sh
• Download and unzip above patches.
• In dir /path/to/jdk8u232-b06, patch -p1 < top.patch
• In dir /path/to/jdk8u232-b06/jdk, patch -p1 < jdk.patch
• In dir /path/to/jdk8u232-b06, bash ./configure --enable-static-build=yes

Then I got the error:

Running generated-configure.sh
configure: error: unrecognized options: --enable-static-build
configure exiting with result code 1

It seems that the configure file does not know the flag --enable-static-build. How to pass it to the configure file?

[gilles-duboscq]

@helloguo, are you sure the patches applied cleanly? The --enable-static-build is provided by top.patch. I ran the exact steps you mentioned and it worked fine.

[dougxc]

BTW, if you're interested in JDK 11, there is now a build script that should make life easier.

[helloguo]

@gilles-duboscq Thanks for checking. I still cannot build it on my machine. Maybe there's something wrong on my machine. If possible, can you share the recipe to build JVMCI-enabled-JDK8 from source?

@dougxc It would be super useful if there's a build script for JDK8.

[dougxc]

It would be super useful if there's a build script for JDK8.

We have a script for building the base jdk8 and static libs but it does little more than launch a build on an internal build farm that has all the right setup for building jdk8.

In terms of building a jvmci-jdk8 on top of a base jdk8 and jdk8 static libs, there's little more to it than:

graal-jvmci-8> export JAVA_HOME=/path/to/base-jdk8
graal-jvmci-8> mx build
graal-jvmci-8> cp /path/to/static/libs/*.a $(mx jdkhome)/jre/lib

After this, JAVA_HOME points to a jvmci-jdk8.

[gilles-duboscq]

can you share the recipe to build JVMCI-enabled-JDK8 from source

Well, it's exactly what you described above followed by make images

[hlolli]

@helloguo just in case someone's here, then the trick to avoid the the unrecognized options: --enable-static-build is to regenerate the configure file, I call (cd common/autoconf && bash autogen.sh) and then there's a new configure file which knows about the patched flag.

But in my case, that took me to a series of errors saying

Error occurred during initialization of VM
java/lang/ClassNotFoundException: error in opening JAR file <Zip file open error> openjdk/jre/lib/rt.jar

My gut tells me that I need to provide dependencies like zlib as .a static lib. (Will try to remember to update here if that's the case).

[tianxiaogu]

I got the same ZIP file opening error with GLIBC 2.5.

The following flags are removed when linking libjava.so with --enable-static-build=yes.

-Xlinker
-version-script=jdk/make/mapfiles/libjava/mapfile-vers

Then the function canonicalize defined in canonicalize_md.c will have a global symbol in libjava.so.

Unfortunately, in GLIBC 2.25, there is a new function canonicalize declared in math/bits/mathcalls.h (see https://sourceware.org/git/?p=glibc.git;a=blob;f=math/bits/mathcalls.h;h=3d3b4142b994e7f6253999318edb969153115f80;hb=58557c229319a3b8d2eefdb62e7df95089eabe37#l435).

The dynamic linker will use the definition of canonicalize in libm.so.

[gilles-duboscq]

I know the original issue is about darwin but here's what i do for Linux.

This is an example using Oracle Linux 7. I would recommend to use something from the OL6 or 7 era to get an older glib and thus better compatibility.

• Install dependencies

$ yum --enablerepo=ol7_optional_latest install tar gzip unzip python3 git java-1.8.0-openjdk-devel zip file which make gcc gcc-c++ libXtst-devel libXt-devel libXrender-devel cups-devel fontconfig-devel alsa-lib-devel libstdc++-static

Get an OpenJDK8 forest, then:

• Configure as necessary

$ sh configure \
--with-milestone=fcs --with-build-number=b09 --with-update-version=271 \
--with-stdc++lib=static \
--disable-debug-symbols \
--disable-freetype-bundling \
--with-zlib=system

I get the best results with:

os	arch	freetype bundling	zlib bundling
linux	amd64	disable	system
darwin	amd64	enable	system
windows	amd64	enable	bundled
solaris	sparcv9	disable	system

• Build the base JDK

$ make images

If you run into warnings while building (this can happen depending on the version of your c/c++ toolchain), you might want to configure with --disable-warnings-as-errors.

• Preserve that somewhere, here in ~

$ cp -a build/*-normal-server-release/images/j2sdk-image ~
$ BASE_JDK=~/j2sdk-image

• Build the static libraries

$ make clean
$ sh configure \
--with-milestone=fcs --with-build-number=b09 --with-update-version=271 \
--with-stdc++lib=static \
--disable-debug-symbols \
--disable-freetype-bundling \
--with-zlib=system \
--enable-static-build=yes
$ make images

Note: use the same configure arguments as for the base JDK build, just adding --enable-static-build=yes.

• Copy static libs into the JVMCI JDK

$ copy_static_libs() {
    find "$1" -name '*.a' | while read STATIC_LIB; do
        REL_PATH="${STATIC_LIB/$1/.}"
        echo "Copying $REL_PATH"
        cp "$STATIC_LIB" "$2/$REL_PATH"
    done
}
$ copy_static_libs build/*-normal-server-release/images/j2sdk-image "$BASE_JDK"

This will get you a base JDK with no JVMCI support.

• Building the JVMCI JDK

Clone graal-jvmci-8, put mx on your PATH and:

$ export MX_PYTHON=python3
$ env INCLUDE_HSDIS=false mx --java-home="${BASE_JDK}" build
$ JVMCI_JDK="$(realpath openjdk1.8.0_*/linux-amd64/product/)"

[helloguo]

@gilles-duboscq Thanks a lot for providing this detailed information!

Just want to confirm, when you say "Get an OpenJDK8 forest", you mean the OpenJDK8 source code with patches (top.patch and jdk.patch) applied, right? Does the above link of patches still work for the newer version of OpenJDK8 (e.g. jdk8u272-b10)? If not, where can I find the right patches? Or is there any repo that contains OpenJDK8 source code with patches applied?

[dougxc]

The latest patches are now here: https://github.com/graalvm/graal-jvmci-8/tree/master/patches
https://github.com/graalvm/graal-jvmci-8/blob/master/README.md describes how to apply the patches.