[GR-59063] Espresso: when continuations are enabled, ensure lambdas a…

…re always serializable.

PullRequest: graal/19055

espresso/docs/continuations.md

@@ -37,7 +37,8 @@ normally, or if an exception escaped).
 `Continuation` implements `Serializable` and can serialize to a backwards compatible format. Because frames can point to
 anything in their parameters and local variables, the class `ContinuationSerializable` provides static
 methods `readObjectExternal` and `writeObjectExternal` which may be used to coordinate serialization of
-continuation-related objects with a non-jdk serialization engine.
+continuation-related objects with a non-jdk serialization engine. Note that when the `--java.Continuum` flag is specified, 
+all lambdas are serializable but deserialization will require special support from your serializer engine.
 
 ## Security
 

espresso/docs/how-espresso-works.md

@@ -168,9 +168,8 @@ conflicts, and they are hidden from guest-world reflection.
 ### The stack
 
 Guest threads are run on host threads 1:1. That means guest stacks use host stacks, and for virtual threads to work the
-hosting JVM must support the combination of virtual threads and Truffle, which as of December 2023 HotSpot does not.
-Likewise guest exceptions are wrapped in `EspressoException` and then thrown, so the JVM running Espresso provides
-stack unwinding services and similar.
+hosting JVM must support the combination of virtual threads and Truffle. Guest exceptions are wrapped in
+`EspressoException` and then thrown, so the JVM running Espresso provides stack unwinding services and similar.
 
 Truffle provides stack frame management. A `nodes.BytecodeNode` receives a Truffle `VirtualFrame` object, which manages
 a series of slots stored on the host stack, or when _materialized_, the frame is stored on the heap. In dynamic Truffle
@@ -186,7 +185,32 @@ change) they are never being used inconsistently. Behind the scenes then we only
 separately from everything else so the GC can find them, and thus other types of stack slot are just stored in
 uninterpreted longs.
 
-## Substitutions and extension modules
+## Substitutions
+
+Espresso can replace or wrap any method in the JDK with code that runs in host space. Defining them is easy. Make a
+class in the `com.oracle.truffle.espresso.substitutions` package annotated with `@EspressoSubstitutions`. The name is a
+simple transform of the class containing the method to replace/wrap: for a class `a.b.Foo` the name of the substitution
+class should be `Target_a_b_Foo`.
+
+You can then define static methods annotated with `@Substitution`. The arguments should be of type `StaticObject` to
+refer to an object in the guest heap, annotated with `@Inject` if you want access to `Meta`, `EspressoLanguage` or other
+internal classes. You must annotate `StaticObject` usages with `@JavaType(Foo.class)` to ensure the prototype matches
+correctly.
+
+A substitution is a Truffle node and you can also define substitutions as `Node` subclasses directly instead of having
+them generated for you by the annotation processor. Therefore, you should carefully consider whether to apply
+`@TruffleBoundary` to your substitutions using the usual heuristics (should it be inlined into the caller?)
+
+If you need to wrap a method define two extra arguments:
+
+```
+@Bind("getMeta()") Meta meta,
+@Cached("create(meta.a_b_Foo.getCallTargetNoSubstitution())") DirectCallNode original
+```
+
+and then use `original.call(self);`. Obviously, add the relevant boilerplate to the `Meta` class for this to work.
+
+## Extension modules
 
 Espresso specific features may need guest-exposed APIs to control them. For example Espresso exposes a HotSwap control
 API that lets apps register for callbacks that run when the program code is mutated.

espresso/src/com.oracle.truffle.espresso/src/com/oracle/truffle/espresso/descriptors/Symbol.java

@@ -389,6 +389,10 @@ public static void ensureInitialized() {
  public static final Symbol<Name> ptypes = StaticSymbols.putName("ptypes");
  public static final Symbol<Name> rtype = StaticSymbols.putName("rtype");
 
+ // java.lang.invoke.LambdaMetafactory
+ public static final Symbol<Name> metafactory = StaticSymbols.putName("metafactory");
+ public static final Symbol<Name> altMetafactory = StaticSymbols.putName("altMetafactory");
+
  // j.l.ref.Finalizer
  public static final Symbol<Name> finalize = StaticSymbols.putName("finalize");
  public static final Symbol<Name> register = StaticSymbols.putName("register");
@@ -896,6 +900,7 @@ public static void ensureInitialized() {
  public static final Symbol<Type> java_lang_invoke_MethodHandles$Lookup = StaticSymbols.putType("Ljava/lang/invoke/MethodHandles$Lookup;");
  public static final Symbol<Type> java_lang_invoke_CallSite = StaticSymbols.putType("Ljava/lang/invoke/CallSite;");
  public static final Symbol<Type> java_lang_invoke_DirectMethodHandle = StaticSymbols.putType("Ljava/lang/invoke/DirectMethodHandle;");
+ public static final Symbol<Type> java_lang_invoke_LambdaMetafactory = StaticSymbols.putType("Ljava/lang/invoke/LambdaMetafactory;");
 
  // MethodHandleNatives is not public.
  public static final Symbol<Type> java_lang_invoke_MethodHandleNatives = StaticSymbols.putType("Ljava/lang/invoke/MethodHandleNatives;");
@@ -1265,6 +1270,21 @@ public static void ensureInitialized() {
  Type.java_lang_Object);
  public static final Symbol<Signature> MethodHandles$Lookup = StaticSymbols.putSignature(Type.java_lang_invoke_MethodHandles$Lookup);
 
+ public static final Symbol<Signature> CallSite_Lookup_String_MethodType_MethodType_MethodHandle_MethodType = StaticSymbols.putSignature(
+ Type.java_lang_invoke_CallSite,
+ Type.java_lang_invoke_MethodHandles$Lookup,
+ Type.java_lang_String,
+ Type.java_lang_invoke_MethodType,
+ Type.java_lang_invoke_MethodType,
+ Type.java_lang_invoke_MethodHandle,
+ Type.java_lang_invoke_MethodType);
+ public static final Symbol<Signature> CallSite_Lookup_String_MethodType_Object_array = StaticSymbols.putSignature(
+ Type.java_lang_invoke_CallSite,
+ Type.java_lang_invoke_MethodHandles$Lookup,
+ Type.java_lang_String,
+ Type.java_lang_invoke_MethodType,
+ Type.java_lang_Object_array);
+
  public static final Symbol<Signature> Field_Object_long_Class = StaticSymbols.putSignature(Type.java_lang_reflect_Field, Type.java_lang_Object, Type._long, Type.java_lang_Class);
 
  public static final Symbol<Signature> Thread$State_int = StaticSymbols.putSignature(Type.java_lang_Thread$State, Type._int);

espresso/src/com.oracle.truffle.espresso/src/com/oracle/truffle/espresso/meta/Meta.java

@@ -842,6 +842,10 @@ public Meta(EspressoContext context) {
  java_lang_invoke_MethodHandleNatives_linkDynamicConstant = null;
  }
 
+ ObjectKlass lambdaMetafactory = knownKlass(Type.java_lang_invoke_LambdaMetafactory);
+ java_lang_invoke_LambdaMetafactory_metafactory = lambdaMetafactory.requireDeclaredMethod(Name.metafactory, Signature.CallSite_Lookup_String_MethodType_MethodType_MethodHandle_MethodType);
+ java_lang_invoke_LambdaMetafactory_altMetafactory = lambdaMetafactory.requireDeclaredMethod(Name.altMetafactory, Signature.CallSite_Lookup_String_MethodType_Object_array);
+
  // Interop
  java_time_Duration = knownKlass(Type.java_time_Duration);
  java_time_Duration_seconds = java_time_Duration.requireDeclaredField(Name.seconds, Type._long);
@@ -1708,6 +1712,9 @@ private DiffVersionLoadHelper diff() {
  public final Method java_lang_invoke_MethodHandleNatives_linkCallSite;
  public final Method java_lang_invoke_MethodHandleNatives_linkDynamicConstant;
 
+ public final Method java_lang_invoke_LambdaMetafactory_metafactory;
+ public final Method java_lang_invoke_LambdaMetafactory_altMetafactory;
+
  public final Method java_lang_Object_wait;
  public final Method java_lang_Object_toString;
 

espresso/src/com.oracle.truffle.espresso/src/com/oracle/truffle/espresso/substitutions/Target_java_lang_invoke_LambdaMetafactory.java

@@ -0,0 +1,114 @@
+/*
+ * Copyright (c) 2021, 2023, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 only, as
+ * published by the Free Software Foundation.
+ *
+ * This code is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * version 2 for more details (a copy is included in the LICENSE file that
+ * accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License version
+ * 2 along with this work; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+ * or visit www.oracle.com if you need additional information or have any
+ * questions.
+ */
+
+package com.oracle.truffle.espresso.substitutions;
+
+import com.oracle.truffle.api.dsl.Bind;
+import com.oracle.truffle.api.dsl.Cached;
+import com.oracle.truffle.api.dsl.Specialization;
+import com.oracle.truffle.api.nodes.DirectCallNode;
+import com.oracle.truffle.espresso.meta.Meta;
+import com.oracle.truffle.espresso.runtime.EspressoContext;
+import com.oracle.truffle.espresso.runtime.staticobject.StaticObject;
+
+import java.lang.invoke.CallSite;
+import java.lang.invoke.LambdaMetafactory;
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodType;
+
+/**
+ * This class exists for serializable continuations. It forcibly overrides the lambda generation
+ * mode to request that serializability is always included. This is insufficient to enable
+ * serialization with ObjectOutputStream because the standard Java protocol requires code generation
+ * by javac, but it <i>is</i> sufficient to make lambdas transparently serializable when using a
+ * custom serializer.
+ */
+@EspressoSubstitutions
+public final class Target_java_lang_invoke_LambdaMetafactory {
+ @Substitution
+ abstract static class Metafactory extends SubstitutionNode {
+ abstract @JavaType(CallSite.class) StaticObject execute(
+ @JavaType(internalName = "Ljava/lang/invoke/MethodHandles$Lookup;") StaticObject caller,
+ @JavaType(String.class) StaticObject interfaceMethodName,
+ @JavaType(MethodType.class) StaticObject factoryType,
+ @JavaType(MethodType.class) StaticObject interfaceMethodType,
+ @JavaType(MethodHandle.class) StaticObject implementation,
+ @JavaType(MethodType.class) StaticObject dynamicMethodType);
+
+ @Specialization
+ @JavaType(CallSite.class)
+ StaticObject doCached(
+ @JavaType(internalName = "Ljava/lang/invoke/MethodHandles$Lookup;") StaticObject caller,
+ @JavaType(String.class) StaticObject interfaceMethodName,
+ @JavaType(MethodType.class) StaticObject factoryType,
+ @JavaType(MethodType.class) StaticObject interfaceMethodType,
+ @JavaType(MethodHandle.class) StaticObject implementation,
+ @JavaType(MethodType.class) StaticObject dynamicMethodType,
+ @Bind("getMeta()") Meta meta,
+ @Cached("create(meta.java_lang_invoke_LambdaMetafactory_altMetafactory.getCallTargetNoSubstitution())") DirectCallNode altMetafactory,
+ @Cached("create(meta.java_lang_invoke_LambdaMetafactory_metafactory.getCallTargetNoSubstitution())") DirectCallNode original,
+ @Bind("getContext()") EspressoContext context) {
+ if (context.getEspressoEnv().Continuum) {
+ // altMetafactory has a curious calling convention, apparently designed for
+ // extensibility.
+ StaticObject extraArgsRef = context.getAllocator().createNewReferenceArray(meta.java_lang_Object, 4);
+ StaticObject[] extraArgs = extraArgsRef.unwrap(context.getLanguage());
+ extraArgs[0] = interfaceMethodType;
+ extraArgs[1] = implementation;
+ extraArgs[2] = dynamicMethodType;
+ extraArgs[3] = (StaticObject) meta.java_lang_Integer_valueOf.getCallTarget().call(LambdaMetafactory.FLAG_SERIALIZABLE);
+ return (StaticObject) altMetafactory.call(caller, interfaceMethodName, factoryType, extraArgsRef);
+ } else {
+ return (StaticObject) original.call(caller, interfaceMethodName, factoryType, interfaceMethodType, implementation, dynamicMethodType);
+ }
+ }
+ }
+
+ @Substitution
+ abstract static class AltMetafactory extends SubstitutionNode {
+ abstract @JavaType(CallSite.class) StaticObject execute(
+ @JavaType(internalName = "Ljava/lang/invoke/MethodHandles$Lookup;") StaticObject caller,
+ @JavaType(String.class) StaticObject interfaceMethodName,
+ @JavaType(MethodType.class) StaticObject factoryType,
+ @JavaType(Object[].class) StaticObject args);
+
+ @Specialization
+ @JavaType(CallSite.class)
+ StaticObject doCached(
+ @JavaType(internalName = "Ljava/lang/invoke/MethodHandles$Lookup;") StaticObject caller,
+ @JavaType(String.class) StaticObject interfaceMethodName,
+ @JavaType(MethodType.class) StaticObject factoryType,
+ @JavaType(Object[].class) StaticObject args,
+ @Bind("getMeta()") Meta meta,
+ @Cached("create(meta.java_lang_invoke_LambdaMetafactory_altMetafactory.getCallTargetNoSubstitution())") DirectCallNode original,
+ @Bind("getContext()") EspressoContext context) {
+ if (context.getEspressoEnv().Continuum) {
+ StaticObject[] extraArgs = args.unwrap(context.getLanguage());
+ extraArgs[3] = meta.boxInteger(meta.unboxInteger(extraArgs[3]) | LambdaMetafactory.FLAG_SERIALIZABLE);
+ return (StaticObject) original.call(caller, interfaceMethodName, factoryType, StaticObject.wrap(extraArgs, meta));
+ } else {
+ return (StaticObject) original.call(caller, interfaceMethodName, factoryType, args);
+ }
+ }
+ }
+}