[POC] [Security Manager Replacement] GraalVM sandboxing #16863

reta · 2024-12-16T20:03:31Z

Description

Use GraalVM capability to spin off a separate JVM to host the sandboxed component. With this model, it becomes possible to:

run OpenSearch core on any JVM that GraalVM supports (without SM)
use older JDK versions (up to 23) with SM enabled for non-trusted components

The POC does a bare minimum work to host the ShiroIdentityPlugin in the separated JVM (21.0.5+11-Ubuntu-1ubuntu124.10) that is running under SecurityManager:

OpenSearch home: /home/opensearch-3.0.0-SNAPSHOT
Host JVM version: 22.0.2+9-jvmci-b01
Polyglot JVM version: 21.0.5+11-Ubuntu-1ubuntu124.10
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/home/opensearch-3.0.0-SNAPSHOT/lib/opensearch-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
Security Manager? org.opensearch.secure_sm.SecureSM@7b303608
[2024-12-16T15:00:18,828][WARN ][stderr                   ] [] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2024-12-16T15:00:18,864][WARN ][stderr                   ] [] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2024-12-16T15:00:18,875][WARN ][stderr                   ] [] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Shiro Plugin? org.opensearch.identity.shiro.ShiroIdentityPlugin@5ddd3150
Exception in thread "main" access denied ("java.net.SocketPermission" "localhost:0" "listen,resolve")
	at <java> checkPermission(Ljava/security/Permission;)V(Unknown)
	at <java> checkPermission(Ljava/security/Permission;)V(java/security/AccessController.java:1071:0)
	at <java> checkPermission(Ljava/security/Permission;)V(java/lang/SecurityManager.java:411:0)
	at <java> checkListen(I)V(java/lang/SecurityManager.java:985:0)
	at <java> bind(Ljava/net/SocketAddress;I)V(java/net/ServerSocket.java:387:0)
	at <java> <init>(IILjava/net/InetAddress;)V(java/net/ServerSocket.java:278:0)
	at <java> <init>(I)V(java/net/ServerSocket.java:171:0)
	at <java> getSocket()Ljava/net/ServerSocket;(org/opensearch/identity/shiro/ShiroIdentityPlugin.java:144:0)
	at <java> RootNode for interop message: 'invokeMember'.(Unknown)
	at org.graalvm.polyglot.Value.invokeMember(Value.java:1021)
	at org.opensearch.espresso.sandbox.Sandbox.loadPlugin(Sandbox.java:86)
	at org.opensearch.espresso.sandbox.Sandbox.main(Sandbox.java:22)

Related Issues

Closes #16861

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-12-16T20:09:39Z

❌ Gradle check result for a8e52f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-12-17T13:05:39Z

❌ Gradle check result for 6201d8c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2024-12-17T05:59:22Z

libs/espresso-sm/src/main/java/org/opensearch/espresso/sandbox/Sandbox.java

+            .allowHostAccess(HostAccess.NONE)
+            .allowIO(IOAccess.NONE)


while i understand this is POC for the shiro plugin, the sandbox restrictions for a different plugin could differ and we'd need to build and load a different context?

wonder if we set allowHostClassLoading(false), we'd still be able to load the plugin.

I think these settings are only "applicable" to embedded context, and once external JVM is started, all restrictions are off (this is why we need SM here).

You can easily see the confirmation here, in theory allowIO(IOAccess.NONE) would cut any I/O off but it does not, the SM does (if we comment out SM initialization, the socket will be created):

at <java> getSocket()Ljava/net/ServerSocket;(org/opensearch/identity/shiro/ShiroIdentityPlugin.java:144:0)

while i understand this is POC for the shiro plugin, the sandbox restrictions for a different plugin could differ and we'd need to build and load a different context?

This is out of scope for sandbox but in scope of SM of the spawn JVM process. We apply the same security policy that OpenSearch core does now.

Gotcha.

We apply the same security policy that OpenSearch core does now.

this also means we cannot apply/use the plugin level security policies?

this also means we cannot apply/use the plugin level security policies?

Quite an opposite :-) the OpenSearch core does include all plugin security policies into consideration. So it is kind of bootstrapping OpenSearch core + plugin(s), the policies will be glued together (as it does now)

You can easily see the confirmation here, in theory allowIO(IOAccess.NONE) would cut any I/O off but it does not, the SM does (if we comment out SM initialization, the socket will be created):

Thanks for pointing that.

kumargu · 2024-12-17T15:18:02Z

Thank-you again for putting this up.

Apart from the debugging pain, which i hope would be one-time while setting up a plugin); I don't see a reason why we would not include this as an alternative for SM. Let's see what others feel about it.

This would look much better, allowing plugins to move to JDK-24 with a real look-and-feel of a plugin sandbox env when Graal fully addresses oracle/graal#10239

reta · 2024-12-17T15:27:50Z

Apart from the debugging pain, which i hope would be one-time while setting up a plugin); I don't see a reason why we would not include this as an alternative for SM. Let's see what others feel about it.

I have included it here #16861 but to reiterate, the most difficult issue with such a model is communication between host and spawned JVM/context: it seems not being possible to wire up the services from the host JVM. I will spend more time to explore the limitations here.

Signed-off-by: Andriy Redko <[email protected]>

reta · 2024-12-18T19:19:02Z

plugins/identity-shiro/src/main/java/org/opensearch/identity/shiro/ShiroIdentityPlugin.java

+    public ServerSocket getSocket(Client client) throws IOException {
+        System.out.println("Client? " + client);
+        System.out.println("AdminClient? " + client.admin());
+        client.admin().cluster().prepareState().execute().actionGet();


Fails with

Exception in thread "main" org.graalvm.polyglot.PolyglotException: java.lang.ClassCastException: Invalid inline context node passed to an inlined field. A receiver of type 'ToReferenceFactory.ToUnknownNodeGen.ForeignWrapperData' was expected but is 'ToReferenceFactory.ToUnknownNodeGen'. Did you pass the wrong node to an execute method of an inlined cached node? at com.oracle.truffle.api.dsl.InlineSupport$UnsafeField.invalidReceiver(InlineSupport.java:1253) at com.oracle.truffle.api.dsl.InlineSupport$UnsafeField.nullError(InlineSupport.java:1239) at com.oracle.truffle.api.dsl.InlineSupport$UnsafeField.resolveReceiverSlow(InlineSupport.java:1203) at com.oracle.truffle.api.dsl.InlineSupport$UnsafeField.resolveReceiver(InlineSupport.java:1175) at com.oracle.truffle.api.dsl.InlineSupport$UnsafeField.getInt(InlineSupport.java:1275) at com.oracle.truffle.api.dsl.InlineSupport$StateField.get(InlineSupport.java:464) at com.oracle.truffle.api.profiles.InlinedBranchProfile.enter(InlinedBranchProfile.java:100) at com.oracle.truffle.espresso.nodes.interop.ToReference$ToUnknown.doForeignWrapper(ToReference.java:2474) at com.oracle.truffle.espresso.nodes.interop.ToReferenceFactory$ToUnknownNodeGen.executeAndSpecialize(ToReferenceFactory.java:10328) at com.oracle.truffle.espresso.nodes.interop.ToReferenceFactory$ToUnknownNodeGen.execute(ToReferenceFactory.java:10262) at com.oracle.truffle.espresso.nodes.interop.ToReference$DynamicToReference.doCached(ToReference.java:321) at com.oracle.truffle.espresso.nodes.interop.ToReferenceFactory$DynamicToReferenceNodeGen.executeAndSpecialize(ToReferenceFactory.java:175) at com.oracle.truffle.espresso.nodes.interop.ToReferenceFactory$DynamicToReferenceNodeGen.execute(ToReferenceFactory.java:141) at com.oracle.truffle.espresso.substitutions.Target_com_oracle_truffle_espresso_polyglot_Interop$InvokeMemberWithCast.doCached(Target_com_oracle_truffle_espresso_polyglot_Interop.java:2104) at com.oracle.truffle.espresso.substitutions.Target_com_oracle_truffle_espresso_polyglot_InteropFactory$InvokeMemberWithCastNodeGen.execute(Target_com_oracle_truffle_espresso_polyglot_InteropFactory.java:4966) at com.oracle.truffle.espresso.substitutions.Target_com_oracle_truffle_espresso_polyglot_Interop_InvokeMemberWithCast__LLLL.invoke(Target_com_oracle_truffle_espresso_polyglot_Interop_InvokeMemberWithCast__LLLL.java:74) at com.oracle.truffle.espresso.nodes.IntrinsicSubstitutorNode.execute(IntrinsicSubstitutorNode.java:74) at com.oracle.truffle.espresso.nodes.EspressoRootNode$Default.execute(EspressoRootNode.java:403) at <java> invokeMemberWithCast(Ljava/lang/Class;Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown) at <java> cluster()Lorg/opensearch/client/ClusterAdminClient;(Unknown)

Should be fixed in 24.2.0

github-actions · 2024-12-18T19:20:20Z

❌ Gradle check result for ed8cd55: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2024-12-19T11:04:33Z

Apart from the debugging pain, which i hope would be one-time while setting up a plugin); I don't see a reason why we would not include this as an alternative for SM. Let's see what others feel about it.

I have included it here #16861 but to reiterate, the most difficult issue with such a model is communication between host and spawned JVM/context: it seems not being possible to wire up the services from the host JVM. I will spend more time to explore the limitations here.

Asking to keep myself upto-date -- I guess you have figured it out usingjava.PolyglotInterfaceMappings?. Just that its broken in 24.1.1?

reta · 2024-12-19T13:11:36Z

Asking to keep myself upto-date -- I guess you have figured it out usingjava.PolyglotInterfaceMappings?. Just that its broken in 24.1.1?

Yes, sadly it is. And AFAIK, the earliest release with the fix is in March (as per Slack thread response), this is a blocker for us at the moment since we cannot support host <-> guest exchange :(

kumargu · 2024-12-23T07:43:34Z

(posting for visibility)
we requested GraalVM to-try-make the fix available via a patch in their upcoming Jan-25 release.

github-actions bot added enhancement Enhancement or improvement to existing feature or request Other labels Dec 16, 2024

reta force-pushed the issue-16861 branch from a8e52f3 to 6201d8c Compare December 17, 2024 12:58

kumargu reviewed Dec 17, 2024

View reviewed changes

[POC] [Security Manager Replacement] GraalVM sandboxing

ed8cd55

Signed-off-by: Andriy Redko <[email protected]>

reta force-pushed the issue-16861 branch from 6201d8c to ed8cd55 Compare December 18, 2024 19:15

reta commented Dec 18, 2024

View reviewed changes

This was referenced Dec 27, 2024

[RFC] JSM alternatives #16913

Closed

[RFC] Replace Java Security Manager (JSM) #1687

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[POC] [Security Manager Replacement] GraalVM sandboxing #16863

[POC] [Security Manager Replacement] GraalVM sandboxing #16863

reta commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 17, 2024

kumargu Dec 17, 2024

kumargu Dec 17, 2024

reta Dec 17, 2024 •

edited

Loading

reta Dec 17, 2024

kumargu Dec 17, 2024

reta Dec 17, 2024

kumargu Dec 17, 2024

kumargu commented Dec 17, 2024

reta commented Dec 17, 2024

reta Dec 18, 2024

github-actions bot commented Dec 18, 2024

kumargu commented Dec 19, 2024

reta commented Dec 19, 2024

kumargu commented Dec 23, 2024 •

edited

Loading

[POC] [Security Manager Replacement] GraalVM sandboxing #16863

Are you sure you want to change the base?

[POC] [Security Manager Replacement] GraalVM sandboxing #16863

Conversation

reta commented Dec 16, 2024

Description

Related Issues

Check List

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 17, 2024

kumargu Dec 17, 2024

Choose a reason for hiding this comment

kumargu Dec 17, 2024

Choose a reason for hiding this comment

reta Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

reta Dec 17, 2024

Choose a reason for hiding this comment

kumargu Dec 17, 2024

Choose a reason for hiding this comment

reta Dec 17, 2024

Choose a reason for hiding this comment

kumargu Dec 17, 2024

Choose a reason for hiding this comment

kumargu commented Dec 17, 2024

reta commented Dec 17, 2024

reta Dec 18, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 18, 2024

kumargu commented Dec 19, 2024

reta commented Dec 19, 2024

kumargu commented Dec 23, 2024 • edited Loading

reta Dec 17, 2024 •

edited

Loading

kumargu commented Dec 23, 2024 •

edited

Loading