RmiRegistrySslTest fails on JDK 8 during SSL lookup with handshake_failure

[jiekang]

Seen during 2024 October CPU (#5692)

RmiRegistrySslTest is failing on all platforms with logs similar to:

RmiRegistrySslTest : SSL RMIRegistry - SSL Lookup
08:45:43  -------------------------------------------------------------
08:45:43  RmiRegistry lookup...
08:45:43  KO: Got unexpected exception!
08:45:43  java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
08:45:43  	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
08:45:43  	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
08:45:43  	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
08:45:43  	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
08:45:43  	at sun.rmi.registry.RegistryImpl_Stub.list(RegistryImpl_Stub.java:90)
08:45:43  	at RmiRegistrySslTest.main(RmiRegistrySslTest.java:75)
08:45:43  Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
08:45:43  	at sun.security.ssl.Alert.createSSLException(Alert.java:131)
08:45:43  	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
08:45:43  	at sun.security.ssl.TransportContext.fatal(TransportContext.java:318)
08:45:43  	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
08:45:43  	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
08:45:43  	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
08:45:43  	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1401)
08:45:43  	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309)
08:45:43  	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
08:45:43  	at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:822)
08:45:43  	at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73)
08:45:43  	at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1184)
08:45:43  	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
08:45:43  	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
08:45:43  	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
08:45:43  	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229)
08:45:43  	... 4 more

[jiekang]

Also related: RmiSslBootstrapTest is failing on a lot of JDK 8 platforms with:

[2024-10-17T12:45:43.422Z] Connection failed for {measure, QED}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Connection failed for {modify, R&D}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Connection failed as expected for {bad.user, R&D}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Connection failed as expected for {measure, bad.password}: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
[2024-10-17T12:45:43.423Z] 	javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
[2024-10-17T12:45:43.423Z] Test Failed: Failed to connect to agent {url=service:jmx:rmi://e9a48cb84f6b/jndi/rmi://localhost:36732/jmxrmi}: java.lang.RuntimeException: Test /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_aarch64_linux_rerun/aqa-tests/TKG/output_17291679023488/jdk_management_0/work/classes/2/sun/management/jmxremote/bootstrap/management_ssltest06_ok.properties failed with 2 error(s)
[2024-10-17T12:45:43.423Z] STDERR:
[2024-10-17T12:45:43.423Z] /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_aarch64_linux_rerun/aqa-tests/openjdk/openjdk-jdk/jdk/test/sun/management/jmxremote/bootstrap/RmiSslBootstrapTest.sh: 37: /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_aarch64_linux_rerun/aqa-tests/openjdk/openjdk-jdk/jdk/test/sun/management/jmxremote/bootstrap/GeneratePropertyPassword.sh: [[: not found

[jiekang]

Related test code has seen one recent change in August 2024, the backport of JDK-8145919
openjdk/jdk8u@09bab6e

From history:
https://github.com/openjdk/jdk8u/commits/master/jdk/test/sun/management/jmxremote/bootstrap

[jiekang]

The JDK-8145919 backport mentioned required followup PR for JDK-8205653, however this was not merged in time for this release:

openjdk/jdk8u-dev#548

[jiekang]

I strongly suspect JDK-8205653 is required for these tests to pass properly.

Based on above analysis, I would not consider these test failures as release blockers.

[jiekang]

I suppose the most correct thing to do is run a build with openjdk/jdk8u-dev#548 incorporated to see if the test passes.