Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] [Enhancement] Enhance validation for create connector API #3287

Open
wants to merge 1 commit into
base: 2.x
Choose a base branch
from

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport 58903ba from #3260

This PR addresses the first part of this enhancement "Validate if connector payload has all the required fields. If not provided, throw the illegal argument exception".
Validation of fields description, parameters, credential, and request_body are missing. That validations are added in this fix.
Added new test cases correspong to these validations and fixed all failing test cases because of these new validations.

Partially Resolves #1382

Signed-off-by: Abdul Muneer Kolarkunnu <[email protected]>
(cherry picked from commit 58903ba)
@mingshl
Copy link
Collaborator

mingshl commented Dec 17, 2024

2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:mlCommonsBwcCluster#twoThirdsUpgradedClusterTask' --tests "org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility" -Dtests.seed=CE4567D78D37D26B -Dtests.security.manager=false -Dtests.locale=ru -Dtests.timezone=SystemV/EST5EDT -Druntime.java=11
  2> org.junit.ComparisonFailure: expected:<[m_l_limit_exceeded]_exception> but was:<[null_pointer]_exception>
        at __randomizedtesting.SeedInfo.seed([CE4567D78D37D26B:257A5A19F7EC07E1]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility(MLCommonsBackwardsCompatibilityIT.java:219)
  2> NOTE: leaving temporary files on disk at: /home/runner/work/ml-commons/ml-commons/plugin/build/testrun/mlCommonsBwcCluster#twoThirdsUpgradedClusterTask/temp/org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT_CE4567D78D37D26B-001
  2> NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=938, maxMBSortInHeap=7.0070272290712925, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ru, timezone=SystemV/EST5EDT
  2> NOTE: Linux 6.8.0-1017-azure amd64/Azul Systems, Inc. 11.0.25 (64-bit)/cpus=4,threads=1,free=391632384,total=536870912
  2> NOTE: All tests run in this JVM: [MLCommonsBackwardsCompatibilityIT]
  1> [2024-12-17T16:48:00,074][INFO ][o.o.m.b.MLCommonsBackwardsCompatibilityIT] [testBackwardsCompatibility] after test

Tests with failures:

1 test completed, 1 failed

=== Standard output of node `node{:opensearch-ml-plugin:mlCommonsBwcCluster0-0}` ===
 - org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility

»    ↓ errors and warnings from /home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/mlCommonsBwcCluster0-0/logs/opensearch.stdout.log ↓
» WARN ][o.o.g.DanglingIndicesState] [mlCommonsBwcCluster0-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
»   ↑ repeated 2 times ↑
» WARN ][o.o.d.FileBasedSeedHostsProvider] [mlCommonsBwcCluster0-0] expected, but did not find, a dynamic hosts list at [/home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/mlCommonsBwcCluster0-0/config/unicast_hosts.txt]
»   ↑ repeated 12 times ↑
» WARN ][o.o.c.c.ClusterFormationFailureHelper] [mlCommonsBwcCluster0-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [mlCommonsBwcCluster0-0, mlCommonsBwcCluster0-1, mlCommonsBwcCluster0-2] to bootstrap a cluster: have discovered [{mlCommonsBwcCluster0-0}{hcpajMarT5q_iYrY0aIT8A}{wCE8oEluQBqPJyeyzuCUbw}{127.0.0.1}{127.0.0.1:43979}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{mlCommonsBwcCluster0-0}{hcpajMarT5q_iYrY0aIT8A}{wCE8oEluQBqPJyeyzuCUbw}{127.0.0.1}{127.0.0.1:43979}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
» ERROR][o.o.m.c.MLCommonsClassLoader] [mlCommonsBwcCluster0-0] Failed to init instance for type PREDICTION
»  java.lang.reflect.InvocationTargetException: null
»  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
»  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
»  	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
»  	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
»  	at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:242) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:206) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.output.MLOutput.fromStream(MLOutput.java:36) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.transport.MLTaskResponse.<init>(MLTaskResponse.java:39) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:85) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:52) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1508) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1495) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:405) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:2[80](https://github.com/opensearch-project/ml-commons/actions/runs/12381938399/job/34561548837?pr=3287#step:5:81)) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at java.base/java.lang.Thread.run(Thread.java:[82](https://github.com/opensearch-project/ml-commons/actions/runs/12381938399/job/34561548837?pr=3287#step:5:83)9) [?:?]
»  Caused by: java.io.EOFException
»  	at org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:74) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:586) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.ml.common.output.MLPredictionOutput.<init>(MLPredictionOutput.java:72) ~[?:?]
»  	... 42 more

@mingshl
Copy link
Collaborator

mingshl commented Dec 17, 2024

Step 'removeUnusedImports' found problem in 'src\test\java\org\opensearch\ml\common\connector\ConnectorActionTest.java':
> Task :opensearch-ml-common:spotlessJava FAILED
103:36: error: unclosed string literal
com.google.googlejavaformat.java.FormatterException: 103:36: error: unclosed string literal

Hi @akolarkunnu in ConnectorActionTest.java, there is unused imported in 2.x branch, can you clone this branch and apply the new commit to fix it? backport/backport-3260-to-2.x

@akolarkunnu
Copy link
Contributor

Step 'removeUnusedImports' found problem in 'src\test\java\org\opensearch\ml\common\connector\ConnectorActionTest.java':
> Task :opensearch-ml-common:spotlessJava FAILED
103:36: error: unclosed string literal
com.google.googlejavaformat.java.FormatterException: 103:36: error: unclosed string literal

Hi @akolarkunnu in ConnectorActionTest.java, there is unused imported in 2.x branch, can you clone this branch and apply the new commit to fix it? backport/backport-3260-to-2.x

ok, I will check.

@akolarkunnu
Copy link
Contributor

akolarkunnu commented Dec 18, 2024

Step 'removeUnusedImports' found problem in 'src\test\java\org\opensearch\ml\common\connector\ConnectorActionTest.java':
> Task :opensearch-ml-common:spotlessJava FAILED
103:36: error: unclosed string literal
com.google.googlejavaformat.java.FormatterException: 103:36: error: unclosed string literal

Hi @akolarkunnu in ConnectorActionTest.java, there is unused imported in 2.x branch, can you clone this branch and apply the new commit to fix it? backport/backport-3260-to-2.x

ok, I will check.

@mingshl There is no any unused imports. But it is complaining about String Blocks(line number 103). String Blocks are introduced in JDK15. And this project(ml-commons 2.x branch) is based on JDK17. So it should not complain. Or Am I missing anything?

@mingshl
Copy link
Collaborator

mingshl commented Dec 23, 2024

@akolarkunnu try checking the Spotless configuration in the build file to ensure it's set up correctly for Java 17. And check if the project's build configuration in build.gradle is correctly set to use Java 17.


XContentBuilder builder = XContentBuilder.builder(XContentType.JSON.xContent());
action.toXContent(builder, ToXContent.EMPTY_PARAMS);
String content = TestHelper.xContentBuilderToString(builder);
Assert.assertEquals("{\"action_type\":\"PREDICT\",\"method\":\"http\",\"url\":\"https://test.com\"}", content);
String expctedContent = """
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't support in Java 17. In main branch, we use JAVA 21 which support this string. So for you I would suggest to do:

  1. Fix main branch code with not using """
  2. And then backport both of the commits together in 2.x branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants