[FEATURE] Communicate implemented extension interfaces to OpenSearch

[dbwiddis]

Is your feature request related to a problem?

Extensions may implement one or more interfaces beyond Extension. For example, script engine and context extension points will be implemented on the ScriptExtension. The ExtensionsManager list of discovered extensions needs to be able to filter that list to determine a subset of extensions which implement specific functionality.

What solution would you like?

Create an Enum as an inner class of the ExtensionsManager with all of the extension interfaces (see #315 for a list of plugins that need corresponding extensions).

Communicate a list of enum values to OpenSearch during initialization and save it in the appropriate object (probably a List<EnumName> on the DiscoveryExtensionNode object).

Do you have any additional context?

Review how various plugin interfaces are filtered in Node.java. For example, ActionPlugin are sent to the ActionModule to register rest handlers. ScriptPlugin are sent to the ScriptModule etc.

[varuntumbe]

Hi @dbwiddis, I would like to implement this. I have some work for next couple of days so I will be touching the code on this weekend if it's okay.

Thanks and Regards
Varun

[varuntumbe]

Hi @dbwiddis,

Can I assign myself this issue?

Thanks and Regards
Varun

[dbwiddis]

Hey @varuntumbe sorry for the delay replying here. Go ahead and work on this if you like.

Note that at this point none of the other extension interfaces have been written (see #318, #320, #321, #322, #323, #324, #325, #326, #327, #328, #329, #330, #331, #332, and #333). You probably want to do at least one of those to incorporate into the code logic, and then whoever handles those issues can build on the work you did.

This will require a PR both here and on the main OpenSearch project.

[varuntumbe]

Okay, I will assign myself one of those first. Is there any communication channel like slack or discord incase if I want to clear up some doubts?

Thanks and regards
Varun

[dbwiddis]

Just ask here on Github if you have any questions! Feel free to tag my username to get my attention faster.

[varuntumbe]

Sure! I have cloned the repo and trying to get it running in local.

Thanks and regards
Varun

[varuntumbe]

Hi @dbwiddis , getting this error while starting the opensearch-sdk-java locally. I am trying to run ./gradlew helloWorld

Task :compileJava
Failed to get resource: HEAD. [HTTP HTTP/1.1 403 Forbidden: https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.14.1/jackson-datatype-jsr310-2.14.1.pom)]

Task :helloWorld
Failed to get resource: HEAD. [HTTP HTTP/1.1 403 Forbidden: https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/org/apache/lucene/lucene-core/9.4.2/lucene-core-9.4.2.pom)]

[varuntumbe]

I am quite new to Java, as per my understanding it is trying to get some artifacts as it is declared in the repositories section of build.grade

repositories {
    mavenLocal()
    maven { url "https://aws.oss.sonatype.org/content/repositories/snapshots" }
    maven { url "https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/" }
    mavenCentral()
}

[dbwiddis]

Hey @varuntumbe I can build fine but see access denied trying to go to that repository. As a troubleshooting step, try to comment out that lucene snapshot URL in bulid.gradle. I think the appropriate snapshots are published at the aws snapshot url.

@saratvemulapalli any idea here?

[varuntumbe]

Hi @dbwiddis ,

It did not work. It tries to look for lucene snapshot in .m2/repositories and in other two locations, but it could find those. Let me troubleshoot some more and will get back to you.

Execution failed for task ':compileJava'.
Could not resolve all files for configuration ':compileClasspath'.
Could not find org.apache.lucene:lucene-core:9.5.0-snapshot-0878271.
Searched in the following locations:
- file:/C:/Users/varun/.m2/repository/org/apache/lucene/lucene-core/9.5.0-snapshot-0878271/lucene-core-9.5.0-snapshot-0878271.pom
- https://aws.oss.sonatype.org/content/repositories/snapshots/org/apache/lucene/lucene-core/9.5.0-snapshot-0878271/lucene-core-9.5.0-snapshot-0878271.pom
- https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-core/9.5.0-snapshot-0878271/lucene-core-9.5.0-snapshot-0878271.pom

[saratvemulapalli]

Hey @varuntumbe I can build fine but see access denied trying to go to that repository. As a troubleshooting step, try to comment out that lucene snapshot URL in bulid.gradle. I think the appropriate snapshots are published at the aws snapshot url.

@saratvemulapalli any idea here?

Usually these artifacts (lucene snapshots) are open to the world. Not sure why @varuntumbe is seeing problems.
@mch2 are these snapshots not open to public?

[mch2]

These snapshots are not published to aws.oss or repo.maven given they are not opensearch artifacts. They exist here right now - https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/. Example:
https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/org/apache/lucene/lucene-core/9.5.0-snapshot-0878271/lucene-core-9.5.0-snapshot-0878271.pom exists & is public. If it is an option I would suggest depending on OpenSearch's buildSrc/src/main/java/org/opensearch/gradle/RepositoriesSetupPlugin.java to configure repositories to fetch these snapshots.

[mch2]

@varuntumbe @dbwiddis I believe you will want to list mavenCentral before the snapshot repo like this:

repositories {
    mavenLocal()
    mavenCentral()
    maven { url "https://aws.oss.sonatype.org/content/repositories/snapshots" }
    maven { url "https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/" }
}

So that we aren't searching aws.oss & the lucene snapshot repo for released dependencies.

[dbwiddis]

I was just coming to that conclusion myself!

[dbwiddis]

@varuntumbe looks like changing the repo order per this comment will fix things. I'll update our main branch here, you can update your local copy and hopefully build.

[varuntumbe]

Yes. I was able to get it up and running. Thanks @dbwiddis @saratvemulapalli @mc2 I will assign myself #331 and later will work on this

[varuntumbe]

Hi @dbwiddis,

So far what I have understood is we need to define one enum datatype consisting of all the existing extension Interface inside ExtensionManager ( which is 1st part, which is an easy part :) ) and then during instantiation of Node class it initializes ExtensionManager based on some setting Flag and it has List as one of the data member which has information on all discovered extensions. OpenSearch node does the tcp handshake with the each extension which is running on separate node and exchange some messages ( custom settings etc ). In this communication, we need to inform opensearch Node about the interfaces being implemented by an extension ( which is the 2nd part, little tricky ). Did I understand the issue correctly ? or am I missing something ?

Thanks and Regards
Varun

[dbwiddis]

Yep. I think you have it right!

[varuntumbe]

Hi,

I facing issues with compiling opensearch node locally. I was trying to follow steps from 'To run OpenSearch from a compiled binary:' and I gave this command ./gradlew assemble . This is the error I am getting.

Task :server:compileJava
Build cache key for task ':server:compileJava' is 8511ee27ab378ac8c2824573dd386fe7
Task ':server:compileJava' is not up-to-date because:
Task has failed previously.
The input changes require a full rebuild for incremental task ':server:compileJava'.
Full recompilation is required because no incremental change information is available. This is usually caused by clean builds or changing compiler arguments.
Compiling with toolchain 'C:\Program Files\Java\jdk-17'.
Starting process 'Gradle Worker Daemon 2'. Working directory: C:\Users\varun.gradle\workers Command: C:\Program Files\Java\jdk-17\bin\java.exe --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED @C:\Users\varun.gradle.tmp\gradle-worker-classpath1213878160051673014txt -Xmx3g -Dfile.encoding=windows-1252 -Duser.country=IN -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 2'
Successfully started process 'Gradle Worker Daemon 2'
Started Gradle worker daemon (0.376 secs) with fork options DaemonForkOptions{executable=C:\Program Files\Java\jdk-17\bin\java.exe, minHeapSize=null, maxHeapSize=3g, jvmArgs=[], keepAliveMode=SESSION}.
Compiling with JDK Java compiler API.
F:\Open_Source_Projects\OpenSearch\server\src\main\java\org\opensearch\index\engine\EngineConfigFactory.java:77: error: cannot find symbol
codecService = enginePlugin.getCustomCodecService(idxSettings);
^
symbol: method getCustomCodecService(IndexSettings)
location: variable enginePlugin of type EnginePlugin
F:\Open_Source_Projects\OpenSearch\server\src\main\java\org\opensearch\index\engine\EngineConfigFactory.java:79: error: cannot find symbol
} else if (enginePlugin.getCustomCodecService(idxSettings).isPresent()) {
^
symbol: method getCustomCodecService(IndexSettings)
location: variable enginePlugin of type EnginePlugin
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
2 errors

Task :server:compileJava FAILED

FAILURE: Build failed with an exception.

I have synced my fork to the latest and pulled latest to local as well. I dont see this getCustomCodecService method in EnginePlugin class.

What can I do regarding this ? I need to start opensearch locally so that I can debug the application to understand the transport communications with opensearch SDK

Thanks and Regards
Varun

[dbwiddis]

A few ideas:

1. make sure you have enabled the feature flag. In the gradle/run.gradle file add the following under runTask:

systemProperty 'opensearch.experimental.feature.extensions.enabled', 'true'

2. try with ./gradlew run rather than assembling.

[varuntumbe]

Hi,

I have done the 1st step and it gives the same error for run as well

[dbwiddis]

So that getCustomCodecService() method is one of the default extension points on EnginePlugin (it's the deprecated one we didn't carry over to EngineExtension in #393). See here.

Is the EnginePlugin class still in your fork or did you delete/rename it to the extension one?

[varuntumbe]

I think I had deleted here as well by mistake, I did the fresh clone now and its coming up, but one last question, where do we have to place extensions.yml file ? I have placed it under extensions folder under project root dir. its not finding the yml file while loading

[2023-02-03T00:01:27,043][INFO ][o.o.e.ExtensionsManager ] [runTask-0] ExtensionsManager initialized
[2023-02-03T00:01:27,043][INFO ][o.o.e.ExtensionsManager ] [runTask-0] Loading extensions : F:\Open_Source_Projects\OpenSearch\build\testclusters\runTask-0\distro\3.0.0-ARCHIVE\extensions
[2023-02-03T00:01:27,044][WARN ][o.o.e.ExtensionsManager ] [runTask-0] Extensions.yml file is not present. No extensions will be loaded.

[dbwiddis]

When you start up OpenSearch it will have a log entry telling you the extensions config dir. Yours is in the log you posted:

F:\Open_Source_Projects\OpenSearch\build\testclusters\runTask-0\distro\3.0.0-ARCHIVE\extensions

I actually have copies of the extensions.yml and run.gradle files in my git root and a bash file in my home directory that I execute to start up OpenSearch and re-copy the files over (they sometimes get erased during a recompile) so change those paths to yours above:

#!bash

mkdir -pv /Users/widdisd/git/OpenSearch/distribution/archives/darwin-tar/build/install/opensearch-3.0.0-SNAPSHOT/extensions
cp -v ~/git/extensions.yml /Users/widdisd/git/OpenSearch/distribution/archives/darwin-tar/build/install/opensearch-3.0.0-SNAPSHOT/extensions
cp -v ~/git/run.gradle ~/git/Opensearch/gradle
./gradlew run

[varuntumbe]

This is helpful, yes I had copied to that location it was getting erased. I will make a batch file or equivalent to this.

Thank you.

[varuntumbe]

Hi @dbwiddis ,

I have added Request and RequestHandler classes in Open Source Extension package, those classes not getting imported locally for open search SDK ( I am still trying to understand how it works ), i will get this by this week

Thanks and regards
Varun

[dbwiddis]

Note the extensions have their own version of ExtensionRestRequest and ExtensionRequestHandler. Not sure which request and handler you're dealing with but if you can be a bit more specific I can offer more specific advice.

[varuntumbe]

Yes, I have created a separate classes for Request and Request handler under extension package for registering interfaces. But after defining everything on the open search side, i need to register the request in SDK side in extensionRunner.java, here I am not able to reference ExtensionsManager.<My_New_Defined_Req> or anything that i have defined in the open search side.

I thought ./gradlew publishMavenToLocal on open search ( which adds all the artifacts and jars to .m2/repositories/.. ) and running gradlew build --refresh-dependencies on SDK would fix it. But it did not work. Did I miss anything here ?

Thanks and regards
Varun

[dbwiddis]

Yes, making the change on your OpenSearch branch and using publishToMavenLocal on that branch should work (if it built successfully) to make those classes available on the SDK side. Can you double check visibility of the classes you're adding?

[varuntumbe]

Yes I published to MavenLocal and in sdk I reloaded the workspace and its taking changes. I tested it as well now, its working. I will do a commit today or by tomorrow

Thanks and Regards
Varun

[dbwiddis]

Done in #508