[BUG] Unable to install lang-painless module when testDistribution is set to INTEG_TEST for k-NN #1676

jmazanec15 · 2021-12-01T21:09:03Z

What is the bug?
For k-NN, I am trying to run integ tests on Mac. To do this, I first switched testDistribution from ARCHIVE to INTEG_TEST. Otherwise, it cannot find the OpenSearch artifact:

Execution failed for task ':run'.
> Could not resolve all files for configuration ':opensearch_distro_extracted_testclusters--integTest-0-1.3.0-SNAPSHOT-'.
   > Could not resolve opensearch-distribution-snapshot:opensearch:1.3.0-SNAPSHOT.
     Required by:
         project :
      > Could not resolve opensearch-distribution-snapshot:opensearch:1.3.0-SNAPSHOT.
         > Could not get resource 'https://artifacts.opensearch.org/snapshots/core/opensearch/1.3.0-SNAPSHOT/opensearch-min-1.3.0-SNAPSHOT-darwin-x64-latest.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/snapshots/core/opensearch/1.3.0-SNAPSHOT/opensearch-min-1.3.0-SNAPSHOT-darwin-x64-latest.tar.gz'. Received status code 403 from server: Forbidden

After making the above switch, it is able to download, but fails with the following error (k-NN has a dependency on lang-painless):

| Exception in thread "main" java.lang.IllegalArgumentException: Missing plugin [lang-painless], dependency of [opensearch-knn]
|       at org.opensearch.plugins.PluginsService.addSortedBundle(PluginsService.java:512)
|       at org.opensearch.plugins.PluginsService.sortBundles(PluginsService.java:480)
|       at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:839)
|       at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:813)
|       at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:858)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:263)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
|       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.Command.main(Command.java:101)
|       at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

This is because the modules do not come with the INTEG_TEST distribution.

So, I create the lang painless module from source and placed it in test resources. Then, I add it as a plugin. However, this then fails with this error:

| ERROR: plugin 'lang-painless' cannot be installed as a plugin, it is a system module

So, I switch this line from plugin to module. However, this then produces the missing plugin error again:

| Exception in thread "main" java.lang.IllegalArgumentException: Missing plugin [lang-painless], dependency of [opensearch-knn]
|       at org.opensearch.plugins.PluginsService.addSortedBundle(PluginsService.java:512)
|       at org.opensearch.plugins.PluginsService.sortBundles(PluginsService.java:480)
|       at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:839)
|       at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:813)
|       at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:858)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:263)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
|       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.Command.main(Command.java:101)
|       at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

I was wondering if there is any way to prevent gradle from attempting to install an extended plugin as a plugin when it is a module.

How can one reproduce the bug?
Steps to reproduce the behavior:

Clone the repo

git clone https://github.com/jmazanec15/k-NN-1.git
cd k-NN-1 
git checkout build-on-mac

Install dependencies

brew install openblas

Run build

./gradlew build

What is the expected behavior?
Expect to be able to add lang-painless as a module.

What is your host/environment?

OS: Mac Big Sur
Version 1.3
Plugins k-NN, lang-painless

The text was updated successfully, but these errors were encountered:

dblock · 2021-12-07T15:40:55Z

Should we move this issue into OpenSearch repo? That’s where we’d be fixing it, right?

saratvemulapalli · 2021-12-08T01:20:18Z

@jmazanec15 Modules are only installed for INTEG_TEST test distribution.
Ref: https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java#L725
Manually installing a module will also fail because its not a plugin.

The rootcause of the problem is 1.3.0-SNAPSHOT opensearch min artifact is not available for darwin.
https://artifacts.opensearch.org/snapshots/core/opensearch/1.3.0-SNAPSHOT/opensearch-min-1.3.0-SNAPSHOT-darwin-x64-latest.tar.gz
Here is the issue: opensearch-project/opensearch-build#38 which is tracking for 1.3 release.

So unless that is resolved running integration tests on darwin would not work.
The alternative solution is to run integration tests while spinning up an external cluster: opensearch-project/opensearch-plugins#103

That said, for the last part of the question:
I was wondering if there is any way to prevent gradle from attempting to install an extended plugin as a plugin when it is a module.
The ARCHIVE test distribution was not intended was these reasons. You should continue to use INTEG_TEST distribution to have modules installed by default.

saratvemulapalli · 2021-12-08T16:11:55Z

Closing this. Feel free to reopen if the comment does not help.

jmazanec15 · 2022-03-17T17:18:35Z

@saratvemulapalli Not sure I have the ability to reopen, but wanted to continue discussion.

I tried switching to INTEG_TEST, but this does not come with modules other than transport-netty.

I was thinking about installing the "lang-painless" in the following way:

Adding a dependency on it in my build.gradle.

configurations {
    painless
}

dependencies {
    ...
    painless "org.opensearch.plugin:lang-painless:${opensearch_version}"
}

Zipping up the files pulled from the dependency as a gradle task

task prepareLangPainlessZip(type: Zip) {
    from configurations.painless.allArtifacts.files
    from configurations.painless
    destinationDirectory = layout.buildDirectory.dir("jack")
    archiveFileName = "lang-painless.zip"
}

Adding the zip as a module in the cluster setup:

testClusters.integTest {
    testDistribution = "INTEG_TEST"
    module(provider(new Callable<RegularFile>(){
        @Override
        RegularFile call() throws Exception {
            return new RegularFile() {
                @Override
                File getAsFile() {
                    return fileTree("build/jack").getSingleFile()
                }
            }
        }
    }))
    ...
}

However, I get the following error:

| Exception in thread "main" java.lang.IllegalStateException: Could not load plugin descriptor for module directory [lang-painless]
|       at org.opensearch.plugins.PluginsService.readPluginBundle(PluginsService.java:452)
|       at org.opensearch.plugins.PluginsService.findBundles(PluginsService.java:437)
|       at org.opensearch.plugins.PluginsService.getModuleBundles(PluginsService.java:425)
|       at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:849)
|       at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:825)
|       at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:870)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
|       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.Command.main(Command.java:101)
|       at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
| Caused by: java.nio.file.NoSuchFileException: .../k-NN-1/build/testclusters/integTest-0/distro/2.0.0-INTEG_TEST/modules/lang-painless/plugin-descriptor.properties
|       at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
|       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
|       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
|       at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
|       at java.base/java.nio.file.Files.newByteChannel(Files.java:375)
|       at java.base/java.nio.file.Files.newByteChannel(Files.java:426)
|       at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
|       at java.base/java.nio.file.Files.newInputStream(Files.java:160)
|       at org.opensearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:199)
|       at org.opensearch.plugins.PluginsService.readPluginBundle(PluginsService.java:449)

I have a couple questions:

Why plugins in https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/ do not contain the plugin descriptors?
Why is module call trying to use plugin cli?
Is there a better way to go about doing this?

Related issue: opensearch-project/opensearch-build#1750 (comment)

saratvemulapalli · 2022-03-17T17:28:29Z

I can re-open the issue.

jmazanec15 · 2022-05-18T17:14:39Z

@ryanbogan @saratvemulapalli @dblock @prudhvigodithi Wanted to follow up on this. Right now, we (k-NN) cannot run integration tests on Mac because we are not able to get the lang-painless zip artifact for the INTEG TEST distribution. I believe that Alerting has the same issue.

With opensearch-project/opensearch-build#716 and opensearch.pluginzip, will we be able to add a dependency to pull core plugin artifacts from Maven so that we can install painless on the testcluster?

dblock · 2022-05-18T21:02:53Z

@prudhvigodithi should be able to answer ^

prudhvigodithi · 2022-05-18T21:20:42Z

Hey @jmazanec15 opensearch.pluginzip, this will publish the generated distribution plugin zip's to maven repo, is lang-painless zip part of any plugin bundle task? If so yes this opensearch.pluginzip can be used to publish to maven repo with right maven coordinates.

jmazanec15 · 2022-06-07T18:42:10Z

@prudhvigodithi I am not sure if its a part of any plugin bundle task. It is a core module. Is there a plan to publish all core plugin/module zips so that they can be consumed?

jmazanec15 · 2022-06-16T22:51:01Z

It seems like there are several other plugins that might have this issue (i.e. cant bring up cluster on mac):

I am wondering if, given that modules are meant to be shipped with OpenSearch, it makes sense to have a testDistribution that includes modules and works on mac.

Im not sure if this means adding support for it through the ARCHIVE distribution, or adding another distribution similar to INTEG_TEST.

What do you think? @dblock @saratvemulapalli @prudhvigodithi @ryanbogan .

amitgalitz · 2022-06-16T22:55:18Z

It seems like there are several other plugins that might have this issue (i.e. cant bring up cluster on mac):

https://github.com/opensearch-project/anomaly-detection/blob/main/build.gradle#L616

https://github.com/opensearch-project/alerting/blob/main/alerting/build.gradle#L82

https://github.com/opensearch-project/asynchronous-search/blob/main/build.gradle#L117

I am wondering if, given that modules are meant to be shipped with OpenSearch, it makes sense to have a testDistribution that includes modules and works on mac.

Im not sure if this means adding support for it through the ARCHIVE distribution, or adding another distribution similar to INTEG_TEST.

What do you think? @dblock @saratvemulapalli @prudhvigodithi @ryanbogan .

+1 on this issue, Anomaly Detection is experiencing the exact same bug.

saratvemulapalli · 2022-06-17T16:23:06Z

@amitgalitz @jmazanec15 I see it as a workaround to help install default modules for a test distribution(INTEG_TEST/ new distribution) to run tests on MacOS.
The ARCHIVE test distribution already is bundled with all default modules (including lang-painless) but its not available on MacOS yet[1].
cc: @bbarani

My thoughts, I would rather have it supported the right way rather than putting in a work around.
That said, if this is impacting your productivity we can look at alternative options.

[1] opensearch-project/opensearch-build#38

jmazanec15 · 2022-06-17T16:47:10Z

That said, if this is impacting your productivity we can look at alternative options.

In general, it creates a lot of extra effort for someone to rampup on the project. They have to figure out a system that will sync mac dev environment with linux. Also, makes running debuggers more difficult. Overall, makes the bar to contribute higher.

opensearch-project/opensearch-build#38 seems to be focused on supporting the bundle as a whole on Mac. I am wondering if as a first step, we could just enable it for OpenSearch and its modules to unblock development. Specifically, focus on making ARCHIVE work on Mac as opposed to waiting for Mac bundle to be ready and supported.

saratvemulapalli · 2022-06-17T17:11:18Z

In general, it creates a lot of extra effort for someone to rampup on the project. They have to figure out a system that will sync mac dev environment with linux. Also, makes running debuggers more difficult. Overall, makes the bar to contribute higher.

Absolutely, I feel the pain as well :).

opensearch-project/opensearch-build#38 seems to be focused on supporting the bundle as a whole on Mac. I am wondering if as a first step, we could just enable it for OpenSearch and its modules to unblock development. Specifically, focus on making ARCHIVE work on Mac as opposed to waiting for Mac bundle to be ready and supported.

Yeah that would be a nice intermediate step which unblocks this, I love this idea.
Which is btw already available from OpenSearch, its just that opensearch-build has to publish the artifact.
Is this something your team could consider ? @bbarani @peterzhuamazon

bbarani · 2022-06-17T21:29:22Z

@saratvemulapalli Can you please clarify the artifact that you are referring in the above comment? Are you referring to opensearch-min artifact?

saratvemulapalli · 2022-06-17T21:32:49Z

@saratvemulapalli Can you please clarify the artifact that you are referring in the above comment? Are you referring to opensearch-min artifact?

Correct, opensearch-min artifact for MacOS (darwin).

dblock · 2022-06-17T23:44:22Z

Does a cross-platform build for MacOS work on MacOS? If so we can probably add to the existing Jenkinsfile a "mac" build that's good enough?

prudhvigodithi · 2022-06-18T15:10:13Z

Does a cross-platform build for MacOS work on MacOS? If so we can probably add to the existing Jenkinsfile a "mac" build that's good enough?

@dblock, or we can just run the distribution build on macOS ec2 jenkins agent, then the logic PLATFORM uname -s will be inferred as Darwin
https://aws.amazon.com/ec2/instance-types/mac/
@bbarani @peterzhuamazon

bbarani · 2022-06-20T17:57:08Z

We are currently exploring this option. We will update this thread soon with our findings.

CC: @prudhvigodithi @saratvemulapalli @jmazanec15 @dblock

prudhvigodithi · 2022-06-20T22:48:17Z

Hey @jmazanec15, can you please test with opensearch-min-2.0.1-darwin-x64.tar.gz
https://ci.opensearch.org/ci/dbc/Playground/distribution-build-opensearch-pgodithi/2.0.1/0/darwin/x64/builds/opensearch/dist/opensearch-min-2.0.1-darwin-x64.tar.gz
NOTE: This is not a final darwin published artifact, this is to just test the OpenSearch min darwin format.

jmazanec15 · 2022-06-20T23:16:59Z

Yes it works!

./bin/opensearch-plugin install file:/Users/jmazane/opensearch-knn-2.0.1.0.zip
warning: ignoring JAVA_TOOL_OPTIONS=-Dlog4j2.formatMsgNoLookups=true
...
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission loadLibrary.opensearchknn_faiss
* java.lang.RuntimePermission loadLibrary.opensearchknn_nmslib
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed opensearch-knn with folder name opensearch-knn

./bin/opensearch
...
[2022-06-20T16:15:10,817][INFO ][o.o.p.PluginsService     ]  loaded plugin [opensearch-knn]
...

curl localhost:9200/_cat/plugins
xxxxx opensearch-knn 2.0.1.0

Just to be sure, this will get published to something like

'https://artifacts.opensearch.org/snapshots/core/opensearch/2.0.1-SNAPSHOT/opensearch-min-2.0.1-SNAPSHOT-darwin-x64-latest.tar.gz'

correct?

navneet1v · 2022-06-21T02:40:08Z

Similar issue but happening for all the OS versions with mavenLocalPublish. Can someone please take a look at this too?

prudhvigodithi · 2022-06-21T11:21:50Z

Just to be sure, this will get published to something like
'https://artifacts.opensearch.org/snapshots/core/opensearch/2.0.1-SNAPSHOT/opensearch-min-2.0.1-SNAPSHOT-darwin-x6

Hey @jmazanec15 will post an update regarding the darwin final download URL.
Thank you

saratvemulapalli · 2022-08-02T05:57:26Z

Closing this. Thanks for all the work @prudhvigodithi.

navneet1v · 2022-08-02T06:20:49Z

@prudhvigodithi Thanks for the work. This was really a blocker for the development on mac.

saratvemulapalli transferred this issue from opensearch-project/opensearch-plugins Dec 8, 2021

saratvemulapalli closed this as completed Dec 8, 2021

dbbaughe mentioned this issue Dec 8, 2021

OpenSearch / OpenSearch Dashboards - macOS distribution (X64) opensearch-project/opensearch-build#38

Open

21 tasks

saratvemulapalli reopened this Mar 17, 2022

saratvemulapalli added the untriaged label Mar 17, 2022

jmazanec15 mentioned this issue Mar 18, 2022

Publish plugin distribution ZIPs to Maven, get rid of checked in snapshot builds opensearch-project/opensearch-build#716

Closed

7 tasks

saratvemulapalli self-assigned this Mar 22, 2022

Rishikesh1159 assigned mch2 and unassigned saratvemulapalli Mar 29, 2022

mch2 assigned ryanbogan and unassigned mch2 Apr 5, 2022

mch2 added untriaged and removed untriaged labels Apr 5, 2022

CEHENKLE removed the untriaged label Apr 5, 2022

saratvemulapalli added the Build Libraries & Interfaces label May 19, 2022

prudhvigodithi mentioned this issue Jun 20, 2022

Build test Darwin version of min OpenSearch distribution to support plugin testing on MacOS. opensearch-project/opensearch-build#2216

Closed

7 tasks

saratvemulapalli unassigned ryanbogan Jun 20, 2022

prudhvigodithi mentioned this issue Jul 19, 2022

Test the existing build_workflow to run the OpenSearch build with platform as darwin. opensearch-project/opensearch-build#2350

Closed

saratvemulapalli closed this as completed Aug 2, 2022

prudhvigodithi self-assigned this Oct 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Unable to install lang-painless module when testDistribution is set to INTEG_TEST for k-NN #1676

[BUG] Unable to install lang-painless module when testDistribution is set to INTEG_TEST for k-NN #1676

jmazanec15 commented Dec 1, 2021

dblock commented Dec 7, 2021

saratvemulapalli commented Dec 8, 2021

saratvemulapalli commented Dec 8, 2021

jmazanec15 commented Mar 17, 2022

saratvemulapalli commented Mar 17, 2022

jmazanec15 commented May 18, 2022

dblock commented May 18, 2022

prudhvigodithi commented May 18, 2022

jmazanec15 commented Jun 7, 2022

jmazanec15 commented Jun 16, 2022

amitgalitz commented Jun 16, 2022

saratvemulapalli commented Jun 17, 2022

jmazanec15 commented Jun 17, 2022

saratvemulapalli commented Jun 17, 2022

bbarani commented Jun 17, 2022 •

edited

Loading

saratvemulapalli commented Jun 17, 2022

dblock commented Jun 17, 2022

prudhvigodithi commented Jun 18, 2022 •

edited

Loading

bbarani commented Jun 20, 2022 •

edited

Loading

prudhvigodithi commented Jun 20, 2022

jmazanec15 commented Jun 20, 2022 •

edited

Loading

navneet1v commented Jun 21, 2022

prudhvigodithi commented Jun 21, 2022 •

edited

Loading

saratvemulapalli commented Aug 2, 2022

navneet1v commented Aug 2, 2022

[BUG] Unable to install lang-painless module when testDistribution is set to INTEG_TEST for k-NN #1676

[BUG] Unable to install lang-painless module when testDistribution is set to INTEG_TEST for k-NN #1676

Comments

jmazanec15 commented Dec 1, 2021

dblock commented Dec 7, 2021

saratvemulapalli commented Dec 8, 2021

saratvemulapalli commented Dec 8, 2021

jmazanec15 commented Mar 17, 2022

saratvemulapalli commented Mar 17, 2022

jmazanec15 commented May 18, 2022

dblock commented May 18, 2022

prudhvigodithi commented May 18, 2022

jmazanec15 commented Jun 7, 2022

jmazanec15 commented Jun 16, 2022

amitgalitz commented Jun 16, 2022

saratvemulapalli commented Jun 17, 2022

jmazanec15 commented Jun 17, 2022

saratvemulapalli commented Jun 17, 2022

bbarani commented Jun 17, 2022 • edited Loading

saratvemulapalli commented Jun 17, 2022

dblock commented Jun 17, 2022

prudhvigodithi commented Jun 18, 2022 • edited Loading

bbarani commented Jun 20, 2022 • edited Loading

prudhvigodithi commented Jun 20, 2022

jmazanec15 commented Jun 20, 2022 • edited Loading

navneet1v commented Jun 21, 2022

prudhvigodithi commented Jun 21, 2022 • edited Loading

saratvemulapalli commented Aug 2, 2022

navneet1v commented Aug 2, 2022

bbarani commented Jun 17, 2022 •

edited

Loading

prudhvigodithi commented Jun 18, 2022 •

edited

Loading

bbarani commented Jun 20, 2022 •

edited

Loading

jmazanec15 commented Jun 20, 2022 •

edited

Loading

prudhvigodithi commented Jun 21, 2022 •

edited

Loading