[BUG] Node bootup failing with "Unable to load plugin class"

[vikasvb90]

Describe the bug

Node bootstrap is failing on main branch (latest commit id cd55bca3ee32d5754018e994b1b87799c1bffc32) as it is unable to load a module class. Here's the complete trace.

[2024-05-11T20:04:55,441][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [master1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: OpenSearchException[Unable to load plugin class [org.opensearch.percolator.PercolatorModulePlugin]]; nested: ClassNotFoundException[org.opensearch.percolator.PercolatorModulePlugin];
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
Caused by: org.opensearch.OpenSearchException: Unable to load plugin class [org.opensearch.percolator.PercolatorModulePlugin]
	at org.opensearch.plugins.PluginsService.loadPluginClass(PluginsService.java:772) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.node.Node.<init>(Node.java:494) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.node.Node.<init>(Node.java:421) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	... 6 more
Caused by: java.lang.ClassNotFoundException: org.opensearch.percolator.PercolatorModulePlugin
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593) ~[?:?]
	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
	at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
	at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
	at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPluginClass(PluginsService.java:770) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.node.Node.<init>(Node.java:494) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.node.Node.<init>(Node.java:421) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	... 6 more
uncaught exception in thread [main]
OpenSearchException[Unable to load plugin class [org.opensearch.percolator.PercolatorModulePlugin]]; nested: ClassNotFoundException[org.opensearch.percolator.PercolatorModulePlugin];
Likely root cause: java.lang.ClassNotFoundException: org.opensearch.percolator.PercolatorModulePlugin
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:534)
	at java.base/java.lang.Class.forName(Class.java:513)
	at org.opensearch.plugins.PluginsService.loadPluginClass(PluginsService.java:770)
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731)
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544)
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196)
	at org.opensearch.node.Node.<init>(Node.java:494)
	at org.opensearch.node.Node.<init>(Node.java:421)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)

Related component

Plugins

To Reproduce

1. Checkout main branch. I ran it on commit cd55bca3ee32d5754018e994b1b87799c1bffc32 .
2. Clean up any local build cache ./gradlew clean
3. Execute ./gradlew localDistro
4. Start a node with ./build/distribution/local/opensearch-3.0.0-SNAPSHOT/bin/opensearch -E cluster.name=vikasvb-cluster -E path.data=<Put data path here> -E node.name=master1 -E node.roles=data,cluster_manager,ingest -E cluster.initial_cluster_manager_nodes=master1

Executed on Mac M1 pro - 64 bit

Expected behavior

Node bootstrap should succeed with all modules loaded successfully.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Apple M1 Pro
• Version 13.6.6 (22G630)

Additional context
Add any other context about the problem here.

[vikasvb90]

Problem seems to be platform specific as it is working on Linux environment.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@vikasvb90 Thanks for creating this issue

[vikasvb90]

Added mac OS type and version.
cc: @reta @andrross

[andrross]

I don't know the cause here, but the service starts up properly if I run the repro steps on my Intel-based Mac running macOS 14.4.1.