[BUG] Error during startup Failed loading builtin log types from disk!

[atulsri]

Describe the bug

I am using docker image opensearchproject/opensearch:2 and the node is getting start up successfully.
However, I am getting below error in console while starting up. Could you tell the root cause of this issue and how to fix the issue? I did not configure any custom log type.

[2024-09-06T09:32:17,972][INFO ][o.o.n.Node ] [dev3-opensearch-0] initialized [2024-09-06T09:32:17,972][INFO ][o.o.n.Node ] [dev3-opensearch-0] starting ... [2024-09-06T09:32:17,979][ERROR][o.o.s.l.BuiltinLogTypeLoader] [dev3-opensearch-0] Failed loading builtin log types from disk! java.nio.file.FileSystemNotFoundException: null at [email protected]/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) ~[?:?] at [email protected]/jdk.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142) ~[?:?] at java.base/java.nio.file.Path.of(Path.java:209) ~[?:?] at java.base/java.nio.file.Paths.get(Paths.java:98) ~[?:?] at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.loadBuiltinLogTypes(BuiltinLogTypeLoader.java:73) ~[opensearch-security-analytics-2.16.0.0.jar:2.16.0.0] at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.ensureLogTypesLoaded(BuiltinLogTypeLoader.java:62) [opensearch-security-analytics-2.16.0.0.jar:2.16.0.0] at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.doStart(BuiltinLogTypeLoader.java:146) [opensearch-security-analytics-2.16.0.0.jar:2.16.0.0] at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77) [opensearch-common-2.16.0.jar:2.16.0] at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) [?:?] at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1116) [?:?] at org.opensearch.node.Node.start(Node.java:1505) [opensearch-2.16.0.jar:2.16.0] at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) [opensearch-2.16.0.jar:2.16.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) [opensearch-2.16.0.jar:2.16.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.16.0.jar:2.16.0] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.16.0.jar:2.16.0] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.16.0.jar:2.16.0] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.16.0.jar:2.16.0] at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.16.0.jar:2.16.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.16.0.jar:2.16.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.16.0.jar:2.16.0]

Related component

Plugins

To Reproduce

1. Setup the dockerFile and run the docker image
2. When opensearch node getting up, it will throw the below error
   [2024-09-06T09:32:17,979][ERROR][o.o.s.l.BuiltinLogTypeLoader] [dev3-opensearch-0] Failed loading builtin log types from disk!
   java.nio.file.FileSystemNotFoundException: null
   at [email protected]/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) ~[?:?]
   at [email protected]/jdk.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142) ~[?:?]
   at java.base/java.nio.file.Path.of(Path.java:209) ~[?:?]
   at java.base/java.nio.file.Paths.get(Paths.java:98) ~[?:?]
   at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.loadBuiltinLogTypes(BuiltinLogTypeLoader.java:73) ~[opensearch-security-analytics-2.16.0.0.jar:2.16.0.0]
   at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.ensureLogTypesLoaded(BuiltinLogTypeLoader.java:62) [opensearch-security-analytics-2.16.0.0.jar:2.16.0.0]
   at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.doStart(BuiltinLogTypeLoader.java:146) [opensearch-security-analytics-2.16.0.0.jar:2.16.0.0]
   at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77) [opensearch-common-2.16.0.jar:2.16.0]
   at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) [?:?]
   at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1116) [?:?]
   at org.opensearch.node.Node.start(Node.java:1505) [opensearch-2.16.0.jar:2.16.0]
   at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) [opensearch-2.16.0.jar:2.16.0]
   at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) [opensearch-2.16.0.jar:2.16.0]
   at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.16.0.jar:2.16.0]
   at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.16.0.jar:2.16.0]
   at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.16.0.jar:2.16.0]
   at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.16.0.jar:2.16.0]
   at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.16.0.jar:2.16.0]
   at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.16.0.jar:2.16.0]
   at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.16.0.jar:2.16.0]

Expected behavior

Exception should not come.

Additional Details

Plugins
All default plugins enabled including opensearch-security. No extra plugin configured.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [RHEL]
• Version [7.2]

Additional context
Add any other context about the problem here.

[mueller-tobias]

I've the same problem on an opensearch cluster running in kubernetes (rke2 1.28) where new nodes get those errors. It also OpenSearch 2.16.0

[reta]

@dblock could you please transfer this issue to https://github.com/opensearch-project/security-analytics ? thank you

[dblock]

[Catch All Triage - 1, 2, 3, 4]

[ymazzer]

Same issue there.

Using rke1.29 with Trident CSI and CIS hardening.

[ilyes-benyahia]

same problem on an opensearch 2.17 running in kubernetes (rke2 1.29)

[JWBrownie]

I am having the same issue with opensearch 2.17 default values no customization, only passing the necessary information as for the documentation for this to run.

[2024-10-10T16:06:00,417][ERROR][o.o.s.l.BuiltinLogTypeLoader] [opensearch-numismatica] Failed loading builtin log types from disk!
java.nio.file.FileSystemNotFoundException: null
at [email protected]/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) ~[?:?]
at [email protected]/jdk.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142) ~[?:?]
at java.base/java.nio.file.Path.of(Path.java:209) ~[?:?]
at java.base/java.nio.file.Paths.get(Paths.java:98) ~[?:?]
at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.loadBuiltinLogTypes(BuiltinLogTypeLoader.java:73) ~[opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.ensureLogTypesLoaded(BuiltinLogTypeLoader.java:62) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.doStart(BuiltinLogTypeLoader.java:146) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77) [opensearch-common-2.17.1.jar:2.17.1]
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) [?:?]
at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1116) [?:?]
at org.opensearch.node.Node.start(Node.java:1564) [opensearch-2.17.1.jar:2.17.1]
at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) [opensearch-2.17.1.jar:2.17.1]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) [opensearch-2.17.1.jar:2.17.1]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.1.jar:2.17.1]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.17.1.jar:2.17.1]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]

[eXou91]

Same problem on azure 1.29.4 , with 2.17 and 2.16 somebody has a quick config fix ?

[adevoss]

Same problem on a Debian 12 VM with Graylog installed (first time user).
hi mongodb-org-database 7.0.14
hi opensearch 2.17.1
hi graylog-server 6.0.7-1

[MaxVorobyev]

In my environment same problem was caused by empty jvm.options file.
When i placed correct jvm.options file, the problem has resolved.

[thomass4t]

@MaxVorobyev I checked /etc/opensearch/jvm.options and /etc/opensearch/jvm.options/jvm.options and both are non-empty.
It didn't help on my system. Maybe someone can verify this too.

[v-miguel]

Same problem here on GKE 1.30.5
opensearch operator version 2.6.1
tried with opensearch versions 2.17.1 and 2.16.0

any fix for this ?