Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] wazuh-cluster.log rotation access denied #205

Closed
AlexRuiz7 opened this issue Apr 15, 2024 · 3 comments · Fixed by #212
Closed

[BUG] wazuh-cluster.log rotation access denied #205

AlexRuiz7 opened this issue Apr 15, 2024 · 3 comments · Fixed by #212
Assignees
@AlexRuiz7
Labels
level/task Task issue type/bug Bug issue

Comments

@AlexRuiz7
Copy link
Member

Description

The daily log file rotation fails due to missing runtime permissions.

ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied
Full log 

Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:181)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:216)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:203)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:398)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:308)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:300)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:560)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.core.Logger.log(Logger.java:163)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2168)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2122)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2105)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1991)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1854)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1288)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.node.Node.<init>(Node.java:428)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.node.Node.<init>(Node.java:401)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.cli.Command.main(Command.java:101)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Apr 10 09:34:58 rhel7.localdomain systemd-entrypoint[1024]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

This problem is also reproducible in OpenSearch (see opensearch-project#9609). We've put into practice the solution proposed in OpenSearch's forums about this exact error. The results turned positive (see wazuh/wazuh-packages#2139 (comment)).

wazuh/wazuh-packages#2139 (comment)
@AlexRuiz7 AlexRuiz7 added level/task Task issue type/bug Bug issue labels Apr 15, 2024
@AlexRuiz7 AlexRuiz7 self-assigned this Apr 15, 2024
@AlexRuiz7
Copy link
Member Author

AlexRuiz7 commented Apr 15, 2024 •
edited
Loading

The solution consists of adding the code below to /etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy and restart the wazuh-indexer.

grant {
  permission java.lang.RuntimePermission "accessUserInformation";
};

[root@rhel7 vagrant]# journalctl --no-pager  -xeu wazuh-indexer
-- Logs begin at Thu 2024-04-11 11:00:16 UTC, end at Thu 2024-04-11 11:01:01 UTC. --
Apr 11 11:00:25 rhel7.localdomain systemd[1]: Starting Wazuh-indexer...
-- Subject: Unit wazuh-indexer.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit wazuh-indexer.service has begun starting up.
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: A terminally deprecated method in java.lang.System has been called
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Apr 11 11:00:27 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager will be removed in a future release
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: A terminally deprecated method in java.lang.System has been called
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.10.0.jar)
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Apr 11 11:00:28 rhel7.localdomain systemd-entrypoint[1015]: WARNING: System::setSecurityManager will be removed in a future release
Apr 11 11:00:36 rhel7.localdomain systemd[1]: Started Wazuh-indexer.
-- Subject: Unit wazuh-indexer.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit wazuh-indexer.service has finished starting up.
-- 
-- The start-up result is done.

@AlexRuiz7 AlexRuiz7 mentioned this issue Apr 15, 2024
Closed
@wazuhci wazuhci moved this to On hold in Release 4.9.0 Apr 15, 2024
@AlexRuiz7
Copy link
Member Author

Update 2024.04.16

We tried this on @Tostti's environment, which runs on Ubuntu 22.04.3 LTS, and the error still persists.

The error didn't reproduce in a RHEL7 environment running on Vagrant.

@AlexRuiz7
Copy link
Member Author

Update 2024.04.24

We reviewed @Tostti's environment on April 17th, and edited the jvm.options file, removing a reference to an outdated security policy file.

The environment has been working since without errors. Evidences below.

root@tostti:/home/tostti# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-04-18 09:27:18 -03; 5 days ago
       Docs: https://documentation.wazuh.com/
   Main PID: 1018 (java)
      Tasks: 117 (limit: 18885)
     Memory: 2.1G
        CPU: 59min 39.846s
     CGroup: /system.slice/wazuh-indexer.service
             └─1018 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=tru>

abr 18 09:26:46 tostti.com systemd[1]: Starting Wazuh-indexer...
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: A terminally deprecated method in java.lang.System has been called
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.8.0.jar)
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
abr 18 09:26:53 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager will be removed in a future release
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: A terminally deprecated method in java.lang.System has been called
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.8.0.jar)
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
abr 18 09:26:56 tostti.com systemd-entrypoint[1018]: WARNING: System::setSecurityManager will be removed in a future release
abr 18 09:27:18 tostti.com systemd[1]: Started Wazuh-indexer.

root@tostti:/home/tostti# ls -l /var/log/wazuh-indexer/
total 72308
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 49308634 abr 24 08:16 gc.log
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:11 gc.log.00
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    41234 abr 10 10:13 gc.log.01
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:13 gc.log.02
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    38455 abr 10 10:16 gc.log.03
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:16 gc.log.04
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    38058 abr 10 10:20 gc.log.05
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:20 gc.log.06
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   103871 abr 10 10:36 gc.log.07
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 10 10:36 gc.log.08
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   922465 abr 10 13:26 gc.log.09
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 10 15:14 gc.log.10
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  1556178 abr 11 09:20 gc.log.11
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 11 09:24 gc.log.12
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   190560 abr 11 10:27 gc.log.13
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 11 10:31 gc.log.14
-rw-r--r-- 1 wazuh-indexer wazuh-indexer   160360 abr 11 11:15 gc.log.15
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 11 11:43 gc.log.16
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  8284785 abr 15 10:47 gc.log.17
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 15 10:47 gc.log.18
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  5423342 abr 17 12:59 gc.log.19
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2031 abr 17 12:59 gc.log.20
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  7266304 abr 18 09:22 gc.log.21
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     2007 abr 18 09:26 gc.log.22
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    75921 abr 11 00:00 wazuh-cluster-2024-04-10-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    64329 abr 11 00:00 wazuh-cluster-2024-04-10-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    36802 abr 12 00:00 wazuh-cluster-2024-04-11-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    33445 abr 12 00:00 wazuh-cluster-2024-04-11-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4423 abr 13 00:00 wazuh-cluster-2024-04-12-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     3510 abr 13 00:00 wazuh-cluster-2024-04-12-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4467 abr 14 00:00 wazuh-cluster-2024-04-13-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     3393 abr 14 00:00 wazuh-cluster-2024-04-13-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4680 abr 15 00:00 wazuh-cluster-2024-04-14-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     3473 abr 15 00:00 wazuh-cluster-2024-04-14-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    14933 abr 16 00:00 wazuh-cluster-2024-04-15-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer    13410 abr 16 00:00 wazuh-cluster-2024-04-15-1.log.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     6572 abr 17 00:00 wazuh-cluster-2024-04-16-1.json.gz
-rw-r--r-- 1 wazuh-indexer wazuh-indexer     4804 abr 17 00:00 wazuh-cluster-2024-04-16-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    14971 abr 18 00:00 wazuh-cluster-2024-04-17-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    13508 abr 18 00:00 wazuh-cluster-2024-04-17-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    16035 abr 19 00:00 wazuh-cluster-2024-04-18-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    14315 abr 19 00:00 wazuh-cluster-2024-04-18-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4366 abr 20 00:00 wazuh-cluster-2024-04-19-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3448 abr 20 00:00 wazuh-cluster-2024-04-19-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4385 abr 21 00:00 wazuh-cluster-2024-04-20-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3477 abr 21 00:00 wazuh-cluster-2024-04-20-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4196 abr 22 00:00 wazuh-cluster-2024-04-21-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3299 abr 22 00:00 wazuh-cluster-2024-04-21-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4777 abr 23 00:00 wazuh-cluster-2024-04-22-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     3753 abr 23 00:00 wazuh-cluster-2024-04-22-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     5610 abr 24 00:00 wazuh-cluster-2024-04-23-1.json.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer     4278 abr 24 00:00 wazuh-cluster-2024-04-23-1.log.gz
-rw-r----- 1 wazuh-indexer wazuh-indexer    34058 abr 24 06:42 wazuh-cluster_deprecation.json
-rw-r----- 1 wazuh-indexer wazuh-indexer    17303 abr 24 06:42 wazuh-cluster_deprecation.log
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_indexing_slowlog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_indexing_slowlog.log
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_search_slowlog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_index_search_slowlog.log
-rw-r----- 1 wazuh-indexer wazuh-indexer    20525 abr 24 08:15 wazuh-cluster.log
-rw-r----- 1 wazuh-indexer wazuh-indexer    58392 abr 24 08:15 wazuh-cluster_server.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_task_detailslog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer        0 abr 10 10:11 wazuh-cluster_task_detailslog.log

@wazuhci wazuhci moved this from On hold to In progress in Release 4.9.0 Apr 24, 2024
@AlexRuiz7 AlexRuiz7 mentioned this issue Apr 24, 2024
Merged
8 tasks
@AlexRuiz7 AlexRuiz7 linked a pull request Apr 24, 2024 that will close this issue
Fix access denied error during log rotation #212
Merged
8 tasks
@wazuhci wazuhci moved this from In progress to Pending final review in Release 4.9.0 Apr 25, 2024
@wazuhci wazuhci moved this from Pending final review to Done in Release 4.9.0 Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AlexRuiz7 AlexRuiz7

Labels
level/task Task issue type/bug Bug issue
Projects
No open projects
Release 4.9.0
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix access denied error during log rotation wazuh/wazuh-indexer
1 participant
@AlexRuiz7