[Doc] Added function introduction of auditloader 4.2.1 (#52612)

Signed-off-by: 吴梦龙 <[email protected]> Co-authored-by: 絵空事スピリット <[email protected]> (cherry picked from commit 1b05fcb)
StarRocks · Nov 26, 2024 · 4f38fde · 4f38fde
1 parent 24e90b8
commit 4f38fde
Show file tree

Hide file tree

Showing 2 changed files with 53 additions and 10 deletions.
diff --git a/docs/en/administration/management/audit_loader.md b/docs/en/administration/management/audit_loader.md
@@ -89,11 +89,14 @@ After a partition is created, you can move on to the next step.
 
 3. Modify **plugin.conf** to configure AuditLoader. You must configure the following items to make sure AuditLoader can work properly:
 
-    - `frontend_host_port`: FE IP address and HTTP port, in the format `<fe_ip>:<fe_http_port>`. The default value is `127.0.0.1:8030`.
+    - `frontend_host_port`: FE IP address and HTTP port, in the format `<fe_ip>:<fe_http_port>`. It is recommended to set it to its default value `127.0.0.1:8030`. Each FE in StarRocks manages its own Audit Log independently, and after installing the plugin, each FE will start its own background thread to fetch and save Audit Logs, and write them via Stream Load. The `frontend_host_port` configuration item is used to provide the IP and port of the HTTP protocol for the background Stream Load task of the plug-in, and this parameter does not support multiple values. The IP part of the parameter can use the IP of any FE in the cluster, but it is not recommended because if the corresponding FE crashes, the audit log writing task in the background of other FEs will also fail due to the failure of communication. It is recommended to set it to the default value `127.0.0.1:8030`, so that each FE uses its own HTTP port to communicate, thus avoiding the impact on the communication in case of an exception of the other FEs (all the write tasks will be forwarded to the FE Leader node to be executed eventually).
     - `database`: name of the database you created to host audit logs.
     - `table`: name of the table you created to host audit logs.
     - `user`: your cluster username. You MUST have the privilege to load data (LOAD_PRIV) into the table.
     - `password`: your user password.
+    - `secret_key`: the key (string, must not be longer than 16 bytes) used to encrypt the password. If this parameter is not set, it indicates that the password in **plugin.conf** will not be encrypted, and you only need to specify the plaintext password in `password`. If this parameter is specified, it indicates that the password is encrypted by this key, and you need to specify the encrypted string in `password`. The encrypted password can be generated in StarRocks using the `AES_ENCRYPT` function: `SELECT TO_BASE64(AES_ENCRYPT('password','secret_key'));`.
+    - `enable_compute_all_query_digest`: whether to generate Hash SQL fingerprint for all queries (StarRocks only enable SQL fingerprint for slow queries by default). Note that the fingerprint calculation in the plugin is different from that of FE, which will [normalize the SQL statement](../Query_planning.md#sql-fingerprint), while the plugin does not. The fingerprint calculation will consume additional computing resources if this feature is enabled.
+    - `filter`: the filter conditions for audit log loading. This parameter is based on the [WHERE parameter](../../sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#opt_properties)  in Stream Load, i.e. `-H “where: <condition>”`, defaults to an empty string. Example: `filter=isQuery=1 and clientIp like '127.0.0.1%' and user='root'`.
 
 4. Zip the files back into a package.
 
@@ -103,6 +106,10 @@ After a partition is created, you can move on to the next step.
 
 5. Dispatch the package to all machines that host FE nodes. Make sure all packages are stored in an identical path. Otherwise, the installation fails. Remember to copy the absolute path to the package after you dispatched the package.
 
+  > **NOTE**
+  >
+  > You can also distribute **auditloader.zip** to an HTTP service accessible to all FEs (for example, `httpd` or `nginx`) and install it using the network. Note that in both cases the **auditloader.zip** needs to be persisted in the path after the installation is performed, and the source files should not be deleted after installation.
+
 ## Install AuditLoader
 
 Execute the following statement along with the path you copied to install AuditLoader as a plugin in StarRocks:
@@ -111,6 +118,20 @@ Execute the following statement along with the path you copied to install AuditL
 INSTALL PLUGIN FROM "<absolute_path_to_package>";
 ```
 
+Example of installation from a local package:
+
+```SQL
+INSTALL PLUGIN FROM "<absolute_path_to_package>";
+```
+
+If you want install the plugin via a network path, you need to provide the md5 of the package in the properties of the INSTALL statement.
+
+Example:
+
+```sql
+INSTALL PLUGIN FROM "http://xx.xx.xxx.xxx/extra/auditloader.zip" PROPERTIES("md5sum" = "3975F7B880C9490FE95F42E2B2A28E2D");
+```
+
 See [INSTALL PLUGIN](../../sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md) for detailed instructions.
 
 ## Verify the installation and query audit logs
@@ -135,8 +156,8 @@ See [INSTALL PLUGIN](../../sql-reference/sql-statements/cluster-management/plugi
     *************************** 2. row ***************************
         Name: AuditLoader
         Type: AUDIT
-    Description: Available for versions 2.3+. Load audit log to starrocks, and user can view the statistic of queries.
-        Version: 4.0.0
+    Description: Available for versions 2.5+. Load audit log to starrocks, and user can view the statistic of queries
+        Version: 4.2.1
     JavaVersion: 1.8.0
     ClassName: com.starrocks.plugin.audit.AuditLoaderPlugin
         SoName: NULL
@@ -194,4 +215,4 @@ If no audit logs are loaded to the table after the dynamic partition is created
 UNINSTALL PLUGIN AuditLoader;
 ```
 
-After all configurations are set correctly, you can follow the above steps to install AuditLoader again.
+Logs of AuditLoader are printed in **fe.log** of each FE, you can retrieve them by searching the keyword `audit` in **fe.log**. After all configurations are set correctly, you can follow the above steps to install AuditLoader again.
diff --git a/docs/zh/administration/management/audit_loader.md b/docs/zh/administration/management/audit_loader.md
@@ -44,7 +44,10 @@ CREATE TABLE starrocks_audit_db__.starrocks_audit_tbl__ (
   `stmt`              VARCHAR(1048576)           COMMENT "原始SQL语句",
   `digest`            VARCHAR(32)                COMMENT "慢SQL指纹",
   `planCpuCosts`      DOUBLE                     COMMENT "查询规划阶段CPU占用（纳秒）",
-  `planMemCosts`      DOUBLE                     COMMENT "查询规划阶段内存占用（字节）"
+  `planMemCosts`      DOUBLE                     COMMENT "查询规划阶段内存占用（字节）",
+  `pendingTimeMs`     BIGINT                     COMMENT "查询在队列中等待的时间（毫秒）",
+  `candidateMVs`      VARCHAR(65533)             COMMENT "候选MV列表",
+  `hitMvs`            VARCHAR(65533)             COMMENT "命中MV列表"
 ) ENGINE = OLAP
 DUPLICATE KEY (`queryId`, `timestamp`, `queryType`)
 COMMENT "审计日志表"
@@ -87,11 +90,14 @@ SHOW PARTITIONS FROM starrocks_audit_db__.starrocks_audit_tbl__;
 
 3. 修改 **plugin.conf** 文件以配置 AuditLoader。您必须配置以下项目以确保 AuditLoader 可以正常工作：
 
-    - `frontend_host_port`：FE 节点 IP 地址和 HTTP 端口，格式为 `<fe_ip>:<fe_http_port>`。 默认值为 `127.0.0.1:8030`。
+    - `frontend_host_port`：FE 节点 IP 地址和 HTTP 端口，格式为 `<fe_ip>:<fe_http_port>`。推荐使用默认值，即 `127.0.0.1:8030` 。StarRocks 中各个 FE 是独立管理各自的审计信息。在安装插件后，各个 FE 分别会启动各自的后台线程进行审计信息的获取和攒批，并通过 Stream Load 写入。 `frontend_host_port` 配置项用于为插件后台 Stream Load 任务提供 HTTP 协议的 IP 和端口，该参数不支持配置为多个值。其中，参数的 IP 部分可以使用集群内任意某个 FE 的 IP，但并不推荐这样配置，因为若对应的 FE 出现异常，其他 FE 后台的审计信息写入任务也会因无法通信导致写入失败。推荐配置为默认的 `127.0.0.1:8030`，让各个 FE 均使用自身的 HTTP 端口进行通信，以此规避其他 FE 异常时对通信的影响（所有的写入任务最终都会被 FE 自动转发到 FE Leader 节点执行）。
     - `database`：审计日志库名。
     - `table`：审计日志表名。
     - `user`：集群用户名。该用户必须具有对应表的 INSERT 权限。
     - `password`：集群用户密码。
+    - `secret_key`：用于加密密码的 Key（字符串，长度不得超过 16 个字节）。如果该参数为空，则表示不对 **plugin.conf** 中的密码进行加解密，您只需在 `password` 处直接配置明文密码。如果该参数不为空，表示需要通过该 Key 对密码进行加解密，您需要在 `password` 处配置加密后的字符串。加密后的密码可在 StarRocks 中通过 `AES_ENCRYPT` 函数生成：`SELECT TO_BASE64(AES_ENCRYPT('password','secret_key'));`。
+    - `enable_compute_all_query_digest`：是否对所有查询都生成 Hash SQL 指纹（StarRocks 默认只为慢查询开启 SQL 指纹）。需注意插件中的指纹计算方法与 FE 内部的方法不一致，FE 会对 SQL 语句[规范化处理](../Query_planning.md#%E6%9F%A5%E7%9C%8B-sql-%E6%8C%87%E7%BA%B9)，而插件不会，且如果开启该参数，指纹计算会额外占用集群内的计算资源。
+    - `filter`：审计信息入库的过滤条件。该参数基于 Stream Load 中的 [WHERE 参数](../../sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#opt_properties) 实现，即 `-H "where: <condition>"`，默认值为空字符串。示例：`filter=isQuery=1 and clientIp like '127.0.0.1%' and user='root'`。
 
 4. 重新打包以上文件。
 
@@ -101,6 +107,10 @@ SHOW PARTITIONS FROM starrocks_audit_db__.starrocks_audit_tbl__;
 
 5. 将压缩包分发至所有 FE 节点运行的机器。请确保所有压缩包都存储在相同的路径下，否则插件将安装失败。分发完成后，请复制压缩包的绝对路径。
 
+  > **注意**
+  >
+  > 您也可将 **auditloader.zip** 分发至所有 FE 都可访问到的 HTTP 服务中（例如 `httpd` 或 `nginx`），然后使用网络路径安装。注意这两种方式下 **auditloader.zip** 在执行安装后都需要在该路径下持续保留，不可在安装后删除源文件。
+
 ## 安装 AuditLoader
 
 通过以下语句安装 AuditLoader 插件：
@@ -109,6 +119,18 @@ SHOW PARTITIONS FROM starrocks_audit_db__.starrocks_audit_tbl__;
 INSTALL PLUGIN FROM "<absolute_path_to_package>";
 ```
 
+以安装本地插件包为例，安装命令示例：
+
+```SQL
+mysql> INSTALL PLUGIN FROM "/opt/module/starrocks/auditloader.zip";
+```
+
+若通过网络路径安装，还需在安装命令的属性中提供插件压缩包的 md5 信息，命令示例：
+
+```sql
+INSTALL PLUGIN FROM "http://xx.xx.xxx.xxx/extra/auditloader.zip" PROPERTIES("md5sum" = "3975F7B880C9490FE95F42E2B2A28E2D");
+```
+
 详细操作说明参阅 [INSTALL PLUGIN](../../sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md)。
 
 ## 验证安装并查询审计日志
@@ -133,8 +155,8 @@ INSTALL PLUGIN FROM "<absolute_path_to_package>";
     *************************** 2. row ***************************
         Name: AuditLoader
         Type: AUDIT
-    Description: Available for versions 2.3+. Load audit log to starrocks, and user can view the statistic of queries.
-        Version: 4.0.0
+    Description: Available for versions 2.5+. Load audit log to starrocks, and user can view the statistic of queries
+        Version: 4.2.1
     JavaVersion: 1.8.0
     ClassName: com.starrocks.plugin.audit.AuditLoaderPlugin
         SoName: NULL
@@ -144,7 +166,7 @@ INSTALL PLUGIN FROM "<absolute_path_to_package>";
     2 rows in set (0.01 sec)
     ```
 
-2. 随机执行 SQL 语句以生成审计日志，并等待60秒（或您在配置 AuditLoader 时在 `max_batch_interval_sec` 项中指定的时间）以允许 AuditLoader 将审计日志攒批导入至StarRocks 中。
+2. 随机执行 SQL 语句以生成审计信息，并等待 60 秒（或您在配置 AuditLoader 时在 `max_batch_interval_sec` 项中指定的时间）以允许 AuditLoader 将审计日志攒批导入至StarRocks 中。
 
 3. 查询审计日志表。
 
@@ -192,4 +214,4 @@ INSTALL PLUGIN FROM "<absolute_path_to_package>";
 UNINSTALL PLUGIN AuditLoader;
 ```
 
-所有配置设置正确后，您可以按照上述步骤重新安装。
+AuditLoader 的日志会打印在各个 FE 的 **fe.log** 中。您可以在 **fe.log** 中检索关键字 `audit`，用排查 Stream Load 任务的思路来定位问题。所有配置设置正确后，您可以按照上述步骤重新安装。