Skip to content

Commit

Permalink
docs: fix docs
Browse files Browse the repository at this point in the history
  • Loading branch information
suxb201 committed Sep 10, 2024
1 parent 6405b4c commit 4944c01
Show file tree
Hide file tree
Showing 4 changed files with 118 additions and 29 deletions.
141 changes: 116 additions & 25 deletions docs/src/en/filter/function.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@
outline: deep
---

# 什么是 function
# What is function

RedisShake 通过提供 function 功能,实现了的 [ETL(提取-转换-加载)](https://en.wikipedia.org/wiki/Extract,_transform,_load) 中的 `transform` 能力。通过利用 function 可以实现类似功能:
* 更改数据所属的 `db`,比如将源端的 `db 0` 写入到目的端的 `db 1`
* 对数据进行筛选,例如,只将 key 以 `user:` 开头的源数据写入到目标端。
* 改变 Key 的前缀,例如,将源端的 key `prefix_old_key` 写入到目标端的 key `prefix_new_key`
RedisShake provides a function feature that implements the `transform` capability in [ETL (Extract-Transform-Load)](https://en.wikipedia.org/wiki/Extract,_transform,_load). By utilizing functions, you can achieve similar functionalities:
* Change the `db` to which data belongs, for example, writing data from source `db 0` to destination `db 1`.
* Filter data, for instance, only writing source data with keys starting with `user:` to the destination.
* Modify key prefixes, such as writing a source key `prefix_old_key` to a destination key `prefix_new_key`.
* ...

要使用 function 功能,只需编写一份 lua 脚本。RedisShake 在从源端获取数据后,会将数据转换为 Redis 命令。然后,它会处理这些命令,从中解析出 `KEYS``ARGV``SLOTS``GROUP` 等信息,并将这些信息传递给 lua 脚本。lua 脚本会处理这些数据,并返回处理后的命令。最后,RedisShake 会将处理后的数据写入到目标端。
To use the function feature, you only need to write a Lua script. After RedisShake retrieves data from the source, it converts the data into Redis commands. Then, it processes these commands, parsing information such as `KEYS`, `ARGV`, `SLOTS`, `GROUP`, and passes this information to the Lua script. The Lua script processes this data and returns the processed commands. Finally, RedisShake writes the processed data to the destination.

以下是一个具体的例子:
Here's a specific example:
```toml
function = """
shake.log(DB)
Expand All @@ -29,28 +29,119 @@ address = "127.0.0.1:6379"
[redis_writer]
address = "127.0.0.1:6380"
```
`DB` RedisShake 提供的信息,表示当前数据所属的 db。`shake.log` 用于打印日志,`shake.call` 用于调用 Redis 命令。上述脚本的目的是丢弃源端 `db` 0 的数据,将其他 `db` 的数据写入到目标端。
`DB` is information provided by RedisShake, indicating the db to which the current data belongs. `shake.log` is used for logging, and `shake.call` is used to call Redis commands. The purpose of the above script is to discard data from source `db 0` and write data from other `db`s to the destination.

除了 `DB`,还有其他信息如 `KEYS``ARGV``SLOTS``GROUP` 等,可供调用的函数有 `shake.log``shake.call`,具体请参考 [function API](#function-api)

关于更多的示例,可以参考 [最佳实践](./best_practices.md)
In addition to `DB`, there is other information such as `KEYS`, `ARGV`, `SLOTS`, `GROUP`, and available functions include `shake.log` and `shake.call`. For details, please refer to [function API](#function-api).

## function API

### 变量
### Variables

因为有些命令中含有多个 key,比如 `mset` 等命令。所以,`KEYS``KEY_INDEXES``SLOTS` 这三个变量都是数组类型。如果确认命令只有一个 key,可以直接使用 `KEYS[1]``KEY_INDEXES[1]``SLOTS[1]`
Because some commands contain multiple keys, such as the `mset` command, the variables `KEYS`, `KEY_INDEXES`, and `SLOTS` are all array types. If you are certain that a command has only one key, you can directly use `KEYS[1]`, `KEY_INDEXES[1]`, `SLOTS[1]`.

| 变量 | 类型 | 示例 | 描述 |
| Variable | Type | Example | Description |
|-|-|-|-----|
| DB | number | 1 | 命令所属的 `db` |
| GROUP | string | "LIST" | 命令所属的 `group`,符合 [Command key specifications](https://redis.io/docs/reference/key-specs/),可以在 [commands](https://github.com/tair-opensource/RedisShake/tree/v4/scripts/commands) 中查询每个命令的 `group` 字段 |
| CMD | string | "XGROUP-DELCONSUMER" | 命令的名称 |
| KEYS | table | \{"key1", "key2"\} | 命令的所有 Key |
| KEY_INDEXES | table | \{2, 4\} | 命令的所有 Key 在 `ARGV` 中的索引 |
| SLOTS | table | \{9189, 4998\} | 当前命令的所有 Key 所属的 [slot](https://redis.io/docs/reference/cluster-spec/#key-distribution-model) |
| ARGV | table | \{"mset", "key1", "value1", "key2", "value2"\} | 命令的所有参数 |

### 函数
* `shake.call(DB, ARGV)`:返回一个 Redis 命令,RedisShake 会将该命令写入目标端。
* `shake.log(msg)`:打印日志。
| DB | number | 1 | The `db` to which the command belongs |
| GROUP | string | "LIST" | The `group` to which the command belongs, conforming to [Command key specifications](https://redis.io/docs/reference/key-specs/). You can check the `group` field for each command in [commands](https://github.com/tair-opensource/RedisShake/tree/v4/scripts/commands) |
| CMD | string | "XGROUP-DELCONSUMER" | The name of the command |
| KEYS | table | {"key1", "key2"} | All keys of the command |
| KEY_INDEXES | table | {2, 4} | The indexes of all keys in `ARGV` |
| SLOTS | table | {9189, 4998} | The [slots](https://redis.io/docs/reference/cluster-spec/#key-distribution-model) to which all keys of the current command belong |
| ARGV | table | {"mset", "key1", "value1", "key2", "value2"} | All parameters of the command |

### Functions
* `shake.call(DB, ARGV)`: Returns a Redis command that RedisShake will write to the destination.
* `shake.log(msg)`: Prints logs.

## Best Practices


### Filtering Keys

```lua
local prefix = "user:"
local prefix_len = #prefix

if string.sub(KEYS[1], 1, prefix_len) ~= prefix then
return
end

shake.call(DB, ARGV)
```

The effect is to only write source data with keys starting with `user:` to the destination. This doesn't consider cases of multi-key commands like `mset`.

### Filtering DB

```lua
shake.log(DB)
if DB == 0
then
return
end
shake.call(DB, ARGV)
```

The effect is to discard data from source `db 0` and write data from other `db`s to the destination.


### Filtering Certain Data Structures

You can use the `GROUP` variable to determine the data structure type. Supported data structure types include: `STRING`, `LIST`, `SET`, `ZSET`, `HASH`, `SCRIPTING`, etc.

#### Filtering Hash Type Data
```lua
if GROUP == "HASH" then
return
end
shake.call(DB, ARGV)
```

The effect is to discard `hash` type data from the source and write other data to the destination.

#### Filtering [LUA Scripts](https://redis.io/docs/interact/programmability/eval-intro/)

```lua
if GROUP == "SCRIPTING" then
return
end
shake.call(DB, ARGV)
```

The effect is to discard `lua` scripts from the source and write other data to the destination. This is common when synchronizing from master-slave to cluster, where there are LUA scripts not supported by the cluster.

### Modifying Key Prefixes

```lua
local prefix_old = "prefix_old_"
local prefix_new = "prefix_new_"

shake.log("old=" .. table.concat(ARGV, " "))

for i, index in ipairs(KEY_INDEXES) do
local key = ARGV[index]
if string.sub(key, 1, #prefix_old) == prefix_old then
ARGV[index] = prefix_new .. string.sub(key, #prefix_old + 1)
end
end

shake.log("new=" .. table.concat(ARGV, " "))
shake.call(DB, ARGV)
```
The effect is to write the source key `prefix_old_key` to the destination key `prefix_new_key`.

### Swapping DBs

```lua
local db1 = 1
local db2 = 2

if DB == db1 then
DB = db2
elseif DB == db2 then
DB = db1
end
shake.call(DB, ARGV)
```

The effect is to write source `db 1` to destination `db 2`, write source `db 2` to destination `db 1`, and leave other `db`s unchanged.
2 changes: 1 addition & 1 deletion docs/src/en/guide/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Versions (configurations are not interchangeable between different versions):

- The [RedisShake 2.x](https://github.com/tair-opensource/RedisShake/tree/v2) version brought a series of improvements and updates, enhancing its overall stability and performance.
- The [RedisShake 3.x](https://github.com/tair-opensource/RedisShake/tree/v3) version was a significant milestone, with the entire codebase being completely rewritten and optimized for better efficiency and availability.
- The [RedisShake 4.x](https://github.com/tair-opensource/RedisShake/tree/v4) version further enhanced features related to the [Reader](../reader/scan_reader.md), configuration, observability, and [function](../function/introduction.md).
- The [RedisShake 4.x](https://github.com/tair-opensource/RedisShake/tree/v4) version further enhanced features related to the [Reader](../reader/scan_reader.md), configuration, observability, and `function`.

## License

Expand Down
2 changes: 0 additions & 2 deletions docs/src/zh/filter/function.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,6 @@ address = "127.0.0.1:6380"

除了 `DB`,还有其他信息如 `KEYS``ARGV``SLOTS``GROUP` 等,可供调用的函数有 `shake.log``shake.call`,具体请参考 [function API](#function-api)

关于更多的示例,可以参考 [最佳实践](./best_practices.md)

## function API

### 变量
Expand Down
2 changes: 1 addition & 1 deletion docs/src/zh/guide/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ RedisShake 是阿里云 [Tair 团队](https://github.com/tair-opensource)
- [RedisShake 2.x](https://github.com/tair-opensource/RedisShake/tree/v2) 版本带来了一系列的改进和更新,提高了其整体稳定性和性能。
- [RedisShake 3.x](https://github.com/tair-opensource/RedisShake/tree/v3) 版本是一个重要的里程碑,整个代码库被完全重写和优化,具有更好的效率和可用性。
- [RedisShake 4.x](https://github.com/tair-opensource/RedisShake/tree/v4) 版本
,进一步增强了 [Reader](../reader/scan_reader.md)、配置、可观察性和 [function](../function/introduction.md) 相关的特性。
,进一步增强了 [Reader](../reader/scan_reader.md)、配置、可观察性和 function 相关的特性。

## 许可证

Expand Down

0 comments on commit 4944c01

Please sign in to comment.