diff --git a/README.md b/README.md index 1bb1dd25..9bd34400 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,105 @@ -# redis-shake +# RedisShake: Redis Data Processing & Migration Tool -[![CI](https://RedisShake/actions/workflows/ci.yml/badge.svg?branch=v3)](https://RedisShake/actions/workflows/ci.yml) +[![CI](https://github.com/tair-opensource/RedisShake/actions/workflows/ci.yml/badge.svg?event=push?branch=v4)](https://github.com/tair-opensource/RedisShake/actions/workflows/ci.yml) +[![CI](https://github.com/tair-opensource/RedisShake/actions/workflows/pages.yml/badge.svg?branch=v4)](https://github.com/tair-opensource/RedisShake/actions/workflows/pages.yml) +[![CI](https://github.com/tair-opensource/RedisShake/actions/workflows/release.yml/badge.svg?branch=v4)](https://github.com/tair-opensource/RedisShake/actions/workflows/release.yml) - [中文文档](https://tair-opensource.github.io/RedisShake/) -- [English Document](https://tair-opensource.github.io/RedisShake/en/) +- [English Documentation](https://tair-opensource.github.io/RedisShake/en/) -redis-shake is a tool for Redis data migration and data filtering. -* 🚄 High performance -* ✅ Tested on Redis 2.8, Redis 3.0, Redis 4.0, Redis 5.0, Redis 6.0 and Redis 7.0 -* 🤗 Support custom filtering rules -* 💪 Support large instance migration -* 💖 Support `restore` mode, `sync` mode and `scan` mode -* ☁️ Support Aliyun Redis and ElastiCache +## Overview -![redis-shake2.PNG](https://s2.loli.net/2022/07/10/OZrSGutknlI8XNp.png) +RedisShake is a tool designed for processing and migrating Redis data. It offers the following features: -![image.png](https://s2.loli.net/2022/06/30/vU346lVBrNofKzu.png) +1. **Redis Compatibility**: RedisShake is compatible with Redis versions ranging from 2.8 to 7.2, and supports various + deployment methods including standalone, master-slave, sentinel, and cluster. + +2. **Cloud Service Compatibility**: RedisShake works seamlessly with popular Redis-like databases provided by leading + cloud service providers, including but not limited to: + - [Alibaba Cloud - ApsaraDB for Redis](https://www.alibabacloud.com/product/apsaradb-for-redis) + - [Alibaba Cloud - Tair](https://www.alibabacloud.com/product/tair) + - [AWS - ElastiCache](https://aws.amazon.com/elasticache/) + - [AWS - MemoryDB](https://aws.amazon.com/memorydb/) + +3. **Module Compatibility**: RedisShake is compatible + with [TairString](https://github.com/tair-opensource/TairString), [TairZSet](https://github.com/tair-opensource/TairZset), + and [TairHash](https://github.com/tair-opensource/TairHash) modules. + +4. **Multiple Export Modes**: RedisShake supports PSync, RDB, and Scan export modes. + +5. **Data Processing**: RedisShake enables data filtering and transformation through custom scripts. + +## Getting Started + +### Installation + +#### Download the Binary Package + +Download the binary package directly from the [Releases](https://github.com/tair-opensource/RedisShake/releases) page. + +#### Compile from Source + +To compile from source, ensure that you have a Golang environment set up on your local machine: + +```shell +git clone https://github.com/alibaba/RedisShake +cd RedisShake +sh build.sh +``` + +### Usage + +Assume you have two Redis instances: + +* Instance A: 127.0.0.1:6379 +* Instance B: 127.0.0.1:6380 + +Create a new configuration file `shake.toml`: + +```toml +[sync_reader] +address = "127.0.0.1:6379" + +[redis_writer] +address = "127.0.0.1:6380" +``` + +To start RedisShake, run the following command: + +```shell +./redis-shake shake.toml +``` + +For more detailed information, please refer to the documentation: + +- [中文文档](https://tair-opensource.github.io/RedisShake/) +- [English Documentation](https://tair-opensource.github.io/RedisShake/en/) + +## Contributing + +We welcome contributions from the community. For significant changes, please open an issue first to discuss what you +would like to change. We are particularly interested in: + +1. Adding support for more modules +2. Enhancing support for Readers and Writers +3. Sharing your Lua scripts and best practices + +## History + +RedisShake is a project actively maintained by the [Tair team](https://github.com/tair-opensource) at Alibaba Cloud. Its +evolution can be traced back to its initial version, which was forked +from [redis-port](https://github.com/CodisLabs/redis-port). + +During its evolution: + +- The [RedisShake 2.x](https://github.com/tair-opensource/RedisShake/tree/v2) version brought a series of improvements + and updates, enhancing its overall stability and performance. +- The [RedisShake 3.x](https://github.com/tair-opensource/RedisShake/tree/v3) version represented a significant + milestone where the entire codebase was completely rewritten and optimized, leading to better efficiency and + usability. +- The current version, [RedisShake 4.x](https://github.com/tair-opensource/RedisShake/tree/v4), has further enhanced + features related to readers, configuration, observability, and functions. + +## License + +RedisShake is open-sourced under the [MIT license](https://github.com/tair-opensource/RedisShake/blob/v2/license.txt). diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 5111ab36..858d3b78 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -2,6 +2,7 @@ import { defineConfig } from 'vitepress' // https://vitepress.dev/reference/site-config export default defineConfig({ + head: [['link', { rel: 'icon', href: '/RedisShake/favicon.ico' }]], base: "/RedisShake/", title: "RedisShake", description: "RedisShake is a tool for processing and migrating Redis data.", @@ -21,48 +22,43 @@ export default defineConfig({ { text: '介绍', items: [ - { text: '什么是 RedisShake', link: '/zh/guide/getting-started' }, + { text: '什么是 RedisShake', link: '/zh/guide/introduction' }, { text: '快速上手', link: '/zh/guide/getting-started' }, - { text: '配置', link: '/zh/guide/getting-started' }, - { text: '迁移模式选择', link: '/zh/guide/getting-started' }, + { text: '配置', link: '/zh/guide/config' }, + { text: '迁移模式选择', link: '/zh/guide/mode' }, ] }, { text: 'Reader', items: [ - { text: 'Sync Reader', link: '/zh/function/introduction' }, - { text: 'Scan Reader', link: '/zh/function/best_practices' }, - { text: 'RDB Reader', link: '/zh/function/best_practices' }, + { text: 'Sync Reader', link: '/zh/reader/sync_reader' }, + { text: 'Scan Reader', link: '/zh/reader/scan_reader' }, + { text: 'RDB Reader', link: '/zh/reader/rdb_reader' }, ] }, { text: 'Writer', items: [ - { text: 'Redis Writer', link: '/zh/function/introduction' }, + { text: 'Redis Writer', link: '/zh/writer/redis_writer' }, ] }, { text: 'function', items: [ { text: '什么是 function', link: '/zh/function/introduction' }, - { - text: '最佳实践', - items: [ - { text: '监控', link: '/zh/function/best_practices' }, - ] - } + { text: '最佳实践', link: '/zh/function/best_practices' } ] }, - { - text: '进阶用法', - items: [ - { text: '监控', link: '/zh/function/best_practices' }, - { text: '双向同步', link: '/zh/function/best_practices' }, - { text: '容器部署', link: '/zh/function/best_practices' }, - { text: '主从实例向集群实例迁移', link: '/zh/function/best_practices' }, - { text: '大 key 重写', link: '/zh/function/best_practices' }, - ] - } + // { + // text: '进阶用法', + // items: [ + // { text: '监控', link: '/zh/function/best_practices' }, + // { text: '双向同步', link: '/zh/function/best_practices' }, + // { text: '容器部署', link: '/zh/function/best_practices' }, + // { text: '主从实例向集群实例迁移', link: '/zh/function/best_practices' }, + // { text: '大 key 重写', link: '/zh/function/best_practices' }, + // ] + // } ], footer: { message: 'Released under the MIT License.', diff --git a/docs/src/index.md b/docs/src/index.md index af9ea6d5..07072da1 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -10,13 +10,15 @@ hero: - theme: brand text: 快速上手 link: /zh/guide/getting-started - + - theme: alt + text: 什么是 RedisShake + link: /zh/guide/introduction features: - title: 数据迁移 details: 支持 sync、scan 和 restore 三种数据迁移模式 - - title: 数据处理 + - title: 数据加工 details: 支持使用 lua 脚本对数据进行过滤与修改 - - title: 云数据库支持 - details: 兼容主流云厂商的多种架构:主从、集群等 + - title: 兼容 + details: 兼容多种 Redis 部署形态,兼容主流云厂商的 Redis-like 数据库 --- diff --git a/docs/src/public/favicon.ico b/docs/src/public/favicon.ico new file mode 100644 index 00000000..8b342de0 Binary files /dev/null and b/docs/src/public/favicon.ico differ diff --git a/docs/src/zh/function/introduction.md b/docs/src/zh/function/introduction.md index 77ac1650..168fcc16 100644 --- a/docs/src/zh/function/introduction.md +++ b/docs/src/zh/function/introduction.md @@ -1,65 +1,8 @@ --- outline: deep --- -# transform 样例 -## 阿里云兼容 +# 什么是 function -```lua --- Aliyun Redis 4.0: skip OPINFO command -function transform(id, is_base, group, cmd_name, keys, slots, db_id, timestamp_ms) - if cmd_name == "OPINFO" then - return 1, db_id -- disallow - else - return 0, db_id -- allow - end -end -``` -## AWS 兼容 - -```lua --- ElastiCache: skip REPLCONF command -function transform(id, is_base, group, cmd_name, keys, slots, db_id, timestamp_ms) - if cmd_name == "REPLCONF" then - return 1, db_id -- disallow - else - return 0, db_id -- allow - end -end -``` - -## 过滤命令 - -### 过滤所有 lua 脚本 - -``` --- skip all scripts included LUA scripts and Redis Functions. -function filter(id, is_base, group, cmd_name, keys, slots, db_id, timestamp_ms) - if group == "SCRIPTING" then - return 1, db_id -- disallow - else - return 0, db_id -- allow - end -end -``` - -## key 操作 - -### 按照前缀过滤 key - -``` --- skip keys prefixed with ABC -function filter(id, is_base, group, cmd_name, keys, slots, db_id, timestamp_ms) - if #keys ~= 1 then - return 0, db_id -- allow - end - - if string.sub(keys[1], 0, 3) == "ABC" then - return 1, db_id -- disallow - end - - return 0, db_id -- allow -end -``` diff --git a/docs/src/zh/guide/config.md b/docs/src/zh/guide/config.md index 9f19a257..b39b41d6 100644 --- a/docs/src/zh/guide/config.md +++ b/docs/src/zh/guide/config.md @@ -1,6 +1,7 @@ --- outline: deep --- + # 配置文件 RedisShake 使用 [TOML](https://toml.io/cn/) 语言书写,所有的配置参数在 all.toml 中均有说明。 @@ -8,7 +9,7 @@ RedisShake 使用 [TOML](https://toml.io/cn/) 语言书写,所有的配置参 配置文件的组成如下: ```toml -transform = "..." +function = "..." [xxx_reader] ... @@ -20,99 +21,25 @@ transform = "..." ... ``` -一般用法下,只需要书写 `xxx_reader`、`xxx_writer` 两个部分即可,`transform` 和 `advanced` 部分为进阶用法,用户可以根据自己的需求进行配置。 - -## reader 配置 - -根据源端的类型,RedisShake 提供了不同的 Reader 配置,用来对接不同的源端。 - -* 对于支持 [Redis Sync/Psync 协议](https://redis.io/docs/management/replication/)的源端,推荐使用 `sync_xxx_reader` -* 对于不支持 [Redis Sync/Psync 协议](https://redis.io/docs/management/replication/)的源端,可以使用 `scan_xxx_reader` -* 对于使用 dump.rdb 文件恢复数据场景,可以使用 `rdb_reader` +一般用法下,只需要书写 `xxx_reader`、`xxx_writer` 两个部分即可,`function` 和 `advanced` 部分为进阶用法,用户可以根据自己的需求进行配置。 -### sync_xxx_reader +## function 配置 -对于源端为单机 Redis-like 数据库时,使用 `sync_standalone_reader`;对于源端为 Redis Cluster 时,使用 `sync_cluster_reader`。 - -#### sync_standlone_reader - -```toml -[sync_standlone_reader] -address = "127.0.0.1:6379" -username = "" # keep empty if not using ACL -password = "" # keep empty if no authentication is required -tls = false -``` +参考 [什么是 function](../function/introduction.md)。 -#### sync_cluster_reader - -```toml -[sync_cluster_reader] -address = "127.0.0.1:6379" -username = "" # keep empty if not using ACL -password = "" # keep empty if no authentication is required -tls = false -``` - -### scan_xxx_reader - -对于源端为单机 Redis-like 数据库时,使用 `scan_standalone_reader`;对于源端为 Redis Cluster 时,使用 `scan_cluster_reader`。 - -#### scan_standlone_reader - -```toml -[scan_standlone_reader] -address = "127.0.0.1:6379" -username = "" # keep empty if not using ACL -password = "" # keep empty if no authentication is required -tls = false -``` - -#### scan_cluster_reader - -```toml -[scan_cluster_reader] -address = "127.0.0.1:6379" -username = "" # keep empty if not using ACL -password = "" # keep empty if no authentication is required -tls = false -``` - -### rdb_reader +## reader 配置 -```toml -[rdb_reader] -filepath = "/path/to/dump.rdb" -``` +RedisShake 提供了不同的 Reader 用来对接不同的源端,配置详见 Reader 章节: -filepath 为 dump.rdb 文件的路径,最好使用绝对路径。 +* [Sync Reader](../reader/sync_reader.md) +* [Scan Reader](../reader/scan_reader.md) +* [RDB Reader](../reader/rdb_reader.md) ## writer 配置 -根据目标端的类型,RedisShake 提供了不同的 Writer 配置,用来对接不同的目标端。 -目前 RedisShake 支持的目标端有: -* 单机 Redis-like 数据库:redis_standalone_writer -* Redis Cluster:redis_cluster_writer - -### redis_standalone_writer - -```toml -[redis_standalone_writer] -address = "127.0.0.1:6380" -username = "" # keep empty if not using ACL -password = "" # keep empty if no authentication is required -tls = false -``` - -### redis_cluster_writer +RedisShake 提供了不同的 Writer 用来对接不同的目标端,配置详见 Writer 章节: -```toml -[redis_cluster_writer] -address = "127.0.0.1:6380" -username = "" # keep empty if not using ACL -password = "" # keep empty if no authentication is required -tls = false -``` +* [Redis Writer](../writer/redis_writer.md) ## advanced 配置 diff --git a/docs/src/zh/guide/getting-started.md b/docs/src/zh/guide/getting-started.md index 16f5e209..1df9f343 100644 --- a/docs/src/zh/guide/getting-started.md +++ b/docs/src/zh/guide/getting-started.md @@ -1,17 +1,14 @@ ---- -outline: deep ---- # 快速上手 ## 安装 ### 下载二进制包 -直接在此处下载:https://github.com/tair-opensource/RedisShake/releases +直接从 [Release](https://github.com/tair-opensource/RedisShake/releases) 下载二进制包。 -### 从源码编译 +### 从源代码编译 -确保本地有 Golang 环境: +要从源代码编译,确保您在本地机器上设置了 Golang 环境: ```shell git clone https://github.com/alibaba/RedisShake @@ -19,30 +16,25 @@ cd RedisShake sh build.sh ``` -## 运行 +## 使用 -假设现在有两个 Redis 实例: +假设你有两个 Redis 实例: * 实例 A:127.0.0.1:6379 * 实例 B:127.0.0.1:6380 -新建文件 `shake.toml`: +创建一个新的配置文件 `shake.toml`: ```toml -[sync_standalone_reader] +[sync_reader] address = "127.0.0.1:6379" -[redis_standalone_writer] +[redis_writer] address = "127.0.0.1:6380" ``` -启动 redis-shake: +要启动 RedisShake,运行以下命令: ```shell ./redis-shake shake.toml - ``` - -运行日志如图: - -![img](./getting-started.png) \ No newline at end of file diff --git a/docs/src/zh/guide/getting-started.png b/docs/src/zh/guide/getting-started.png deleted file mode 100644 index 99785120..00000000 Binary files a/docs/src/zh/guide/getting-started.png and /dev/null differ diff --git a/docs/src/zh/guide/introduction.md b/docs/src/zh/guide/introduction.md new file mode 100644 index 00000000..feaaee96 --- /dev/null +++ b/docs/src/zh/guide/introduction.md @@ -0,0 +1,43 @@ +--- +outline: deep +--- + +# 什么是 RedisShake + +RedisShake 是一个用于处理和迁移 Redis 数据的工具,它提供以下特性: + +1. **Redis 兼容性**:RedisShake 兼容从 2.8 到 7.2 的 Redis 版本,并支持各种部署方式,包括单机,主从,哨兵和集群。 +2. **云服务兼容性**:RedisShake 与主流云服务提供商提供的流行 Redis-like 数据库无缝工作,包括但不限于: + - [阿里云-云数据库 Redis 版](https://www.aliyun.com/product/redis) + - [阿里云-云原生内存数据库Tair](https://www.aliyun.com/product/apsaradb/kvstore/tair) + - [AWS - ElastiCache](https://aws.amazon.com/elasticache/) + - [AWS - MemoryDB](https://aws.amazon.com/memorydb/) +3. **Module 兼容**:RedisShake + 与 [TairString](https://github.com/tair-opensource/TairString),[TairZSet](https://github.com/tair-opensource/TairZset) + 和 [TairHash](https://github.com/tair-opensource/TairHash) 模块兼容。 +4. **多种导出模式**:RedisShake 支持 PSync,RDB 和 Scan 导出模式。 +5. **数据处理**:RedisShake 通过自定义脚本实现数据过滤和转换。 + +## 贡献 + +我们欢迎社区的贡献。对于重大变更,请先开一个 issue 来讨论你想要改变的内容。我们特别感兴趣的是: + +1. 添加对更多模块的支持 +2. 提高对 Readers 和 Writers 的支持 +3. 分享你的 Lua 脚本和最佳实践 + +## 历史 + +RedisShake 是阿里云 [Tair 团队](https://github.com/tair-opensource) +积极维护的一个项目。它的演变可以追溯到其初始版本,该版本是从 [redis-port](https://github.com/CodisLabs/redis-port) 分支出来的。 + +版本(不同版本间配置不通用): + +- [RedisShake 2.x](https://github.com/tair-opensource/RedisShake/tree/v2) 版本带来了一系列的改进和更新,提高了其整体稳定性和性能。 +- [RedisShake 3.x](https://github.com/tair-opensource/RedisShake/tree/v3) 版本是一个重要的里程碑,整个代码库被完全重写和优化,具有更好的效率和可用性。 +- [RedisShake 4.x](https://github.com/tair-opensource/RedisShake/tree/v4) 版本 + ,进一步增强了 [Reader](../reader/scan_reader.md)、配置、可观察性和 [function](../function/introduction.md) 相关的特性。 + +## 许可证 + +RedisShake 在 [MIT 许可证](https://github.com/tair-opensource/RedisShake/blob/v2/license.txt) 下开源。 \ No newline at end of file diff --git a/docs/src/zh/guide/mode.md b/docs/src/zh/guide/mode.md new file mode 100644 index 00000000..8a0c39c8 --- /dev/null +++ b/docs/src/zh/guide/mode.md @@ -0,0 +1,11 @@ +--- +outline: deep +--- + +# 迁移模式选择 + +目前 RedisShake 有三种迁移模式:`PSync`、`RDB` 和 +`SCAN`,分别对应 [`psync_reader`](../reader/psync_reader.md)、[`rdb_reader`](../reader/rdb_reader.md) +和 [`scan_reader`](../reader/scan_reader.md)。这三种模式各有适合的场景,本文根据场景特点介绍如何选择。 + +TODO diff --git a/docs/src/zh/reader/rdb_reader.md b/docs/src/zh/reader/rdb_reader.md new file mode 100644 index 00000000..1865353b --- /dev/null +++ b/docs/src/zh/reader/rdb_reader.md @@ -0,0 +1,13 @@ +# rdb_reader + +## 介绍 + +可以使用 `rdb_reader` 来从 RDB 文件中读取数据,然后写入目标端。常见于从备份文件中恢复数据。 + +## 配置 + +```toml +[rdb_reader] +filepath = "/tmp/dump.rdb" +``` + diff --git a/docs/src/zh/reader/scan_reader.md b/docs/src/zh/reader/scan_reader.md new file mode 100644 index 00000000..3abd2c1b --- /dev/null +++ b/docs/src/zh/reader/scan_reader.md @@ -0,0 +1,34 @@ +# Scan Reader + +## 介绍 + +当源端数据库不兼容 PSync 协议时,推荐使用 `scan_reader`。 + +优势:兼容性好,只要源端数据库支持 `SCAN` 与 `DUMP` 命令,就可以使用 `scan_reader`。 + +劣势: + +1. 数据一致性不如 [`sync_reader`](./sync_reader.md)。 +2. `SCAN` 命令与 `DUMP` 命令会占用源端数据库较多的 CPU 资源 + +## 配置 + +```toml +[scan_reader] +cluster = false +address = "127.0.0.1:6379" # when cluster is true, address is one of the cluster node +username = "" # keep empty if not using ACL +password = "" # keep empty if no authentication is required +tls = false +ksn = false +``` + +默认情况下,RedisShake 会使用 `SCAN` 命令来遍历一遍所有的 Key,分别使用 `DUMP` 与 `RESTORE` 命令来从源端读取 Key +的内容,并写入目标端。Redis 的 SCAN 命令只保证 SCAN 的开始与结束之前均存在的 Key 一定会被返回,但是新写入的 Key 有可能会被遗漏。 + +如果需要提高数据一致性,可以开启 `ksn` 参数,这样 RedisShake 会在 `SCAN` +之前使用 [Redis keyspace notifications](https://redis.io/docs/manual/keyspace-notifications/) +能力来订阅 Key 的变化。当 Key 发生变化时,RedisShake 会使用 `DUMP` 与 `RESTORE` 命令来从源端读取 Key 的内容,并写入目标端。 +::: warning +Redis keyspace notifications 不会感知到 `FLUSHALL` 与 `FLUSHDB` 命令,因此在使用 `ksn` 参数时,需要确保源端数据库不会执行这两个命令。 +::: diff --git a/docs/src/zh/reader/sync_reader.md b/docs/src/zh/reader/sync_reader.md new file mode 100644 index 00000000..16b8e06f --- /dev/null +++ b/docs/src/zh/reader/sync_reader.md @@ -0,0 +1,23 @@ +# Sync Reader + +## 介绍 + +当源端数据库兼容 PSync 协议时,推荐使用 `sync_reader`。兼容 PSync 协议的数据库有: + +* Redis +* Tair +* ElastiCache 部分兼容 +* MemoryDB 部分兼容 + +优势:数据一致性最佳,可以实现不停机的切换 + +## 配置 + +```toml +[sync_reader] +cluster = false +address = "127.0.0.1:6379" # when cluster is true, address is one of the cluster node +username = "" # keep empty if not using ACL +password = "" # keep empty if no authentication is required +tls = false +``` diff --git a/docs/src/zh/writer/redis_writer.md b/docs/src/zh/writer/redis_writer.md new file mode 100644 index 00000000..9d6fec1c --- /dev/null +++ b/docs/src/zh/writer/redis_writer.md @@ -0,0 +1,16 @@ +# Redis Writer + +## 介绍 + +`redis_writer` 用于将数据写入 Redis-like 数据库。 + +## 配置 + +```toml +[redis_writer] +cluster = false +address = "127.0.0.1:6379" # when cluster is true, address is one of the cluster node +username = "" # keep empty if not using ACL +password = "" # keep empty if no authentication is required +tls = false +``` diff --git a/shake.toml b/shake.toml index 6f5824d4..2b0afb87 100644 --- a/shake.toml +++ b/shake.toml @@ -16,11 +16,7 @@ tls = false # tls = false # [rdb_reader] -# address = "127.0.0.1:6379" -# username = "" # keep empty if not using ACL -# password = "" # keep empty if no authentication is required -# tls = false - +# filepath = "/tmp/dump.rdb" [redis_writer] cluster = false