Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support PITR #661

Merged
merged 2 commits into from
Oct 30, 2023
Merged

Support PITR #661

merged 2 commits into from
Oct 30, 2023

Conversation

bug-superman
Copy link
Contributor

@bug-superman bug-superman commented Aug 27, 2023

此pr在RedisShake的sync模块中增加了aof恢复数据的功能(增加了aof_reader),提供两个功能:
1、支持将任意合法的AOF(redis 7.0之前的single AOF和redis 7.0之后的multi part AOF)回放到目标redis
2、对于redis 7.0之后产生的AOF,如果AOF中具有timestamp annotation,则在回放时支持按时间点过滤能力

ref issue: #593

@CLAassistant
Copy link

CLAassistant commented Aug 27, 2023

CLA assistant check
All committers have signed the CLA.

@suxb201
Copy link
Member

suxb201 commented Aug 28, 2023

Great job! Would you be willing to rebase your changes onto the v4 branch? We are currently working on the development of the v4 branch. You can refer to this documentation: https://tair-opensource.github.io/RedisShake/

@chenyang8094
Copy link
Collaborator

ref issue: #593

@chenyang8094 chenyang8094 self-requested a review September 6, 2023 05:25
@bug-superman
Copy link
Contributor Author

有两问题:1.为什么在执行rdb恢复的时候没有检查文件完整性,例如rdb文件可以有crc64校验码,chensum。但我们并没有进行检查
2.现有的rdb是支持文件解析但不能进行offset返回。在使用rdb 混合aof格式时。无法使用现有的rdb解析代码完成,需要对此部分代码进行修改。
我们在发送之前做的检查我理解现在就是检查了命令是否存在。而不是针对文件进行校验和检查,是因为效率的原因么。读一遍文件做为检查的代价太大了不能接受么。

@bug-superman
Copy link
Contributor Author

干得好!您愿意将更改重构到v4分支上吗?我们目前正在开发v4分支。您可以参考此文档:https://tair-opensource.github.io/RedisShake/

cool !

@chenyang8094
Copy link
Collaborator

chenyang8094 commented Sep 12, 2023

有两问题:1.为什么在执行rdb恢复的时候没有检查文件完整性,例如rdb文件可以有crc64校验码,chensum。但我们并没有进行检查 2.现有的rdb是支持文件解析但不能进行offset返回。在使用rdb 混合aof格式时。无法使用现有的rdb解析代码完成,需要对此部分代码进行修改。 我们在发送之前做的检查我理解现在就是检查了命令是否存在。而不是针对文件进行校验和检查,是因为效率的原因么。读一遍文件做为检查的代价太大了不能接受么。

我认为你可以不用考虑preamble格式,因为aof annotation和multi part aof都是在redis 7.0中release的,因此在7.0中,一旦开启preamble,那么只有base aof是rdb格式,incr aof依然是传统的resp格式。所以,你需要做的,就是支持multi part aof的增量文件按时间点闪回就好了(base文件如果有就全部回放)。

image

@bug-superman
Copy link
Contributor Author

对于用户给的aof文件我们是否需要进行完整性检查,在写入前对文件的命令、格式进行check。目前代码似乎就是读取然后解析命令,然后写入,在解析途中遇到问题会报一个warnning。这是因为出于性能原因考虑吗,完整check一次文件太浪费时间?

@chenyang8094 chenyang8094 changed the title add restore aof function and parse rdb at an offset Support PITR Sep 14, 2023
@chenyang8094
Copy link
Collaborator

对于用户给的aof文件我们是否需要进行完整性检查,在写入前对文件的命令、格式进行check。目前代码似乎就是读取然后解析命令,然后写入,在解析途中遇到问题会报一个warnning。这是因为出于性能原因考虑吗,完整check一次文件太浪费时间?

可以暂时不考虑,因为aof没有rdb那样的checksum,想要校验的话只能从头捋一遍。

PITR时一般木标redis都是空实例,这个时候如果PITR中途失败了shake直接打印错误日志并退出就好,用户可以选择稍后flushall目标redis再使用正确的备份集重试PITR

@bug-superman
Copy link
Contributor Author

看v4版目前没有了fliter,是打算后续采用acl 命令来做吗

@suxb201
Copy link
Member

suxb201 commented Sep 25, 2023

看v4版目前没有了fliter,是打算后续采用acl 命令来做吗

这个:https://tair-opensource.github.io/RedisShake/zh/function/introduction.html

@chenyang8094
Copy link
Collaborator

chenyang8094 commented Sep 25, 2023

@bug-superman 在PR顶部描述下你具体做了哪些改动,引入了什么功能、新增了什么配置、具体用法更新到使用文档。处理下test失败。

internal/reader/parsing_aof.go Outdated Show resolved Hide resolved
tests/cases/aof.py Show resolved Hide resolved
shake.toml Show resolved Hide resolved
shake.toml Outdated Show resolved Hide resolved
docs/src/zh/guide/config.md Show resolved Hide resolved
internal/reader/aof_reader.go Outdated Show resolved Hide resolved
internal/reader/aof_reader.go Outdated Show resolved Hide resolved
internal/reader/parsing_aof.go Outdated Show resolved Hide resolved
internal/reader/parsing_aof.go Outdated Show resolved Hide resolved
internal/reader/parsing_aof.go Outdated Show resolved Hide resolved
docs/src/zh/reader/aof_reader.md Outdated Show resolved Hide resolved
docs/src/en/reader/aof_reader.md Outdated Show resolved Hide resolved
internal/reader/parsing_aof.go Outdated Show resolved Hide resolved
internal/aof/aof.go Outdated Show resolved Hide resolved
shake.toml Outdated Show resolved Hide resolved
tests/appendonlydir/appendonly.aof.1.incr.aof Outdated Show resolved Hide resolved
internal/reader/parsing_aof.go Show resolved Hide resolved
@chenyang8094
Copy link
Collaborator

@suxb201 你可以看下这个PR还有没有其他的问题。

@chenyang8094 chenyang8094 self-assigned this Oct 27, 2023
tests/cases/aof.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@chenyang8094 chenyang8094 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

docs/src/en/reader/aof_reader.md Outdated Show resolved Hide resolved
Copy link
Member

@suxb201 suxb201 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chenyang8094 chenyang8094 merged commit 6e51110 into tair-opensource:v4 Oct 30, 2023
7 checks passed
@suxb201
Copy link
Member

suxb201 commented Oct 30, 2023

🎉

@Exile-Twilight
Copy link

image
请问解析aof过程中出于什么考虑限制了命令中参数的个数,Redis好像没有这方面的限制

@suxb201
Copy link
Member

suxb201 commented Aug 2, 2024

@Exile-Twilight 是的,不应该限制。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants