Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

请教一下将随机写转换为对应分区的顺序写 #1

Open
zhng1456 opened this issue Dec 27, 2018 · 2 comments
Open

请教一下将随机写转换为对应分区的顺序写 #1

zhng1456 opened this issue Dec 27, 2018 · 2 comments

Comments

@zhng1456
Copy link

您好,请问一下,getPartition()那里,为什么这样做转化为了顺序写?key的分布是完全均匀的吗?谢谢

@AlexZFX
Copy link

AlexZFX commented Jan 16, 2019

key和value的分布在本次比赛中基本是完全均匀的,无论是根据后几位进行hash或是根据前几位进行hash都能将key 分散写入到n个文件中去,但是初赛的时候 check阶段应该是重复写入了一个key很多次,所以没处理好可能会有问题。

他把文件分成了1024个key文件和1024个value文件的话,写入的时候只要key和value对应到同一个index的文件中去就可以不用存储offset,写入的时候就是简单的追加写,读取之后,在对key排序那一块进行了去重等操作。

@lexburner
Copy link
Owner

@zhng1456 就是 @AlexZFX 他说的这个意思,有些你看不懂的地方,可能就是针对赛题做的简化操作。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants