This repository has been archived by the owner on Mar 10, 2023. It is now read-only.

去除重复内容 #9

Open

gricn opened this issue Aug 27, 2021 · 0 comments

Labels

bug

Member

gricn commented Aug 27, 2021 •

edited

Loading

由于在之前版本更新策略的失误，出现了一些重复爬取的文章。

以“国家卫生健康委员会医师资格考试委员会公告”为例，出现了4处相同的文章

搜索界面

Kibana查看情况

能否参照以下内容对仓库去重呢？
https://github.com/alexander-marquardt/deduplicate-elasticsearch/blob/master/deduplicate-elaticsearch.py
https://stackoverflow.com/questions/29886477/how-to-remove-duplicate-search-result-in-elasticsearch

gricn added the bug label

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.