New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use intersection of tokens founded at different sources #24

Open

fabmue opened this issue Dec 5, 2017 · 0 comments

Contributor

fabmue commented Dec 5, 2017

We collected data from 1) medical PDFs, 2) medical websites (crawler), 3) medical wikipedia articles (from wiki-dumps)

To get rid of unwanted tokens only a subset of tokens, which appear in more than 1 source.
e.g. intersection = pdfs & (crawler | wiki_dumps)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment