Home

Workflow of Pubmed Parser with PySpark

We includes PySpark snippets on how to parse Pubmed Open-Access and MEDLINE dataset on wiki page here

• Setup Spark 2.1
• MEDLINE
• Pubmed Open-Access Subset

Alternative parsers

• Alternative parser http://fnl.es/medline-kung-fu.html
• Transform MEDLINE XML to json using node, https://github.com/ldbib/MEDLINEXMLToJSON

Source links

• Pubmed Open-Access (OA) dataset is available at http://www.ncbi.nlm.nih.gov/pmc/tools/ftp/
• the MEDLINE XMLs are available here ftp://ftp.nlm.nih.gov/nlmdata/.medleasebaseline/gz/
• the MEDLINE XMLs weekly updates are available here ftp://ftp.nlm.nih.gov/nlmdata/.medlease/gz/
• MEDLINE DTD file is available at this link. We can use it to see available tag from MEDLINE xml.

PMC Copyright Notice

• Please see copyright notice when you scrape data from website here