Automatic extraction of data from clinical trial reports
RobotReviewer is a system for providing automatic annotations from clinical trials (in PDF format). Currently, RobotReviewer provides data on the trial PICO characteristics (Population, Interventions/Comparators, and Outcomes), and also automatically assesses trials for likely biases using the Cochrane Risk of Bias tool.
This software is the web-service version, meaning it's aimed at people who make systematic review software.
For most systematic review authors, if you want to try out RobotReviewer, you'd probably be better using the demo version on our website, available here. If you like it, you could email the person who maintains your systematic review software a link to this site - they might be interested in adding it.
(Alternatively, individual authors who are adept at installing unix software from the terminal are free to install this version on their own machines by following the optional 'Web UI' instructions below).
RobotReviewer is open source and free to use under the GPL licence, version 3.0 (see the LICENCE.txt file in this repository).
We offer RobotReviewer free of charge, but we'd be most grateful if you would cite us if you use it. We're academics, and thrive on links and citations! Getting RobotReviewer widely used and cited helps us obtain the funding to maintain the project and make RobotReviewer better.
It also makes your methods transparent to your readers, and not least we'd love to see where RobotReviewer is used! :)
We'd appreciate it you could:
- Display the text, 'Risk of Bias automation by RobotReviewer (how to cite)' on the same screen or webpage on which the RobotReviewer results (highlighted text or risk of bias judgements) are displayed.
- For web-based tools, the text 'how to cite' should link to our website
http://vortext.systems/robotreviewer - For desktop software, you should usually link to the same website. If this is not possible, you may alternately display the text and example citations from the 'How to cite RobotReviewer' section below.
You can cite RobotReviewer as:
Marshall IJ, Kuiper J, & Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association 2015. doi:10.1093/jamia/ocv044
A BibTeX entry for LaTeX users is
@article{RobotReviewer2015,
title = {{RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials}},
author = {Marshall, Iain J and Kuiper, Jo\"{e}l and Wallace, Byron C},
doi = {10.1093/jamia/ocv044},
url = {http://dx.doi.org/10.1093/jamia/ocv044},
journal = {Journal of the American Medical Informatics Association},
year = {2015}
month = jun,
pages = {ocv044}
}
-
Ensure you have a working version of 3.4+. We recommend using Python from the Anaconda Python distribution for a quicker and more reliable experience. However, if you have Python already installed that will probably work fine too.
-
Install git-lfs for managing the model file versions (on Mac:
brew install git-lfs). NB! If you already have git lfs installed, make sure it's the most recent version, since older versions have not downloaded files properly. -
Get a copy of the RobotReviewer repo, and go into that directory
git clone https://github.com/ijmarshall/robotreviewer3.git cd robotreviewer3 -
Install the Python libraries that RobotReviewer needs - do one of the following.
a. If you are using Anaconda:
conda config --add channels spacy # only needed once
conda install flask numpy scipy scikit-learn spacy Flask-WTF requests pandas
pip install fuzzywuzzy mpld3 python-dateutil python-Levenshtein # (not yet in the anaconda repo)b. If you are NOT using Anaconda:
pip install flask numpy scipy scikit-learn spacy fuzzywuzzyFor everyone:
# install theano and keras from source
pip install git+https://github.com/Theano/Theano.git
pip install git+https://github.com/fchollet/[email protected]
# then install sentence processing data
python -m spacy.en.download
python -m nltk.downloader punktEnsure keras is set to use theano as its default backend. Steps on how to do this can be found here.
-
This version of RobotReviewer requires Grobid, which in turn uses Java. Follow the instructions here to download and build it.
-
Create the
robotreviewer/config.jsonfile and ensure it contains the path to the directory where you have installed Grobid. (RobotReviewer will start it automatically in a subprocess). Note that this should be the path to the entire (parent) Grobid directory, not the bin subfolder. An example of this file is provided inrobotreviewer/config.json.example(it is only necessary to change thegrobid_pathattribute).
The following
python -m robotreviewerwill start a flask server running on http://localhost:5000. You can run the server in development mode by passing DEBUG=true python -m robotreviewer. Visiting this address from a browser will show the new multiple PDF synthesis demonstration.
We have included some example reports, with open access RCT PDFs in order to demonstrate RobotReviewer. These are saved in the default database, and can be accessed via the following links.
Decision aids: http://localhost:5000/#report/Tvg0-pHV2QBsYpJxE2KW-
Influenza vaccination: http://localhost:5000/#report/_fzGUEvWAeRsqYSmNQbBq
Hypertension: http://localhost:5000/#report/HBkzX1I3Uz_kZEQYeqXJf
The big change in this version of RobotReviewer is that we now deal with groups of clinical trial reports, rather than one at a time. This is to allow RobotReviewer to synthesise the results of multiple trials.
As a consequence, the API has become more sophisticated than previously and we will add further documentation about it here.
In the meantime, the code for the API endpoints can be found in /robotreviewer/app.py.
Some things remain simple; e.g., for an example of using RR to classify abstracts as RCTs (or not) see this gist.
If you are interested in incorporating RobotReviewer into your own software, please contact us and we'd be pleased to assist.
The following
python -m unittestwill run the testing modules. These should be used to assure that changes made do not break or have an affect on the core of the code. If Ran X tests in Ys is displayed, the tests have completed successfully.
Feel free to contact us at [email protected] with any questions.
- Marshall, I. J., Kuiper, J., & Wallace, B. C. (2015). RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association. [doi]
- Zhang Y, Marshall I. J., & Wallace, B. C. (2016) Rationale-Augmented Convolutional Neural Networks for Text Classification. Conference on Empirical Methods on Natural Language Processing. [preprint]
- Marshall, I., Kuiper, J., & Wallace, B. (2015). Automating Risk of Bias Assessment for Clinical Trials. IEEE Journal of Biomedical and Health Informatics. [doi]
- Kuiper, J., Marshall, I. J., Wallace, B. C., & Swertz, M. A. (2014). Spá: A Web-Based Viewer for Text Mining in Evidence Based Medicine. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2014) (Vol. 8726, pp. 452–455). Springer Berlin Heidelberg. [doi]
- Marshall, I. J., Kuiper, J., & Wallace, B. C. (2014). Automating Risk of Bias Assessment for Clinical Trials. In Proceedings of the ACM Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) (pp. 88–95). ACM. [doi]
Copyright (c) 2017 Iain Marshall, Joël Kuiper, and Byron Wallace
This work is supported by: National Institutes of Health (NIH) under the National Library of Medicine, grant R01-LM012086-01A1, "Semi-Automating Data Extraction for Systematic Reviews", and by NIH grant 5UH2CA203711-02, "Crowdsourcing Mark-up of the Medical Literature to Support Evidence-Based Medicine and Develop Automated Annotation Capabilities", and the UK Medical Research Council (MRC), through its Skills Development Fellowship program, grant MR/N015185/1