pfp -- pretty fast statistical parser for natural languages

pfp is a pretty fast statistical parser for probabilistic context free grammars. pfp uses the exhaustive CYK algorithm found in the Stanford NLP parser (and gratefully uses its trained grammar), but boasts the following improvements:

• 3-4x faster than the Stanford parser
• Uses 5-8x less resident memory
• Thread-safe/multi-core
• Python bindings
• Lightweight threadpool http server
• Command-line client

Installing

The following works on Ubuntu 10.04 LTS:

sudo apt-get install -y git-core cmake build-essential libboost-all-dev python-dev
git clone http://github.com/wavii/pfp.git && cd pfp
cmake .
make
./test && sudo make install

To install the python library:

sudo python setup.py install

If you are running OS X using Homebrew you may need to first get your dependencies in order:

brew install icu4c boost boost-jam
ln -s /usr/local/Cellar/icu4c/4.4.1/include/unicode /usr/local/include/
ln -s /usr/local/Cellar/icu4c/4.4.1/include/layout /usr/local/include/
ls -1 /usr/local/Cellar/icu4c/4.4.1/lib/ | xargs -IFILE ln -s /usr/local/Cellar/icu4c/4.4.1/lib/FILE /usr/local/lib

Benchmarks

Parse times and maximum memory usage were recorded on a c1.medium Amazon EC2 instance. pfp is compared to Stanford Parser version 1.6.3 running on sun-java6.

Sentence Length  Stanford (avg ms)  pfp (avg ms)   Stanford (res mb) pfp (res mb)
0-9              21.0407725322      5.12853470437  83                10
10-19            104.675603217      34.9076086957  83                14
20-29            363.596421471      124.78         105               21
30-39            830.084942085      320.664122137  150               30
40-45            1607.43820225      542.533333333  150               35

Usage

pfpc is a command-line client that reads from stdin and writes parses to stdout:

$ echo "I love monkeys." | pfpc 2>/dev/null
(ROOT (S (NP (PRP I)) (VP (VBP love) (NP (NNS monkeys))) (. .)) )

pfpd is a threadpool web server that wraps pfp:

$ pfpd localhost 8080 2>/dev/null &
[1] 3600
$ curl http://localhost:8080/parse/I+love+monkeys.
(ROOT (S (NP (PRP I)) (VP (VBP love) (NP (NNS monkeys))) (. .)) )

pypfp are python bindings for pfp:

$ python
>>> import pfp
>>> pfp.Parser().parse("I love monkeys.")
'(ROOT (S (NP (PRP I)) (VP (VBP love) (NP (NNS monkeys))) (. .)) )'

License

(GPL v2)

Copyright (c) 2010 Wavii, Inc. http://wavii.com/

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.