Merge pull request #132 from mbwolff/FixFrStopwords

Fix fr stopwords

README.md

@@ -3,9 +3,19 @@
 
 About
 -----
-[Serendip-o-matic](http://serendipomatic.org/) connects your sources to digital materials located in libraries, museums, and archives around the world. By first examining your research interests, and then identifying related content in locations such as the Digital Public Library of America (DPLA), Europeana, and Flickr Commons, our serendipity engine helps you discover photographs, documents, maps and other primary sources.
-
-Whether you begin with text from an article, a Wikipedia page, or a full Zotero collection, Serendip-o-matic's special algorithm extracts key terms and returns a surprising reflection of your interests. Because the tool is designed mostly for inspiration, search results aren't meant to be exhaustive, but rather suggestive, pointing you to materials you might not have discovered. At the very least, the magical input-output process helps you step back and look at your work from a new perspective. Give it a whirl. Your sources may surprise you.
+[Serendip-o-matic](http://serendipomatic.org/) connects your sources to digital materials
+located in libraries, museums, and archives around the world. By first examining your
+research interests, and then identifying related content in locations such as the Digital
+Public Library of America (DPLA), Europeana, and Flickr Commons, our serendipity engine
+helps you discover photographs, documents, maps and other primary sources.
+
+Whether you begin with text from an article, a Wikipedia page, or a full Zotero
+collection, Serendip-o-matic's special algorithm extracts key terms and returns a
+surprising reflection of your interests. Because the tool is designed mostly for
+inspiration, search results aren't meant to be exhaustive, but rather suggestive,
+pointing you to materials you might not have discovered. At the very least, the magical
+input-output process helps you step back and look at your work from a new perspective.
+Give it a whirl. Your sources may surprise you.
 
 Installation notes for developers
 ---------------------------------

smartstash/core/utils.py

@@ -33,7 +33,11 @@
 
 def tokenize(text, lang='en'):
     # if language is not specified or not in our list, fall back to english
-    stopwords = nltk.corpus.stopwords.words(stopword_lang.get(lang, 'english'))
+    stopwords = nltk.corpus.stopwords.words(stopword_lang.get(lang))
+    if lang == 'fr':
+    	stopwords.append('les')
+    	stopwords.append('a')
+
     tokens = nltk.word_tokenize(text)
     words = [w.lower() for w in tokens
              if w.isalnum() and w.lower() not in stopwords]

smartstash/nltk_data/corpora/stopwords/french

@@ -15,6 +15,7 @@ il
 je
 la
 le
+les
 leur
 lui
 ma
@@ -116,6 +117,7 @@ eu
 eue
 eues
 eus
+a
 ai
 as
 avons