Detecting Diachronic Meaning Shifts in Dutch

This is the source code for the work presented in the CLIN talk ''Can we spot meaning shifts in diachronic representations?'' (abstract | slides). A paper based on the experiments presented in this talk, ''A Critical Assessment of a Method for Detecting Diachronic Meaning Shifts: Lessons Learnt from Experiments on Dutch'' (pdf), is published in the CLIN Journal (pdf).

You can use this code to create diachronic meaning representations based on any significantly large diachronic corpus of Dutch, and investigate and analyse those representations to detect known meaning shifts.

Requirements

To run this code, you'll need:

• Python 2.7.6
• gensim 0.13.2
• scipy 0.18.1
• matplotlib 2.0.0

Different versions might work just as well, but cannot be guaranteed. Most scripts also expect there to be a folder called working in the main code directory.

In addition, we made use of:

• The Elephant tokenizer
• The LASSY Small corpus

Additional Files

• most_frequent_*_5000.txt: Lists of the 5000 most frequent words/nouns/verbs in LASSY
• most_frequent_words_5000_average_similarity.json: Average similarity in the Trouw and Volkskrant corpora for the 5000 most frequent words in LASSY
• word_list_combined.txt: Initial list of words to investigate (before frequency cut-off)
• word_list_combined_results.json: Counts, frequencies and self-similarities in both corpora for the words in the list above
• plots/*.png: Plots of the words discussed in the paper and the talk, as generated by the script analyze_and_plot_words.py.