This repository provides a reference implementation of Layer-wise Relevance Propagation (LRP) for LSTMs, as initially proposed in the paper Explaining Recurrent Neural Network Predictions in Sentiment Analysis, L. Arras, G. Montavon, K.-R. Müller and W. Samek WASSA@EMNLP'2017 [arXiv:1706.07206].
Additionally it includes an implementation of Sensitivity Analysis (SA) and Gradient x Input (GI), i.e. of gradient-based relevances.
Note that our implementation is generic and can be easily extended to unidirectional LSTMs, or to other application domains than Natural Language Processing.
A few hints on how to apply and extend the code to your needs can be found here.
Python>=3.5 + Numpy + Matplotlib, or alternatively, simply install Anaconda.
Using conda, you can e.g. create a Python 3.6 environment: conda create -n py36 python=3.6 anaconda
Then activate it with: source activate py36
Before being able to use the code, you might need to run in the terminal: export PYTHONPATH=$PYTHONPATH:$pwd
The folder model/ contains a word-based bidirectional LSTM model, that was trained for five-class sentiment prediction of phrases and sentences on the Stanford Sentiment Treebank (SST) dataset, as released by the authors in Visualizing and Understanding Neural Models in NLP, J. Li, X. Chen, E. Hovy and D. Jurafsky, 2016
The folder data/ contains the test set sentences of the Stanford Sentiment Treebank (SST), preprocessed by lowercasing, as was done in Visualizing and Understanding Neural Models in NLP, J. Li, X. Chen, E. Hovy and D. Jurafsky, 2016
The notebook run_example.ipynb provides a usage example of the code, its performs LRP, SA and GI on a test sentence.
(To correctly render the notebook on GitHub you can copy the notebook's URL to nbviewer)
Here are some follow-up works using LRP on various recurrent neural network models and tasks (non-exhaustive list):
[1] Evaluating Neural Network Explanation Methods using Hybrid Documents and Morphosyntactic Agreement, N. Poerner, H. Schütze and B. Roth, ACL 2018 [arXiv:1801.06422] [code]
[2] Explaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks, Y. Yang, V. Tresp, M. Wunderle and P.A. Fasching, IEEE ICHI 2018 [preprint] [code]
[3] Analyzing Neuroimaging Data Through Recurrent Deep Learning Models, A.W. Thomas, H.R. Heekeren, K.-R. Müller and W. Samek, Frontiers in Neuroscience 2019 [blog]
[4] Evaluating Recurrent Neural Network Explanations, L. Arras, A. Osman, K.-R. Müller and W. Samek, BlackboxNLP@ACL 2019 [arXiv:1904.11829] [oral presentation slides]
[5] Explaining and Interpreting LSTMs, L. Arras, J. Arjona-Medina, M. Widrich, G. Montavon, M. Gillhofer, K.-R. Müller, S. Hochreiter and W. Samek, Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS vol 11700 2019 [arXiv:1909.12114]
[6] Evaluating Explanation Methods for Deep Learning in Security, A. Warnecke, D. Arp, C. Wressnegger and K. Rieck, IEEE European Symposium on Security and Privacy 2020 [arXiv:1906.02108] [code_main] [code_lrp]
LRP for LSTMs self-contained implementation in Tensorflow 2.1 (allowing efficient batch and GPU processing) by Alexander Warnecke: github repository
Visualizing and Understanding Neural Models in NLP, J. Li, X. Chen, E. Hovy and D. Jurafsky, code
Long Short Term Memory Units, repo by W. Zaremba
Stanford Sentiment Treebank (SST), dataset by R. Socher et al., 2013
If you find this project useful, please cite our original paper (or one of our follow-up publications see above [4,5]):
@INPROCEEDINGS{arras2017,
title = {{Explaining Recurrent Neural Network Predictions in Sentiment Analysis}},
author = {Leila Arras and Gr{\'e}goire Montavon and Klaus-Robert M{\"u}ller and Wojciech Samek},
booktitle = {Proceedings of the EMNLP 2017 Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
year = {2017},
pages = {159-168},
publisher = {Association for Computational Linguistics},
doi = {10.18653/v1/W17-5221},
url = {https://www.aclweb.org/anthology/W17-5221}
}
For other research and projects involving LRP, you can visit the website heatmapping.org