domain_adaptation

Lecture slides - by Boris Kovarsky

More materials

• Russian videos (2021): lecture and practice
• Materials '2020 : lecture, seminar
• An overview of proxy-label approaches for semi-supervised learning - blog post
• A Little Review of Domain Adaptation in 2017 - blog post
• Awesome Transfer Learning - list of papers repo
• Awesome Domain Adaptation - list of papers repo

Practice

Optional assignment: ./homework.ipynb

Videos

• The Future of Natural Language Processing by Thomas Wolf [youtube]
• Tutorial on Domain Adaptation by Hal Daumé III [youtube]
• Domain Adaptation with Structural Correspondence Learning by John Blitzer [youtube]
• Learning with Scarce and Biased Data in Natural Language Processing by Barbara Plank [youtube]

Articles

General articles

• Neural Unsupervised Domain Adaptation in NLP---A Survey by Alan Ramponi, Barbara Plank, 2021 [article]

Theory

• Learning Bounds for Domain Adaptation Blitzer et al. 2007 [article]
• A theory of learning from different domains Shai Ben-David, John Blitzer 2010 [article]
• Analysis of Representations for Domain Adaptation Ben-Daivid et al. 2006 [article]

In-domain vs out-domain generalization

• Learning and Evaluating General Linguistic Intelligence Yogatama et al. 2019 [article]
• BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance R. Thomas McCoy, Junghyun Min, Tal Linzen 2019 [article]

Data-centric methods for unsupervised domain adaptation

Unsupervised pre-training

• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks Gururangan et al. 2020 [article]
• Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling Xiaochuang Han, Jacob Eisenstein 2020 [article]
• UDALM: Unsupervised Domain Adaptation through Language Modeling Constantinos Karouzos et al. 2021 [article]

Instance weighting/Data selection

• Instance Weighting for Domain Adaptation in NLP Jiang, Zhai 2007 [pdf]
• Instance Weighting for Neural Machine Translation Domain Adaptation Wang et al. 2017 [pdf]
• Cost Weighting for Neural Machine Translation Domain Adaptation Chen et al. 2017 [pdf]
• Learning to select data for transfer learning with Bayesian Optimization by Ruder, Plank 2017 [article]

Proxy-label methods

• Asymmetric Tri-training for Unsupervised Domain Adaptation Saito et al. 2017 [arxiv]
• Strong Baselines for Neural Semi-supervised Learning under Domain Shift Sebastian Ruder, Barbara Plank 2018 [arxiv]

Back-translation

• Improving Neural Machine Translation Models with Monolingual Data Sennrich et al. 2016 [arxiv]
• Iterative Back-Translation for Neural Machine Translation Hoang et al. 2018 [pdf]

Model-centric methods for unsupervised domain adaptation

Deep Feature Alignment/Adversarial methods

• Unsupervised Domain Adaptation by Backpropagation Yaroslav Ganin, Victor Lempitsky 2014 [arxiv]
• Learning Transferable Features with Deep Adaptation Networks Long et al. 2015 [arxiv]
• Adversarial and Domain-Aware BERT for Cross-Domain Sentiment Analysis Du et al. 2020 [article]
• Domain Adaptation with Structural Correspondence Learning Blitzer et al. 2007 [article]

Pivot

Batch Normalization-like:

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe, Christian Szegedy, 2015 [arxiv]
• Revisiting Batch Normalization For Practical Domain Adaptation Li et al. 2016 [arxiv]
• AutoDIAL: Automatic DomaIn Alignment Layers Carlucci et al. 2017 [arxiv]

Knowledge Distillation

• Distilling the Knowledge in a Neural Network Hinton et al. 2015 [arxiv]
• Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data Dakwale, Monz 2017 [pdf]