Skip to content

Latest commit

 

History

History
125 lines (105 loc) · 15.7 KB

README.md

File metadata and controls

125 lines (105 loc) · 15.7 KB
Raw

Awesome Transformer Architecture Search: Awesome

To keep track of the large number of recent papers that look at the intersection of Transformers and Neural Architecture Search (NAS), we have created this awesome list of curated papers and resources, inspired by awesome-autodl, awesome-architecture-search, and awesome-computer-vision. Papers are divided into the following categories:

  1. General Transformer search
  2. Domain Specific, applied Transformer search (divided into NLP, Vision, ASR)
  3. Transformers Knowledge: Insights / Searchable parameters / Attention
  4. Transformer Surveys
  5. Foundation Models
  6. Misc Resources

This repository is maintained by Yash Mehta, please feel free to reach out, create pull requests or open an issue to add papers. Please see this Google Doc for a comprehensive list of papers at ICML 2023 on foundation models/large language models.

General Transformer Search

Title Venue Group
Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models NeurIPS'22 MSR
Training Free Transformer Architecture Search CVPR'22 Tencent & Xiamen University
LiteTransformerSearch: Training-free On-device Search for Efficient Autoregressive Language Models AutoML Conference 2022 Workshop Track MSR
Searching the Search Space of Vision Transformer NeurIPS'21 MSRA, Stony Brook University
UniNet: Unified Architecture Search with Convolutions, Transformer and MLP ECCV'22 SenseTime
Analyzing and Mitigating Interference in Neural Architecture Search ICML'22 Tsinghua, MSR
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search ICCV'21 Sun Yat-sen University
Memory-Efficient Differentiable Transformer Architecture Search ACL-IJCNLP'21 MSR, Peking University
Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition arxiv [Aug'20] Google Research
AutoTrans: Automating Transformer Design via Reinforced Architecture Search NLPCC'21 Fudan University
NASABN: A Neural Architecture Search Framework for Attention-Based Networks IJCNN'20 Chinese Academy of Sciences
NAT: Neural Architecture Transformer for Accurate and Compact Architectures NeurIPS'19 Tencent AI
The Evolved Transformer ICML'19 Google Brain

Domain Specific Transformer Search

Vision

Title Venue Group
𝛼NAS: Neural Architecture Search using Property Guided Synthesis ACM Programming Languages'22 MIT, Google
NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training ICLR'22 Meta Reality Labs
AutoFormer: Searching Transformers for Visual Recognition ICCV'21 MSR
GLiT: Neural Architecture Search for Global and Local Image Transformer ICCV'21 University of Sydney
Searching for Efficient Multi-Stage Vision Transformers ICCV'21 workshop MIT
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers CVPR'21 Bytedance Inc.

Natural Language Processing

Title Venue Group
AutoBERT-Zero: Evolving the BERT backbone from scratch AAAI'22 Huawei Noah’s Ark Lab
Primer: Searching for Efficient Transformers for Language Modeling NeurIPS'21 Google
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models ACL'21 Tsinghua, Huawei Naoh's Ark
NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search KDD'21 MSR, Tsinghua University
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing ACL'20 MIT

Automatic Speech Recognition

Title Venue Group
SFA: Searching faster architectures for end-to-end automatic speech recognition models Computer Speech and Language'23 Chinese Academy of Sciences
LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search ICASSP'21 MSR
Efficient Gradient-Based Neural Architecture Search For End-to-End ASR ICMI-MLMI'21 NPU, Xi'an
Evolved Speech-Transformer: Applying Neural Architecture Search to End-to-End Automatic Speech Recognition INTERSPEECH'20 VUNO Inc.

Transformers Knowledge: Insights, Searchable parameters, Attention

Title Venue Group
RWKV: Reinventing RNNs for the Transformer Era arxiv [May'23] EleutherAI
Patches are All You Need ? TMLR'23 CMU
Seperable Self Attention for Mobile Vision Transformers TMLR'23 Apple
Parameter-efficient Fine-tuning for Vision Transformers AAAI'23 MSR & UCSC
EfficientFormer: Vision Transformers at MobileNet Speed NeurIPS'22 Snap Inc
Neighborhood Attention Transformer CVPR'23 Meta AI
Training Compute Optimal Large Language Models NeurIPS'22 DeepMind
CMT: Convolutional Neural Networks meet Vision Transformers CVPR'22 Huawei Noah’s Ark Lab
Patch Slimming for Efficient Vision Transformers CVPR'22 Huawei Noah’s Ark Lab
Lite Vision Transformer with Enhanced Self-Attention CVPR'22 Johns Hopkins University, Adobe
TubeDETR: Spatio-Temporal Video Grounding with Transformers CVPR'22 (Oral) CNRS & Inria
Beyond Fixation: Dynamic Window Visual Transformer CVPR'22 UT Sydney & RMIT University
BEiT: BERT Pre-Training of Image Transformers ICLR'22 (Oral) MSR
How Do Vision Transformers Work? ICLR'22 (Spotlight) NAVER AI
Scale Efficiently: Insights from Pretraining and FineTuning Transformers ICLR'22 Google Research
Tuformer: Data-Driven Design of Expressive Transformer by Tucker Tensor Representation ICLR'22 UoMaryland
DictFormer: Tiny Transformer with Shared Dictionary ICLR'22 Samsung Research
QuadTree Attention for Vision Transformers ICLR'22 Alibaba AI Lab
Expediting Vision Transformers via Token Reorganization ICLR'22 (Spotlight) UC San Diego & Tencent AI Lab
UniFormer: Unified Transformer for Efficient Spatial-Temporal Representation Learning ICLR'22 SIAT-SenseTime
Hierarchical Transformers Are More Efficient Language Models NAACL'22 Google Research, UoWarsaw
Transformer in Transformer NeurIPS'21 Huawei Noah's Ark
Long-Short Transformer: Efficient Transformers for Language and Vision NeurIPS'21 NVIDIA
Memory-efficient Transformers via Top-k Attention EMNLP Workshop '21 Allen AI
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows ICCV'21 best paper MSR
Rethinking Spatial Dimensions of Vision Transformers ICCV'21 NAVER AI
What makes for hierarchical vision transformers arxiv [Sept'21] HUST
AutoAttend: Automated Attention Representation Search ICML'21 Tsinghua University
Rethinking Attention with Performers ICLR'21 Oral Google
LambdaNetworks: Modeling long-range Interactions without Attention ICLR'21 Google Research
HyperGrid Transformers ICLR'21 Google Research
LocalViT: Bringing Locality to Vision Transformers arxiv [April'21] ETH Zurich
Compressive Transformers for Long Range Sequence Modelling ICLR'20 DeepMind
Improving Transformer Models by Reordering their Sublayers ACL'20 FAIR, Allen AI
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned ACL'19 Yandex

Transformer Surveys

Title Venue Group
Transformers in Vision: A Survey ACM Computing Surveys'22 MBZ University of AI
A Survey of Vision Transformers TPAMI'22 CAS
Efficient Transformers: A Survey ACM Computing Surveys'22 Google Research
Neural Architecture Search for Transformers: A Survey IEEE xplore [Sep'22] Iowa State Uni

Foundation Models

Title Venue Group
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models arxiv'23 Amazon Alexa AI

Misc resources