Skip to content

Latest commit

 

History

History
133 lines (109 loc) · 11.7 KB

README.md

File metadata and controls

133 lines (109 loc) · 11.7 KB
Raw

LLaMA-Paper-List

Collection of papers using LLaMA as backbone model.

Contributors

Table of Contents

Papers

Original LLaMA paper

  • LLaMA: Open and Efficient Foundation Language Models. arxiv 2023. paper. code
    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
  • Llama 2: Open Foundation and Fine-Tuned Chat Models. Meta AI 2023. paper. code
    Hugo Touvron, Louis Martin, Kevin Stone et al.
  • The Llama 3 Herd of Models. arxiv 2024. paper. code
    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey et al.

Related theory with LLaMA

  • Large Language Models Are Zero-Shot Time Series Forecasters. NeurIPS 2023. paper. code
    Nate Gruver, Marc Finzi, Shikai Qiu, Andrew Gordon Wilson
  • Training Compute-Optimal Large Language Models. NeurIPS 2022. paper.
    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
  • Root Mean Square Layer Normalization. NeurIPS 2019. paper. code
    Biao Zhang, Rico Sennrich
  • GLU Variants Improve Transformer. arxiv 2020. paper. code
    Noam Shazeer
  • RoFormer: Enhanced Transformer with Rotary Position Embedding. arxiv 2021. paper. code
    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu
  • Decoupled Weight Decay Regularization. ICLR 2019. paper. code
    Ilya Loshchilov, Frank Hutter
  • Self-attention Does Not Need $O(n^2)$ Memory. arxiv 2021. paper. code
    Markus N. Rabe and Charles Staats
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arxiv 2022. paper. code
    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
  • Reducing Activation Recomputation in Large Transformer Models. arxiv 2022. paper.
    Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro

LLaMA with parameter efficiency

  • LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arxiv 2023. paper. code
    Zhang, Renrui and Han, Jiaming and Zhou, Aojun and Hu, Xiangfei and Yan, Shilin and Lu, Pan and Li, Hongsheng and Gao, Peng and Qiao, Yu
  • LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model. arxiv 2023. paper. code
    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, Yu Qiao
  • LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. arxiv 2023. paper.
    Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, Xing Xu, Soujanya Poria
  • A Simple and Effective Pruning Approach for Large Language Models. arxiv 2023. paper. code
    Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter
  • LLM-Pruner: On the Structural Pruning of Large Language Models. arxiv 2023. paper. code
    Xinyin Ma, Gongfan Fang, Xinchao Wang

Fine-tune LLaMA on downstream tasks

  • Graph of Thoughts: Solving Elaborate Problems with Large Language Models. AAAI 2024. paper. code.
    Maciej Besta, Nils Blach, Ales Kubicek et al.
  • How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. NeurIPS 2023. paper. code
    Yizhong Wang, Hamish Ivison, Pradeep Dasigi et al.
  • Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision. NeurIPS 2023. paper. code
    Zhiqing Sun, Yikang Shen, Qinhong Zhou et al.
  • ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge. arxiv 2023. paper.
    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, You Zhang
  • Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca. arxiv 2023. paper. code
    Yiming Cui, Ziqing Yang, Xin Yao
  • PMC-LLaMA: Further Finetuning LLaMA on Medical Papers. arxiv 2023. paper.
    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie
  • Dr. LLaMA: Improving Small Language Models on PubMedQA via Generative Data Augmentation. arxiv 2023. paper.
    Zhen Guo, Peiqi Wang, Yanwei Wang, Shangdi Yu
  • Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks. arxiv 2023. paper.
    Tiedong Liu, Bryan Kian Hsiang Low
  • WizardLM: Empowering Large Language Models to Follow Complex Instructions. arxiv 2023. paper. code
    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Daxin Jiang
  • Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. arxiv 2023. paper. code
    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, BoWen Zhou
  • LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction. arxiv 2023. paper. code
    Abdullatif Köksal, Timo Schick, Anna Korhonen, Hinrich Schütze
  • In-Context Learning User Simulators for Task-Oriented Dialog Systems. arxiv 2023. paper. code
    Silvia Terragni, Modestas Filipavicius, Nghia Khau, Bruna Guedes, André Manso, Roland Mathis
  • NetGPT: A Native-AI Network Architecture Beyond Provisioning Personalized Generative Services. arxiv 2023. paper. code
    Yuxuan Chen, Rongpeng Li, Zhifeng Zhao, Chenghui Peng, Jianjun Wu, Ekram Hossain, Honggang Zhang
  • On decoder-only architecture for speech-to-text and large language model integration. arxiv 2023. paper. code
    Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, Yu Wu

LLaMA combined with multi-modal

  • MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024. paper. code
    Xiang Yue, Yuansheng Ni, Kai Zhang et al.

LLaMA with retrieval

  • Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge Retrieval from Foundation Language Models. arxiv 2023. paper. code
    Tim Schott, Daniel Furman, Shreshta Bhat
  • ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models paper. code
    Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, Dongkuan Xu
  • Landmark Attention: Random-Access Infinite Context Length for Transformers. arxiv 2023. paper. code
    Amirkeivan Mohtashami, Martin Jaggi

LLaMA using reinforcement learning

  • LIMA: Less Is More for Alignment. arxiv 2023. paper. code
    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy
  • RRHF: Rank Responses to Align Language Models with Human Feedback without tears. paper. code
    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang

Quantitative analysis of LLaMA

  • SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression. arxiv 2023. paper. code
    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, Dan Alistarh
  • SqueezeLLM: Dense-and-Sparse Quantization. arxiv 2023. paper. code
    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer

Prompting LLaMA

  • Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition.. arxiv 2023. paper.
    Yuang Li, Yu Wu, Jinyu Li, Shujie Liu

How to contribute

Contributions are welcome! Please refer to CONTRIBUTING.md for contribution guidelines.