Awesome-LLM4IE-Papers

Awesome papers about generative Information extraction using LLMs

The organization of papers is discussed in our survey: Large Language Models for Generative Information Extraction: A Survey.

If you have any suggestions or come across any mistakes, missing information, please feel free to let us know via email at [email protected]. We appreciate your feedback and help in improving our work.

If you find our survey useful for your research, please cite the following paper:

@misc{xu2023large,
    title={Large Language Models for Generative Information Extraction: A Survey}, 
    author={Derong Xu and Wei Chen and Wenjun Peng and Chao Zhang and Tong Xu and Xiangyu Zhao and Xian Wu and Yefeng Zheng and Enhong Chen},
    year={2023},
    eprint={2312.17617},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Table of Contents

• Information Extraction tasks
  • Named Entity Recognition
  • Relation Extraction
  • Event Extraction
  • Universal Information Extraction
• Learning Paradigms
  • Supervised Fine-tuning
  • Few-shot
  • Zero-shot
  • Data Augmentation
• Specific Domain
• Evaluation and Analysis
• Datasets

Information Extraction tasks

A taxonomy by various tasks.

Named Entity Recognition

Models targeting only ner tasks.

Entity Typing

Paper	Venue	Date	Code
Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing	EMNLP Findings	2023-12	Github
Generative Entity Typing with Curriculum Learning	EMNLP	2022-12	Github

Entity Identification & Typing

Paper	Venue	Date	Code
2INER: Instructive and In-Context Learning on Few-Shot Named Entity Recognition	EMNLP Findings	2023-12
In-context Learning for Few-shot Multimodal Named Entity Recognition	EMNLP Findings	2023-12
Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!	EMNLP Findings	2023-12	Github
Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset	EMNLP	2023-12	Github
LLMaAA: Making Large Language Models as Active Annotators	EMNLP Findings	2023-12	Github
Prompting ChatGPT in MNER: Enhanced Multimodal Named  Entity Recognition with Auxiliary Refined Knowledge	EMNLP Findings	2023-12	Github
Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models	Arxiv	2023-11
GPT-NER: Named Entity Recognition via Large Language Models	Arxiv	2023-10	Github
Prompt-NER: Zero-shot Named Entity Recognition in Astronomy Literature via Large Language Models	Arxiv	2023-10
Inspire the Large Language Model by External Knowledge on BioMedical Named Entity Recognition	Arxiv	2023-09
One Model for All Domains: Collaborative Domain-Prefx Tuning for Cross-Domain NER	IJCAI	2023-09	Github
Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction	Arxiv	2023-08
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition	Arxiv	2023-08	Github
Debiasing Generative Named Entity Recognition by Calibrating Sequence Likelihood	ACL Short	2023-07
Entity-to-Text based Data Augmentation for various Named Entity Recognition Tasks	ACL Findings	2023-07
Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction	BioNLP	2023-07	Github
PromptNER : Prompting For Named Entity Recognition	Arxiv	2023-06	Github
Does Synthetic Data Generation of LLMs Help Clinical Text Mining?	Arxiv	2023-04
Structured information extraction from complex scientific text with fine-tuned large language models	Arxiv	2022-12	Demo
LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting	COLING	2022-10	Github
De-bias for generative extraction in unified NER task	ACL	2022-05
InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER	Arxiv	2022-03
Document-level Entity-based Extraction as Template Generation	EMNLP	2021-11	Github
A Unified Generative Framework for Various NER Subtasks	ACL	2021-08	Github
Template-Based Named Entity Recognition Using BART	ACL Findings	2021-08	Github

Relation Extraction

Models targeting only RE tasks.

Relation Classification

Paper	Venue	Date	Code
GPT-RE: In-context Learning for Relation Extraction using Large Language Models	EMNLP	2023-12	Github
Guideline Learning for In-context Information Extraction	EMNLP	2023-12
Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!	EMNLP Findings	2023-12	Github
LLMaAA: Making Large Language Models as Active Annotators	EMNLP Findings	2023-12	Github
Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs	EMNLP	2023-12	Github
Revisiting Large Language Models as Zero-shot Relation Extractors	EMNLP Findings	2023-12
Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors	ACL Findings	2023-07	Github
How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?	ACL Workshop	2023-07	Github
STAR: Improving Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models	Arxiv	2023-05
Does Synthetic Data Generation of LLMs Help Clinical Text Mining?	Arxiv	2023-04
REBEL: Relation Extraction By End-to-end Language generation	EMNLP Findings	2021-11	Github

Relation Triplet

Paper	Venue	Date	Code
Document-level Entity-based Extraction as Template Generation	EMNLP	2021-11	Github

Relation Strict

Paper	Venue	Date	Code
REBEL: Relation Extraction By End-to-end Language generation	EMNLP Findings	2021-11	Github

Event Extraction

Models targeting only EE tasks.

Event Detection

Paper	Venue	Date	Code
Unleash GPT-2 Power for Event Detection	ACL	2021-08

Event Argument Extraction

Paper	Venue	Date	Code
Context-Aware Prompt for Generation-based Event Argument Extraction with Diffusion Models	CIKM	2023-10
Code4Struct: Code Generation for Few-Shot Event Structure Prediction	ACL	2023-07	Github
Event Extraction as Question Generation and Answering	ACL short	2023-07	Github
Prompt for extraction? PAIE: prompting argument interaction for event argument extraction	ACL	2022-05	Github

Event Detection & Argument Extraction

Paper	Venue	Date	Code
Guideline Learning for In-context Information Extraction	EMNLP	2023-12
DemoSG: Demonstration-enhanced Schema-guided Generation for Low-resource Event Extraction	EMNLP Findings	2023-12	Github
Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!	EMNLP Findings	2023-12	Github
DICE: Data-Efficient Clinical Event Extraction with Generative Models	ACL	2023-07	Github
A Monte Carlo Language Model Pipeline for Zero-Shot Sociopolitical Event Extraction	Arxiv	2023-05
STAR: Improving Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models	Arxiv	2023-05
DEGREE: A Data-Efficient Generative Event Extraction Model	NAACL	2022-07	Github
ClarET: Pre-training a correlation-aware context-to-event transformer for event-centric generation and classification	ACL	2022-05	Github
Dynamic prefix-tuning for generative template-based event extraction	ACL	2022-05
Text2event: Controllable sequence-to- structure generation for end-to-end event extraction	ACL	2021-08	Github
Document-level event argument extraction by conditional generation	NAACL	2021-06	Github

Universal Information Extraction

Unified models targeting multiple IE tasks.

NL-LLMs based

Paper	Venue	Date	Code
Set Learning for Generative Information Extraction	EMNLP	2023-12
GIELLM: Japanese General Information Extraction Large Language Model Utilizing Mutual Reinforcement Effect	Arxiv	2023-11
InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction	Arxiv	2023-04	Github
Zero-Shot Information Extraction via Chatting with ChatGPT	Arxiv	2023-02	Github
GenIE: Generative Information Extraction	NAACL	2022-07	Github
DEEPSTRUCT: Pretraining of Language Models for Structure Prediction	ACL Findings	2022-05	Github
Unified Structure Generation for Universal Information Extraction	ACL	2022-05	Github
Structured prediction as translation between augmented natural languages	ICLR	2021-01	Github

Code-LLMs based

Paper	Venue	Date	Code
GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction	Arxiv	2023-12	Github
Retrieval-Augmented Code Generation for Universal Information Extraction	Arxiv	2023-11
CODEIE: Large Code Generation Models are Better Few-Shot Information Extractors	ACL	2023-07	Github
CodeKGC: Code Language Model for Generative Knowledge Graph Construction	Arxiv	2023-04	Github

Learning Paradigms

A taxonomy by Learning Paradigms.

Supervised Fine-tuning

Paper	Venue	Date	Code
GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction	Arxiv	2023-12	Github
Set Learning for Generative Information Extraction	EMNLP	2023-12
DemoSG: Demonstration-enhanced Schema-guided Generation for Low-resource Event Extraction	EMNLP Findings	2023-12	Github
Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing	EMNLP Findings	2023-12
GIELLM: Japanese General Information Extraction Large Language Model Utilizing Mutual Reinforcement Effect	Arxiv	2023-11
Context-Aware Prompt for Generation-based Event Argument Extraction with Diffusion Models	CIKM	2023-10
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition	Arxiv	2023-08	Github
Debiasing Generative Named Entity Recognition by Calibrating Sequence Likelihood	ACL short	2023-07
DICE: Data-Efficient Clinical Event Extraction with Generative Models	ACL	2023-07	Github
Event Extraction as Question Generation and Answering	ACL short	2023-07	Github
InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction	Arxiv	2023-04	Github
Structured information extraction from complex scientific text with fine-tuned large language models	Arxiv	2022-12	Demo
Generative Entity Typing with Curriculum Learning	EMNLP	2022-12	Github
LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model	NeurIPS	2022-10	Github
LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting	COLING	2022-10	Github
GenIE: Generative Information Extraction	NAACL	2022-07	Github
DEGREE: A Data-Efficient Generative Event Extraction Model	NAACL	2022-07	Github
ClarET: Pre-training a correlation-aware context-to-event transformer for event-centric generation and classification	ACL	2022-05	Github
DEEPSTRUCT: Pretraining of Language Models for Structure Prediction	ACL Findings	2022-05	Github
Dynamic prefix-tuning for generative template-based event extraction	ACL	2022-05
Prompt for extraction? PAIE: prompting argument interaction for event argument extraction	ACL	2022-05	Github
Unified Structure Generation for Universal Information Extraction	ACL	2022-05	Github
De-bias for generative extraction in unified NER task	ACL	2022-05
Document-level Entity-based Extraction as Template Generation	EMNLP	2021-11	Github
REBEL: Relation Extraction By End-to-end Language generation	EMNLP Findings	2021-11	Github
A Unified Generative Framework for Various NER Subtasks	ACL	2021-08	Github
Template-Based Named Entity Recognition Using BART	ACL Findings	2021-08	Github
Text2event: Controllable sequence-to- structure generation for end-to-end event extraction	ACL	2021-08	Github
Document-level event argument extraction by conditional generation	NAACL	2021-06	Github
Structured prediction as translation between augmented natural languages	ICLR	2021-01	Github

Few-shot

Few-shot Fine-tuning

Paper	Venue	Date	Code
DemoSG: Demonstration-enhanced Schema-guided Generation for Low-resource Event Extraction	EMNLP Findings	2023-12	Github
One Model for All Domains: Collaborative Domain-Prefx Tuning for Cross-Domain NER	IJCAI	2023-09	Github
LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting	COLING	2022-10	Github
Unified Structure Generation for Universal Information Extraction	ACL	2022-05	Github
InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER	Arxiv	2022-03
Template-Based Named Entity Recognition Using BART	ACL Findings	2021-08	Github
Structured prediction as translation between augmented natural languages	ICLR	2021-01	Github

In-Context Learning

Paper	Venue	Date	Code
GPT-RE: In-context Learning for Relation Extraction using Large Language Models	EMNLP	2023-12	Github
Guideline Learning for In-context Information Extraction	EMNLP	2023-12
Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!	EMNLP Findings	2023-12	Github
Retrieval-Augmented Code Generation for Universal Information Extraction	Arxiv	2023-11
Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models	Arxiv	2023-11
GPT-NER: Named Entity Recognition via Large Language Models	Arxiv	2023-10	Github
Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors	ACL Findings	2023-07	Github
Code4Struct: Code Generation for Few-Shot Event Structure Prediction	ACL	2023-07	Github
CODEIE: Large Code Generation Models are Better Few-Shot Information Extractors	ACL	2023-07	Github
How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?	ACL Workshop	2023-07	Github
PromptNER : Prompting For Named Entity Recognition	Arxiv	2023-06	Github
CodeKGC: Code Language Model for Generative Knowledge Graph Construction	Arxiv	2023-04	Github

Zero-shot

Zero-shot Prompting

Paper	Venue	Date	Code
Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs	EMNLP	2023-12	Github
Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models	Arxiv	2023-11
Prompt-NER: Zero-shot Named Entity Recognition in Astronomy Literature via Large Language Models	Arxiv	2023-10
Revisiting Large Language Models as Zero-shot Relation Extractors	EMNLP Findings	2023-10
Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors	ACL Findings	2023-07	Github
Code4Struct: Code Generation for Few-Shot Event Structure Prediction	ACL	2023-07	Github
A Monte Carlo Language Model Pipeline for Zero-Shot Sociopolitical Event Extraction	Arxiv	2023-05
CodeKGC: Code Language Model for Generative Knowledge Graph Construction	Arxiv	2023-04	Github
Zero-Shot Information Extraction via Chatting with ChatGPT	Arxiv	2023-02	Github

Cross-Domain Learning

Paper	Venue	Date	Code
GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction	Arxiv	2023-12	Github
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition	Arxiv	2023-08	Github
InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction	Arxiv	2023-04	Github
DEEPSTRUCT: Pretraining of Language Models for Structure Prediction	ACL Findings	2022-05	Github
Multilingual generative language models for zero-shot cross-lingual event argument extraction	ACL	2022-05	Github

Cross-Type Learning

Paper	Venue	Date	Code
Document-level event argument extraction by conditional generation	NAACL	2021-06	Github

Data Augmentation

Data Annotation

Paper	Venue	Date	Code
LLMaAA: Making Large Language Models as Active Annotators	EMNLP Findings	2023-12	Github
Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs	EMNLP	2023-12	Github
Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models	Arxiv	2023-11	Github
How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?	ACL Workshop	2023-07	Github
Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction	bioNLP Workshop	2023-07	Github
Does Synthetic Data Generation of LLMs Help Clinical Text Mining?	Arxiv	2023-04
Unleash GPT-2 Power for Event Detection	ACL	2021-08

Knowledge Retrieval

Paper	Venue	Date	Code
Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset	EMNLP	2023-12	Github
Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary  Refined Knowledge	EMNLP Findings	2023-12	Github
Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction	Arxiv	2023-08

Inverse Generation

Paper	Venue	Date	Code
Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction	EMNLP	2023-12	Github
Entity-to-Text based Data Augmentation for various Named Entity Recognition Tasks	ACL Findings	2023-07
Event Extraction as Question Generation and Answering	ACL Short	2023-07	Github
STAR: Improving Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models	Arxiv	2023-05

Specific Domain

Paper	Domain	Venue	Date	Code
Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with  Auxiliary Refined Knowledge	Multimodal	ENMLP Findings	2023-12	Github
In-context Learning for Few-shot Multimodal Named Entity Recognition	Multimodal	ENMLP Findings	2023-12
PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literature	Polymer Material	Arxiv	2023-11	Github
Prompt-NER: Zero-shot Named Entity Recognition in Astronomy Literature via Large Language Models	Astronomical	Arxiv	2023-10
Inspire the Large Language Model by External Knowledge on BioMedical Named Entity Recognition	Biomedical	Arxiv	2023-09
Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction	Multimodal	Arxiv	2023-08
DICE: Data-Efficient Clinical Event Extraction with Generative Models	Clinical	ACL	2023-07	Github
How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain	Medical	Arxiv	2023-07	Github
Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction	Multilingual / Clinical	BioNLP	2023-07	Github
Does Synthetic Data Generation of LLMs Help Clinical Text Mining?	Clinical	Arxiv	2023-04
Yes but.. Can ChatGPT Identify Entities in Historical Documents	Historical	JCDL	2023-03
Zero-shot Clinical Entity Recognition using ChatGPT	Clinical	Arxiv	2023-03
Structured information extraction from complex scientific text with fine-tuned large language models	Scientific	Arxiv	2022-12	Demo
Multilingual generative language models for zero-shot cross-lingual event argument extraction	Multilingual	ACL	2022-05	Github

Evaluation and Analysis

Paper	Venue	Date	Code
Empirical Study of Zero-Shot NER with ChatGPT	EMNLP	2023-12	Github
NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval	EMNLP Findings	2023-12	Github
Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction	EMNLP	2023-12	Github
PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literature	Arxiv	2023-11	Github
XNLP: An Interactive Demonstration System for Universal Structured NLP	Arxiv	2023-08	Demo
A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks	Arxiv	2023-07
How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain	Arxiv	2023-07	Github
Revisiting Relation Extraction in the era of Large Language Models	ACL	2023-07	Github
Zero-shot Temporal Relation Extraction with ChatGPT	BioNLP	2023-07
InstructIE: A Chinese Instruction-based Information Extraction Dataset	Arxiv	2023-05	Github
Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors	Arxiv	2023-05	Github
Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness	Arxiv	2023-04	Github
Exploring the Feasibility of ChatGPT for Event Extraction	Arxiv	2023-03
Yes but.. Can ChatGPT Identify Entities in Historical Documents	JCDL	2023-03
Zero-shot Clinical Entity Recognition using ChatGPT	Arxiv	2023-03
Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again	EMNLP Findings	2022-12	Github
Large Language Models are Few-Shot Clinical Information Extractors	EMNLP	2022-12	Huggingface

Datasets

* denotes the dataset is multimodal. # refers to the number of categories or sentences.
The link will be updated soon~

Task	Dataset	Domain	#Class	#Train	#Val	#Test	Link
NER	ACE04	News	7	6202	745	812	Link
	ACE05	News	7	7299	971	1060	Link
	BC5CDR	Biomedical	2	4560	4581	4797	Link
	Broad Twitter Corpus	Social Media	3	6338	1001	2000	Link
	CADEC	Biomedical	1	5340	1097	1160	Link
	CoNLL03	News	4	14041	3250	3453	Link
	CoNLLpp	News	4	14041	3250	3453	Link
	CrossNER-AI	Artificial Intelligence	14	100	350	431	Link
	CrossNER-Literature	Literary	12	100	400	416
	CrossNER-Music	Musical	13	100	380	465
	CrossNER-Politics	Political	9	199	540	650
	CrossNER-Science	Scientific	17	200	450	543
	FabNER	Scientific	12	9435	2182	2064	Link
	Few-NERD	General	66	131767	18824	37468	Link
	FindVehicle	Traffic	21	21565	20777	20777	Link
	GENIA	Biomedical	5	15023	1669	1854	Link
	HarveyNER	Social Media	4	3967	1301	1303	Link
	MIT-Movie	Social Media	12	9774	2442	2442	Link
	MIT-Restaurant	Social Media	8	7659	1520	1520	Link
	MultiNERD	Wikipedia	16	134144	10000	10000	Link
	NCBI	Biomedical	4	5432	923	940	Link
	OntoNotes 5.0	General	18	59924	8528	8262	Link
	ShARe13	Biomedical	1	8508	12050	9009	Link
	ShARe14	Biomedical	1	17404	1360	15850	Link
	SNAP*	Social Media	4	4290	1432	1459	Link
	Temporal Twitter Corpus (TTC)	Social Meida	3	10000	500	1500	Link
	Tweebank-NER	Social Media	4	1639	710	1201	Link
	Twitter2015*	Social Media	4	4000	1000	3357	Link
	Twitter2017*	Social Media	4	3373	723	723	Link
	TwitterNER7	Social Media	7	7111	886	576	Link
	WikiDiverse*	News	13	6312	755	757	Link
	WNUT2017	Social Media	6	3394	1009	1287	Link
RE	ACE05	News	7	10051	2420	2050	Link
	ADE	Biomedical	1	3417	427	428	Link
	CoNLL04	News	5	922	231	288	Link
	DocRED	Wikipedia	96	3008	300	700	Link
	MNRE*	Social Media	23	12247	1624	1614	Link
	NYT	News	24	56196	5000	5000	Link
	Re-TACRED	News	40	58465	19584	13418	Link
	SciERC	Scientific	7	1366	187	397	Link
	SemEval2010	General	19	6507	1493	2717	Link
	TACRED	News	42	68124	22631	15509	Link
	TACREV	News	42	68124	22631	15509	Link
EE	ACE05	News	33/22	17172	923	832	Link
	CASIE	Cybersecurity	5/26	11189	1778	3208	Link
	GENIA11	Biomedical	9/11	8730	1091	1092	Link
	GENIA13	Biomedical	13/7	4000	500	500	Link
	PHEE	Biomedical	2/16	2898	961	968	Link
	RAMS	News	139/65	7329	924	871	Link
	WikiEvents	Wikipedia	50/59	5262	378	492	Link