Skip to content

Latest commit

 

History

History
67 lines (50 loc) · 3.59 KB

README.md

File metadata and controls

67 lines (50 loc) · 3.59 KB

TAC corpus

The work is part of the "Networked Mathematics" project at the Topos Institute.

You can read about the project in the blog posts:

There are also the preprints:

There were some preliminary investigations in the Parmesan0.11 and Parmesan0.12 prototypes. For the unmodified results of our 2022 paper, see this branch.

Parmesan 0.2 is now up at http://www.jacobcollard.com/parmesan2/

This repository

This repository contains a corpus based on the contents of abstracts of the electronic journal Theory and Applications of Categories (TAC) as of c. December 2020. This is used as a training/testing corpus for mathematical NLP and machine learning projects.

The corpus contains the following data files:

  • tac.conll contains an automatically annotated version of the corpus, with dependency structures and POS tags.
  • tac.json contains the original corpus, in JSON format.
  • tac_metadata.json contains the original corpus, in JSON format, with additional metadata such as authors and keywords.
  • tac_stats.json contains some basic statistics about the corpus, including the frequency of common words and parts of speech.

The tac-experiments folder contains a series of simple experiments evaluating various automatic terminology extraction methods on the TAC corpus. To run the original experiments, you would need an installation of DyGIE++ and Parmenides. Unfortunately, the latter is not freely available, but you can contact the authors for distribution.

Parmesan 0.2 can be run without problems, see instructions in https://github.com/ToposInstitute/parmesan.

Corpus statistics

There are two types of part-of-speech tags in the corpus statistics, both generated by spaCy. The first tagset, labeled "pos" in nlab_stats.json, represents coarse-grained parts of speech and is taken from the Universal POS tag set. The second tagset, "tag", is specific to spaCy's pretrained English model.

Details about the different tagsets, as well as other label schemes for this model can be found on spaCy's website.