An Information Theoretic Metric for Multi-Class Categorization

python

The implementation of the Proficiency Metric in various settings:

• predeval.py:ConfusionMX: Classification

• predeval.py:MuLabCat: Multi-Label Categorization

paper

The research paper describing the Proficiency Metric.

data

The test data for the results in the paper:

• annotated: human-labeled examples

• results: algorithmically categorized examples

• labeler[123].txt: KDD Cup 2005

• Trec_beta: Example test data for ERD 2014 - http://web-ngram.research.microsoft.com/erd2014/Datasets.aspx

• sk: 100 Magnetic queries annotated with Wikipedia concepts in similar format as ERD 2014 (Trec_beta)

• magnetic: hand-annotated examples from searches in the Magnetic.com database of search events