Jens Krinke [email protected]
This script generates some metrics from the reviews on EasyChair to make your life as a Chair easier. It generates the following metrics for each of your reviewers:
- Average score over all reviews of the reviewer.
- Average absolute score over all reviews of the reviewer.
- Average confidence over all reviews of the reviewer.
- Average length over all reviews of the reviewer.
Most of the above statistics should be clear. The average absolute score reveals how strong the opinion of the reviewer on average is. For example, assume a reviewer who gave scores +2 and -2 and another reviewer who gave scores +1 and -1. Both have the same normal average, 0, but the absolute averages 2 and 1 reveal that the first reviewer has stronger opinions.
The script also lists the subreviewer, score and confidence for each review for every reviewer.
To find if the PC is clustered, a graph can be generated where edges represent that the two connected reviewers did a review for the same paper.
First, you have to download all reviews. You'll find the function in EasyChair under Administration -> Other utilities -> List of reviews. Select everything to download the reviews, which will the download a file "review_list.txt".
usage: pcmetrics.py [-h] [-a A [A ...]] [-s S] [-c C [C ...]] [-f] [-g] file
Generate metrics over reviews from EasyChair.
positional arguments:
file review_list.txt file
optional arguments:
-h, --help show this help message and exit
-a A [A ...] list of names to be anonimised
-s S max. length of short reviews
-c C [C ...] list of conflicted papers
-f write metrics to csv file
-g generate a review cluster graph
The script can anonimize a list of names given via -a.
The script can also ignore a list of papers given via -c, so that conflicts can be observed.
The script produces a list of short reviews and the maximal length of a short review can be given via -s, the default is 1500.
The generated statistics can be written to a file data.csv if -f is given.
The graph of who has reviewed together with whom can be generated by specifying -g, the data is written to a file data.rsf in RSF format.