Inter-/Intra-Observer Agreement Report

[mzur]

With the annotation sessions implemented, we can offer a report that computes the inter-/intra-observer agreement for each user/label. These reports are only available on a transect level since they require the selection of annotation sessions.

Intra observer agreement report:
The user who requests the report selects two annotation sessions and a user who participated in both annotation sessions. The report then contains one entry for each label that the user has used for an annotation during one of the annotation sessions. Each entry has the following information:

• Label ID
• Label name
• Annotation count (session 1)
• Annotation count (session 2)
• Agreement (0 = no annotation matches another annotation, 1 = all annotations match another annotation)

An annotation "matches" another annotation if:

• They belong to different annotation sessions.
• They have the same label attached to them.
• They are associated by a certain proximity/equality measure.

Inter observer agreement report:
The user who requests the report selects an annotation session and multiple users who participated in the annotation session (default are all).The report then contains one entry for each label that any user used for an annotation during the annotation session. An entry looks like this:

• Label ID
• Label name
• Total annotation count
• Agreement (0 = no annotation matches another annotation, 1 = all annotations match annotations of all other users)

An annotation "matches" another annotation if:

• They belong to different users.
• They have the same label attached to them.
• They are associated by a certain proximity/equality measure.

The proximity/equality measure that is required for the "matching" of annotations can be implemented in association with #20.