Mappings Comparison

Context

Knews extracts its frame types starting from the output of Boxer and UKB (or alternatively Babelfy). WordNet synsets from UKB are aligned to the event predicate symbol from Boxer, then the WordNet synsets are mapped to FrameNet frames using a pre-existent mapping file. However, this mapping is still small. Recently, STLab developed new mappings within Framester and made them available.

Objectives

The main objective of this experiment is to evaluate which of these mappings offer better coverage in terms of number of frame instances, frame types, roles and elements.

Experiment

In this experiment, we used Knews to extract frames over a collection of 1653 documents using the both mapping options (Knews mappings and Framester mappings). For this, we created a new branch where is possible to select the mapping to be used just changing the configuration in this file.

Results

In Table 1 and Table 2 are showed some statistics in the identification of frame instances, frame types, roles and elements using both mappings. As we can see, Knews mappings offer a slightly better coverage in the identification of these items. The main reason why Knews has better coverage is that its mappings have more frames and synsets associated between them (see Table 3).

Table 1: Knews Mappings

Item	All values	Unique values
Instances	114517	114517
Types	97154	2274
Elements	154422	3830
Roles	154422	222

Table 2: Framester Mappings

Item	All values	Unique values
Instances	114212	114212
Types	95347	2013
Elements	154095	3824
Roles	154095	222

Table 3: Resources

Item	Knews Mappings	Framester Mappings
Frames	881	870
Synsets	9080	8169

In qualitative terms, apparently the results are equivalent. In the outputs of these mappings, frames have similar structures (elements and roles). So, talking in terms of recall/coverage and precision, apparently Knews mappings offer a slightly better coverage than Framester but their precisions are equivalents.

Output samples for three frames:

Knews

http://framebase.org/ns/frame-Commerce_buy-buy.v
http://framebase.org/ns/fe-Buyer
http://framebase.org/ns/fe-Goods

http://framebase.org/ns/frame-Attack-lay.v
http://framebase.org/ns/fe-Assailant
http://framebase.org/ns/fe-Victim

http://framebase.org/ns/frame-Being_named-move.v
http://framebase.org/ns/fe-Speaker
http://framebase.org/ns/fe-Entity

Framester

http://framebase.org/ns/frame-Commerce_buy-buy.v
http://framebase.org/ns/fe-Buyer
http://framebase.org/ns/fe-Goods

http://framebase.org/ns/frame-Attack-lay.v
http://framebase.org/ns/fe-Assailant
http://framebase.org/ns/fe-Victim

http://framebase.org/ns/frame-Being_named-move.v
http://framebase.org/ns/fe-Speaker
http://framebase.org/ns/fe-Entity