elasticsearch-analysis-annotation

Analysis plugin for ElasticSearch providing capability for processing inline annotations in documents.

Description

Inline annotations are considered to be simple semantic informations inlined in source text, that are removed from the indexed text and injected as synonyms at positions of the words they are related to.

This plugin provides analyzer AnnotationAnalyzer as well as filter InlineAnnotationFilter. AnnotationAnalyzer is composed of WhitespaceTokenizer, LowerCaseFilter and InlineAnnotationFilter (with default settings). More sophisticated analyzers (equivalent to StandardAnalyzer or SnowballAnalyzer) can be configure via configuration file elasticsearch.yml or web API.

Example

Let's say we have this documents

"Mozart[artist] was born[lifeEvent] in Salzburg[city;Austria]"

If we parse this with StandardAnalyzer equivalent with annotation analysis added to it we get these tokens - some are omitted due to used StopFilter.

                           | [austria]
[artist] | | [lifeevent] | |  [city]
 mozart  | |    born     | | salzburg

If we use StandardAnalyzer the result would be

mozart | artist | | born | lifeevent | | salzburg | city | austria

Installation

This plugin follows conventions for elasticsearch plugins, thus can be installed in a standard manner - see http://www.elasticsearch.org/guide/reference/modules/plugins/

Using this plugin

To use those custom analyzers/filters you need to modify elasticsearch.yml configuration file - see http://www.elasticsearch.org/guide/reference/index-modules/analysis/

The following example configuration contains definitions for analyzers based on behaviour of StandardAnalyzer and SnowballAnalyzer.

Please note that standard_annotation and snowball_annotation analyzers use standard tokenizer, which removes all non-alphanumeric characters and thus makes it impossible to process inline annotations marked with [,],; (which are used in default behaviour of InlineAnnotationFilter).

For this purpose we need to use mapping char filter, which remaps those special characters to their equivalent, which will be accepted by standard tokenizer as part of the token.

 index :
     analysis :
         char_filter : 
             annotation_remap : 
                 type : mapping
                 mappings : ["[=>__annotation_start__", "]=>__annotation_end__",";=>__annotation_delimiter__"] #substituované řetězce
         analyzer :                
             standard_annotation :
                 type : custom
                 tokenizer : standard
                 char_filter : annotation_remap
                 filter : [standard, lowercase, annotation_filter, stop] #sada filtrů používaná StandardAnalyzerem
             snowball_annotation :
                 type : custom
                 tokenizer : standard
                 char_filter : annotation_remap
                 filter : [standard, lowercase, annotation_filter, stop, snowball] #sada filtrů používaná SnowballAnalyzerem
         filter :
             annotation_filter :
                 type : annotation_filter
                 start : __annotation_start__
                 end : __annotation_end__
                 delimiter : __annotation_delimiter__

To test the analyzer you can query the following http://localhost:9200/test/_analyze?analyzer=annotation&text="Mozart[city;Salzburg]"

Customization

The InlineAnnotationFilter can be slightly customized.

List of supported options

• start - start delimiter for inline annotation
• end - end delimiter for inline annotation
• prefix - string to be prepended to synonym, that is created from inline annotation
• suffix - string to be apended to synonym, that is created from inline annotation
• token-type - token type of synonym
• delimiter - delimiter for multiple inline annotations

Example providing default values

index :
    analysis :
        analyzer :                
            annotation :
                type : annotation
                start : [
                end : ]
                prefix : [
                suffix : ]
                token-type: synonym
                delimiter : ;

Elasticsearch version

This plugin was successfuly tested on elasticsearch version 0.90.2