The Datasets API

The Dataset Classes

Ovation provides classes for accessing several datasets through the same API. The dataset files are expected to be located in /data/datasets/<dataset-name>. Each dataset is divided in three splits: train, validation and test.

Supported Datasets

The table below show the datasets supported by Ovation. It may be still incomplete, and we are working on supporting more datasets in the future. The descriptions of the datasets are generally excerpts of the official websites.

Dataset	Task	Licence	Description
SICK	SC	(couldn't find)
Microsoft Paraphrase Dataset	SC	(couldn't find)
PPDB: The Paraphrase Database	SC	CC-BY 3.0
Quora Similar questions dataset	SC	https://www.quora.com/about/tos
SemEval	SC	CC-BY-SA 3.0
Stack Exchange	SC	CC-BY-SA 3.0
Annotated Corpus for NER	NER	(couldn't find)
GermEval 2014 NER	NER	CC-BY 4.0	The GermEval 2014 NER Shared Task builds on a new dataset with German Named Entity annotation with the following properties: (1) The data was sampled from German Wikipedia and News Corpora as a collection of citations. (2) The dataset covers over 31,000 sentences corresponding to over 590,000 tokens. (3)The NER annotation uses the NoSta-D guidelines, which extend the Tübingen Treebank guidelines, using four main NER categories with sub-structure, and annotating embeddings among NEs such as [ORG FC Kickers [LOC Darmstadt]].
GerSEN	SA	GerSEN's license (basically: the dataset private, non-commercial, and publications must cite it)	The dataset consists of 2,369 sentiment-annotated sentences. The sentences are extracted from German news articles. The news articles are related to the Berlin's universities.
Hotel Review Dataset	SA	(couldn't find)	This dataset consists of 878561 reviews (1.3GB) from 4333 hotels crawled from TripAdvisor.
CrowdFlower Twitter Emotion Dataset	SA	CC Public Domain	In a variation on the popular task of sentiment analysis, this dataset contains labels for the emotional content (such as happiness, sadness, and anger) of texts. Hundreds to thousands of examples across 13 labels. A subset of this data is used in an experiment we uploaded to Microsoft's Cortana Intelligence Gallery. Added: July 15, 2016 by CrowdFlower
Amazon Review Dataset DE	SA	Private (scraped by Insiders)	Crawled data from Amazon German reviews
Germeval 2017 SA	SA	CC-BY 4.0	All together the data consists of 22,000 messages from various social media and web sources.

Examples

Getting a batch

The code below initializes an object of the type Acner. Acner is one of the 14 datasets supported by Ovaion. (you can get an overview of the supported datasets in the section Supported Datasets below.)

# Instantiates a new element of the class of the dataset
acner = Acner()

# You can use the variable `epochs_completed` for 
while acner.train.epochs_completed < 10:

    # You can get a new batch with `next_batch()`
    train_batch = acner.train.next_batch(
                             # By default, the batch size is always 64
                             batch_size=64,
                             # `pad` makes sense for sequences. Here
                             # we pad the sequences with an invalid
                             # character so that all instances of the
                             # batch have 40 elements
                             pad=40,
                             # If `one_hot` is not True, we get only
                             # a sequence of numbers with the index
                             # of each word of the sequence in the
                             # vocabulary.
                             one_hot=True)

    # do something with the batch. E.g.,
    train_step(train_batch.sentences,
               train_batch.pos,
               train_batch.ner)