-
Notifications
You must be signed in to change notification settings - Fork 17
The Datasets API
John Gamboa edited this page Sep 12, 2017
·
11 revisions
Ovation provides classes for accessing several datasets through the same API. The dataset files are expected to be located in /data/datasets/<dataset-name>
. Each dataset is divided in three splits: train
, validation
and test
.
The table below show the datasets supported by Ovation. It may be still incomplete, and we are working on supporting more datasets in the future. The descriptions of the datasets are generally excerpts of the official websites.
Dataset | Task | Licence | Description |
---|---|---|---|
SICK | SC | (couldn't find) | |
Microsoft Paraphrase Dataset | SC | (couldn't find) | |
PPDB: The Paraphrase Database | SC | CC-BY 3.0 | |
Quora Similar questions dataset | SC | https://www.quora.com/about/tos | |
SemEval | SC | CC-BY-SA 3.0 | |
Stack Exchange | SC | CC-BY-SA 3.0 | |
Annotated Corpus for NER | NER | (couldn't find) | |
GermEval 2014 NER | NER | CC-BY 4.0 | The GermEval 2014 NER Shared Task builds on a new dataset with German Named Entity annotation with the following properties: (1) The data was sampled from German Wikipedia and News Corpora as a collection of citations. (2) The dataset covers over 31,000 sentences corresponding to over 590,000 tokens. (3)The NER annotation uses the NoSta-D guidelines, which extend the Tübingen Treebank guidelines, using four main NER categories with sub-structure, and annotating embeddings among NEs such as [ORG FC Kickers [LOC Darmstadt]]. |
GerSEN | SA | GerSEN's license (basically: the dataset private, non-commercial, and publications must cite it) | The dataset consists of 2,369 sentiment-annotated sentences. The sentences are extracted from German news articles. The news articles are related to the Berlin's universities. |
Hotel Review Dataset | SA | (couldn't find) | This dataset consists of 878561 reviews (1.3GB) from 4333 hotels crawled from TripAdvisor. |
CrowdFlower Twitter Emotion Dataset | SA | CC Public Domain | In a variation on the popular task of sentiment analysis, this dataset contains labels for the emotional content (such as happiness, sadness, and anger) of texts. Hundreds to thousands of examples across 13 labels. A subset of this data is used in an experiment we uploaded to Microsoft's Cortana Intelligence Gallery. Added: July 15, 2016 by CrowdFlower |
Amazon Review Dataset DE | SA | Private (scraped by Insiders) | Crawled data from Amazon German reviews |
Germeval 2017 SA | SA | CC-BY 4.0 | All together the data consists of 22,000 messages from various social media and web sources. |
The code below initializes an object of the type Acner
. Acner
is one of the 14 datasets supported by Ovaion. (you can get an overview of the supported datasets in the section
Supported Datasets
below.)
# Instantiates a new element of the class of the dataset
acner = Acner()
# You can use the variable `epochs_completed` for
while acner.train.epochs_completed < 10:
# You can get a new batch with `next_batch()`
train_batch = acner.train.next_batch(
# By default, the batch size is always 64
batch_size=64,
# `pad` makes sense for sequences. Here
# we pad the sequences with an invalid
# character so that all instances of the
# batch have 40 elements
pad=40,
# If `one_hot` is not True, we get only
# a sequence of numbers with the index
# of each word of the sequence in the
# vocabulary.
one_hot=True)
# do something with the batch. E.g.,
train_step(train_batch.sentences,
train_batch.pos,
train_batch.ner)