Around 260,000 threads / comments scraped from Reddit. Useful dataset for NLP projects.
Scraped using omega-red
The .csvs are named <metareddit>_<subreddit>.csv. The headers are described here and in headers.txt.
Headers are:
text,id,subreddit,meta,time,author,ups,downs,authorlinkkarma,authorkarma,authorisgold
text: Text of the comment / threadid: Unique reddit id for the comment / threadsubreddit: Subreddit that the comment / thread belongs tometa: Metareddit that the comment / thread belongs to. Subreddits belong to metareddits. A subreddit can beleagueoflegends. The metareddit for that subreddit would begaming, which can also include the subredditdota2time: UNIX timestamp of the comment / threadauthor: Username of the author of the comment / threadups: Number of upvotes the comment / thread receiveddowns: Number of downvotes the comment / thread receivedauthorlinkkarma: The author's link karma. What is Link Karma?authorkarma: The author's karma. Reddit FAQ explaining karma.authorisgold: Boolean indicator for the gold status of the user.1for gold users,0for non-gold (normal) users. Reddit FAQ explaining gold status.
If you prefer the texts with punctuation, they are included (as the original output files from omega-red) at https://mega.nz/#F!NtsCGTgD!urXdXLJ6yITYdWEdWN-H1w
are in /threads.csv at https://mega.nz/#F!NtsCGTgD!urXdXLJ6yITYdWEdWN-H1w
Headers are:
'text', 'title', 'url', 'id', 'subreddit', 'meta', 'time', 'author', 'ups', 'downs', 'authorlinkkarma', 'authorcommentkarma', 'authorisgold'
text: text of the threadtitle: title of the threadurl: url of the threadid: unique ID of the threadsubreddit: subreddit that the thread belongs tometa: meta tag assigned to the subreddit of the thread inconfig.jsontime: timestamp of the threadauthor: username of the author of the threadups: number of ups the thread has receiveddowns: number of downs the thread has receivedauthorlinkkarma: the author's link karmaauthorcommentkarma: the author's comment karmaauthorisgold:1if the author has gold status,0otherwise
are in comments.csv at https://mega.nz/#F!NtsCGTgD!urXdXLJ6yITYdWEdWN-H1w
Heades are (different from threads.csv)
'text', 'id', 'subreddit', 'meta', 'time', 'author', 'ups', 'downs', 'authorlinkkarma', 'authorcommentkarma', 'authorisgold'
text: text of the commentid: unique ID of the commentsubreddit: subreddit that the thread belongs tometa: meta tag assigned to the subreddit of the thread inconfig.jsontime: timestamp of the threadauthor: username of the author of the threadups: number of ups the thread has receiveddowns: number of downs the thread has receivedauthorlinkkarma: the author's link karmaauthorcommentkarma: the author's comment karmaauthorisgold:1if the author has gold status,0otherwise
All text is normalized to lower case, tokenized using a TreebankTokenizer from natural, then joined with spaces. This results in punctuation being separated from words, a desired effect.
The MIT License (MIT)
Copyright (c) 2016, Linan Qiu
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
