Social and News Media Analysis

By: Austin Bristol

This project was completed for a senior thesis project at Allegheny College. The system was written in Python 3, and uses a suite of technologies to perform sentiment analysis of social media and news data.

Table of Contents

• Overview

  • Examples
  • Technology

• Prerequisites

• Installation

Overview

This software system allows users to collect large amounts of media data from Twitter and various news sources, and analyzes this information to extract valuable insights in the sentiment that users display on the platforms. By using an easy-to-use web interface, users can set up a cluster of AWS machines to perform the analytics with a click of a button. Also, the user can easily query the system for whatever search topic that they want to perform analytics on, as well as saving previous results to be viewed later. Visualizations are produced as a result of the analytics, and can be used to gain insights on the input topic.

Examples

You can view how the system works all together by viewing the following demo video. Also, below shows various example visualizations that are produced by this software system. Visualizations include, the overall sentiment visualization, a sentiment bar graph, mood pie chart, sentiment change line graph, and emoticon table.

Technology

There were a suite of technologies used in the development of the project. Specifically, a lot of different Python packages were used to complete this project. Also, Amazon Web Services was used in great detail as well. Below is a listing of the technologies that were used in this project:

• Python 3.6

• Flask

• python-twitter

• News API

• Newspaper3k

• Natural Language Toolkit

• Matplotlib

• Paramiko

• scp

• AWS SDK for Python

• AWS EC2

• PostgreSQL

Prerequisites

In order to be able to correctly install this software system and get it up and running, there are few things you need to do. This ranges from creating accounts on different platforms, as well as set up internal files. Below is a list of tasks that must be completed before installing the system.

1. Create a developer account on Twitter. Traverse to this link and create an account. Then, you will be able to go to the apps page, and create a 'new app'. Here you can create new application by following the prompts. Finally, once the app is created, you will need to save all the API keys and tokens found in the 'Keys and tokens' tab.

2. Create an account on the newsapi website. Once you create a free account, you will need to save the API key found in the account page.

3. Create an account on AWS. This will make you set up your account with a credit card and other information.

4. Set up your credentials file on you local machine. In order to use any AWS APIs, you will need to set up a credentials file on your machine. First, create a new set of keys by going to the credentials page of the AWS console, and traversing to the 'Access keys' tab. Next, you will need to place the content in a credentials file located at ~/.aws/credentials on Linux, macOS, or Unix and C:\Users\USERNAME.aws\credentials on Windows. The contents of the credentials file should appear in the following format:

   [default]
   aws_access_key_id = your_access_key_id
   aws_secret_access_key = your_secret_access_key
   

   For more information on how this works, please check out the following web page: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

5. In the AWS console create a new security group. Add two inbound rules: One which is the 'PostgreSQL' type and the source is from anywhere. Another which is the 'SSH' type and the source is also from anywhere. After this security group is created, make sure you save the 'Group ID' of this security group.

6. You must have Python 3 and Pip installed on your machine.

Installation

The final step of getting the system up and running is to actual install everything. You will need to not only install the code but the dependancies as well. Below is a list of steps to set up and run the code.

1. Open a terminal window and traverse to the directory you would like to install the code.

2. Run git clone [email protected]:bristola/social_news_analysis.git

3. Run pip3 install -r requirements.txt on Linux or pip install -r requirements.txt on Windows

4. Traverse to the correct directory by typing: cd src and cd project

5. Create a config.txt file in the project directory with the following format:

   Twitter API key = <Your API key>
   Twitter API secret key = <Your API secret>
   Twitter Access token = <Your token>
   Twitter Access token secret = <Your token secret>
   News API key = <Your API key>
   AWS Key Name = <Your AWS PEM name>
   AWS Machine Type = t2.micro
   AWS Security Group ID = <Your AWS security ID>
   AWS Image ID = <Your Image ID>
   Path to pem = <Your absolute path to PEM file>
   

6. Execute the code using python3 app.py on Linux or py app.py on Windows. Python command can change based on how it is installed on the machine.

7. Finally, after executing the code, you can traverse to http://127.0.0.1:5000 to start using the system.

Testing

You can run a Python test suite by traversing to the root of the project and running the following command:

pytest