Skip to content

Home

Jump to bottom
Jaimie Murdock edited this page Dec 8, 2017 · 18 revisions

Welcome to the aws-documentation wiki!

Configuration

SEP mirror

User: ec2-user

~/bin/do_mirror

inpho

User: inphosite

Python Environment: Miniconda 2

Additional yum packages: tmux nginx redhat-lsb-core

Additional conda packages (for user inphosite): gunicorn mysql-python docutils sphinx nltk

Installation

Rough outline of install from scratch:

sudo mkdir /var/inpho
sudo chown inphosite:inpho /var/inpho

sudo su inphosite

cd /tmp
wget https://repo.continuum.io/miniconda/Miniconda2-4.3.30-Linux-x86_64.sh
bash Miniconda2-4.3.30-Linux-x86_64.sh
conda install mysql-python sphinx docutils sphinx nltk
python -m nltk.downloader all # TODO: get minimal spanning set of nltk packages, use global NLTK_DATA directory.

cd /var/inpho
git clone https://github.com/inpho/inpho.git
cd inpho
python setup.py develop

cd ..
git clone https://github.com/inpho/inphosite.git
cd inphosite
python setup.py develop

Amazon RDS

Configure database on Amazon RDS. I migrated us from MySQL to MariaDB, which is the pure open-source fork of MySQL.

Security policy has to allow access on port 3306 to your machine.

I uploaded database from my own machine using a database connetion:

mysql -h inpho.c1qnbhebkxsc.us-east-2.rds.amazonaws.com -P 3306 -u inpho -p seponto < /var/inpho/backup/sql/20171207.sql

Web server

The web server for inphosite is gunicorn + nginx.

Gunicorn is a WSGI-compliant, Python-based server optimized for high throughput. nginx is an alternative to Apache which was optimized to maximize the number of concurrent connections through an event loop. It also has simpler path routing for static resources on shared hosts with dynamic resources. The memory model is superior as it does not embed the WSGI process in the worker processes.

See also: Apache vs. nginx: Practical Considerations (Digital Ocean)

Gunicorn

Sample configuration:

sudo yum install nginx
sudo su inphosite
conda install gunicorn

Test server:

gunicorn --paste production.ini -b :5000 --chdir /var/inpho/inphosite
Service deployment

If the server is successfully started and you can test using port 5000, then we can deploy a service.

First, create /etc/init.d/inphosite:

#! /bin/bash
### BEGIN INIT INFO
# Provides:          inphosite
# Required-Start:    nginx
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: The main pylons process
# Description:       The gunicorn process that receives HTTP requests
#                    from nginx
#
### END INIT INFO
#
# Author:       Jaimie Murdock <[email protected]>
#
APPNAME=inphosite
USER=inphosite
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/home/inphosite/miniconda2/bin
CHROOT=/var/inpho/inphosite
CONFIG=production.ini
DAEMON=gunicorn
#BIND=0.0.0.0:5000
BIND=unix:/var/inpho/inphosite.sock
PIDFILE=/var/run/gunicorn.pid
LOGFILE=/var/log/$DAEMON.log
WORKERS=2


. /lib/lsb/init-functions


if [ -e "/etc/default/$APPNAME" ]
then
    . /etc/default/$APPNAME
fi


case "$1" in
  start)
        #log_daemon_msg "Starting deferred execution scheduler" "$APPNAME"
        cd $CHROOT
        $DAEMON --daemon --bind=$BIND --pid=$PIDFILE --workers=$WORKERS --user=$USER --log-file=$LOGFILE --paste $CONFIG --chdir $CHROOT
        #log_end_msg $?
    ;;
  stop)
        #log_daemon_msg "Stopping deferred execution scheduler" "APPNAME"
        killproc -p $PIDFILE $DAEMON
        #log_end_msg $?
    ;;
  force-reload|restart)
    $0 stop
    $0 start
    ;;
  status)
    #status_of_proc -p $PIDFILE $DAEMON && exit 0 || exit $?
    killall -0 $DAEMON && echo "Running $APPNAME" && exit 0 || exit $?
    ;;
  *)
    echo "Usage: /etc/init.d/$APPNAME {start|stop|restart|force-reload|status}"
    exit 1
    ;;
esac

exit 0

Then run sudo service inphosite start.

Check the status with sudo service inphosite status

Stop with sudo service inphosite stop

It may be possible to enable these services without sudo privileges, but I haven't looked into that yet for init.d system configs. TODO: Enable service permissions for all users in inphosite group.

nginx

These commands will be run as the root account.

sudo yum install nginx
sudo mkdir /etc/nginx/sites-available
sudo touch /etc/nginx/sites-available/inphosite.conf
sudo ln -s /etc/nginx/sites-available/inphosite.conf /etc/nginx/sites-enabled/inphosite.conf

Edit the /etc/nginx/nginx.conf to add sites-enabled directory:

...

http {

    ...

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*.conf;

    ...

}

Edit /etc/nginx/sites-available/inphosite.conf:

server {
    listen 80;
    server_name inphoproject.org www.inphoproject.org;

    location = /favicon.ico { access_log off; log_not_found off; }

    location / {
        include proxy_params;
        proxy_pass http://unix:/var/inpho/inphosite.sock;
    }
}

DLV Prolog

First download DLV, which is free for academic and non-commercial educational use: http://www.dlvsystem.com/dlv/

Make sure it is installed into the path. Easiest way is to just drop it in /usr/bin/ since it's a totally self-contained binary.

cd /tmp
wget http://www.dlvsystem.com/files/dlv.x86-64-linux-elf-static.bin
sudo mv dlv.x86-64-linux-elf-static.bin /usr/bin/dlv
sudo chmod 755 /usr/bin/dlv

Answer set program:

cd /var/inpho/
git clone https://github.com/inpho/answer-set.git
mkdir -p /var/inpho/backup/asp/

Cronjob

Edit /var/inpho/bin/aspcron.sh:

#!/bin/sh
DATE=$(date +\%Y\%m\%d)
LOGPATH=/var/inpho/log/asp/$DATE.asp.log
BACKUPPATH=/var/inpho/backup/asp
DLVPROGRAM=/var/inpho/answer-set/program.txt
TOINPHO=/var/inpho/answer-set/toinpho.py

python -m inpho.dlv 1> $BACKUPPATH/$DATE.expertlevels.txt 2>> $LOGPATH
dlv -stats -pfilter=ins,link $BACKUPPATH/$DATE.expertlevels.txt $DLVPROGRAM 1> $BACKUPPATH/$DATE.stratifiedtest.txt 2>> $LOGPATH
python $TOINPHO $BACKUPPATH/$DATE.stratifiedtest.txt 1>> $LOGPATH 2>> $LOGPATH

TODO: Port votingfilter to python: perl -w votingfilter3.pl DLVexpertlevels.txt votingout.txt 1>> $LOGPATH 2>> $LOGPATH

hypershelf

User: hypershelf

Python Environment: Miniconda 3

AWS Machines

Clone this wiki locally