mtg

mtg card price visualization project

apache notes

run server undaemonized:
sudo /home/mig/anaconda3/envs/mtg/bin/mod_wsgi-express start-server ~/mtg/app/app/app.wsgi --port=80 --user www-data --group www-data --log-to-terminal

run server daemonized: sudo /home/mig/anaconda3/envs/mtg/bin/mod_wsgi-express setup-server ~/mtg/app/app/app.wsgi --port=80 --user www-data --group www-data --server-root=/etc/mod_wsgi-express-80 --log-to-terminal

then to start daemon:
sudo /etc/mod_wsgi-express-80/apachectl start

then stop daemon:
sudo /etc/mod_wsgi-express-80/apachectl stop

ERRORS:
(to view errors, stop daemon server and start an undaemonized one)

ModuleNotFoundError: No module named 'app':
- check if apahe has permissions to read and execute files in the project folder
- to check apache user, run: apachectl -S
- So, in order to make a directory writable by the webserver we have to set the directory’s owner or group to Apache’s owner or group and enable the write permission for it. Usually, we set the directory to belong to the Apache group (apache or `www-data or whatever user is used to launch the child processes) and enable the write permission for the group.
chgrp www-data ~/mtg/app/
chmod g+rwx ~/mtg/app/
- also check if any imports are failing
- make sure the paths written in python strings are correct

postgres notes

psql [database_name] [user_name]
psql mtg mig
https://www.freecodecamp.org/news/how-to-get-started-with-postgresql-9d3bc1dd1b11/

CREATE DATABASE mtg;

sudo su - postgres
psql
\c DBNAME <- to access the created db
create database mtg;
\l # list databases, then press q to go back to db console

\q # to exit:

\dt # list tables of the public schema

CREATE TABLE card_listings (
card_name varchar(100),
ts timestamp,
avg_sell_price float,
seller_name varchar(30),
seller_sales int,
seller_available_items int,
item_price real,
item_amount int,
item_location varchar(30),
item_condition char(2),
item_language varchar(20),
item_is_playset boolean,
item_is_foil boolean
);

create user mig with password 'password';

GRANT ALL PRIVILEGES ON DATABASE mtg to mig;

GRANT CONNECT ON DATABASE mtg TO mig;
GRANT USAGE ON SCHEMA public TO mig;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO mig;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO mig;

GRANT USAGE ON SCHEMA public TO mig;
GRANT SELECT, UPDATE, INSERT, DELETE ON ALL TABLES IN SCHEMA public TO mig;

ALTER TABLE card_listings DROP COLUMN list_order;

ALTER TABLE card_listings ADD COLUMN list_order int;

-- Firstly, remove PRIMARY KEY attribute of former PRIMARY KEY ALTER TABLE card_listings DROP CONSTRAINT card_listings_pkey;

-- Lastly set your new PRIMARY KEY ALTER TABLE card_listings ADD PRIMARY KEY (card_name, ts, list_order);

cron notes

https://askubuntu.com/questions/420981/how-do-i-save-terminal-output-to-a-file

https://askubuntu.com/questions/420981/how-do-i-save-terminal-output-to-a-file

crontab -e crontab -u mig -e

change defaulty shell for cron

SHELL=/bin/bash

#regular prototype_scraping job 0 * * * * export DISPLAY=:0.0 ; /home/mig/anaconda3/envs/mtg/bin/python ~/mtg/prototype_scraping.py |& tee -a ~/mtg/logs/regular_log.txt

#regular prototype_web_plot_update_data job 30 * * * * /home/mig/anaconda3/envs/mtg/bin/python ~/mtg/prototype_web_plot_update_data.py |& tee -a ~/mtg/logs/plot_update_log.txt

#experimental prototype_scraping job 25 * * * * export DISPLAY=:0.0 ; /home/mig/anaconda3/envs/mtg/bin/python ~/mtg/prototype_scraping.py |& tee -a ~/mtg/logs/experimental_log.txt

SSH access remote server, and running graphics applications (browser windows, for example)

https://askubuntu.com/questions/213678/how-to-install-x11-xorg

https://unix.stackexchange.com/questions/353258/how-to-run-google-chrome-or-chromium-on-a-remote-ssh-session

do this on mig-pc:
~/.ssh/config
Host mig-server
HostName mtgdata.ml
User mig
Port 49000

do this on mig-server:
https://pt.godaddy.com/help/alterar-a-porta-ssh-para-o-seu-servidor-com-o-linux-7306

ssh -X -p 49000 user@hostname
ssh -X -p 49000 [email protected]
ssh -X -p 49000 [email protected] ssh -X -p 49000 [email protected]

check if port is open or closed: nmap -p 49000 79.168.14.53 nmap -p 49000 mtgdata.ml

allow connections on specific port: sudo ufw allow 49000

How do I properly test whether my Port Forwarding works?

https://superuser.com/questions/307820/how-do-i-properly-test-whether-my-port-forwarding-works

timezone notes

https://serverfault.com/questions/554359/postgresql-timezone-does-not-match-system-timezone
The solution for your case is quite simple, just change the TimeZone setting on postgresql.conf to the value you want:

TimeZone = 'Europe/Vienna'

After that you need to reload the service:

sudo su - postgres -c "psql mtg -c 'SELECT pg_reload_conf()'"

https://stackoverflow.com/questions/3602450/where-are-my-postgres-conf-files
ask your database:
$ psql -U postgres -c 'SHOW config_file'

SET TIME ZONE 'UTC';
select now();
show timezone;

use this setting in pandas as well (UTC)

ALTER TABLE card_listings
ALTER COLUMN item_price TYPE real;