Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Readme and build db script for ubuntu 22.04 #133

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Update Readme and build db script for ubuntu 22.04 #133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,16 @@ Les données de Mayotte ainsi que de l'Alsace et de la Moselle sont manquantes c
Une API développée par Christian Quest est disponible [ici](http://api.cquest.org/dvf), avec sa [documentation](http://data.cquest.org/dgfip_dvf/LISEZ_MOI.txt).

## Installation
*(testé avec succès le 24/04/2023)*

### 1- Pré-requis

Il faut un serveur Linux (ici, nous prendrons Ubuntu 18.04 comme exemple)
Il faut un serveur Linux (ici, nous prendrons Ubuntu 22.04 comme exemple)

### 2- Récupération du dépôt
```bash
$ git clone https://github.com/etalab/DVF-app
$ cd DVF
$ cd DVF-app
```

### 3- Installation minimale : pour développer l'interface utilisateur seule (Front End)
Expand All @@ -30,7 +31,7 @@ Pour participer au développement de l'interface utilisateur, il n'est pas néce
```bash
$ sudo apt-get update && sudo apt install nodejs
$ nodejs -v
v8.10.0
v12.22.9
```

- [yarn](https://yarnpkg.com)
Expand All @@ -39,7 +40,7 @@ $ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
$ sudo apt-get update && sudo apt-get install yarn
$ yarn --version
1.15.2
1.22.19
```

Un script [Node.js](https://nodejs.org) permet de servir l'interface et de faire proxy vers l'API de production.
Expand All @@ -58,9 +59,9 @@ $ yarn start

PostgreSQL
```bash
$ sudo apt-get update && sudo apt install postgresql-10
$ sudo apt-get update && sudo apt install postgresql-14
$ psql -V
psql (PostgreSQL) 10.7 (Ubuntu 10.7-0ubuntu0.18.04.1)
psql (PostgreSQL) 14.7 (Ubuntu 14.7-0ubuntu0.22.04.1)
```

Creation de la base de données et import des données :
Expand All @@ -70,6 +71,7 @@ Le script commence par créer une base de données PostgreSQL et une table, puis
```bash
$ sh db/build_db.sh
```
Le nom de la base de données est affiché lors de l'import

Configuration de l'accès à la base de données

Expand All @@ -81,14 +83,14 @@ Enter new password: <YOUR PASSWORD>
Enter it again: <YOUR PASSWORD>
postgres=# \q

$ echo -e "postgres\n<YOUR PASSWORD>\nlocalhost" > config.csv
$ echo -e "postgres\n<YOUR PASSWORD>\nlocalhost<DB NAME>" > config.csv
```

Installation des packages pythons :
```bash
$ sudo apt-get update && sudo apt install python3
$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10
$ sudo apt install libpq-dev python-dev
$ sudo apt install libpq-dev
$ sudo apt install python3-psycopg2 python3-flask python3-pandas python3-sqlalchemy python3-psycopg2
```

Expand Down
67 changes: 49 additions & 18 deletions db/build_db.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,64 @@
#!/bin/bash
# Script de création de la base de donnees PostgreSQL
#set -x
set -e

YEAR_START="2018"
YEAR_END="2022"
DB_NAME="dvf_db"
DATA_DIRECTORY="data"

DIR=$(echo $(dirname $0))
cd $DIR
pwd

YEARS=$( eval echo {$YEAR_START..$YEAR_END} )

month="12"
year_end="2022"
echo "[Téléchargement des données dans $DATA_DIRECTORY pour les années $YEARS]"
mkdir -p $DATA_DIRECTORY

sudo -u postgres psql -c "DROP DATABASE IF EXISTS dvf_${year_end}${month};"
sudo -u postgres psql -c "CREATE DATABASE dvf_${year_end}${month};"
sudo -u postgres psql -c "ALTER DATABASE dvf_${year_end}${month} SET datestyle TO ""ISO, DMY"";"
sudo -u postgres psql -d dvf_${year_end}${month} -f "create_table.sql"
for YEAR in $YEARS
do

URL="https://files.data.gouv.fr/geo-dvf/latest/csv/$YEAR/full.csv.gz"
CSV_FILE="$DATA_DIRECTORY/full_$YEAR.csv.gz"
printf "$YEAR $url => $CSV_FILE :"
if [ -e "$CSV_FILE" ]; then
echo "already done"
else
echo ""
wget -r -np -nH -N --cut-dirs 5 $url -O $CSV_FILE
fi
done

# Chargement des données sur le serveur
DATADIR="data"
mkdir -p $DATADIR

for YEAR in {2018..2022}
echo "[Décompression des données]"
for YEAR in $YEARS
do
[ ! -f $DATADIR/full_$YEAR.csv.gz ] && wget -r -np -nH -N --cut-dirs 5 https://files.data.gouv.fr/geo-dvf/latest/csv/$YEAR/full.csv.gz -O $DATADIR/full_$YEAR.csv.gz
gz_file="$DATA_DIRECTORY/full_$YEAR.csv.gz"
CSV_FILE="$DATA_DIRECTORY/full_$YEAR.csv"
printf "zcat $gz_file > $CSV_FILE :"
if [ -e "$CSV_FILE" ]; then
echo "already done"
else
echo ""
zcat $gz_file > $CSV_FILE
fi
done

find $DATADIR -name '*.gz' -exec gunzip -f '{}' \;
echo "[Réinitilisation de la base de données: $DB_NAME]";
sudo -u postgres psql -c "DROP DATABASE IF EXISTS $DB_NAME;"
sudo -u postgres psql -c "CREATE DATABASE $DB_NAME;"
sudo -u postgres psql -c "ALTER DATABASE $DB_NAME SET datestyle TO ""ISO, DMY"";"
sudo -u postgres psql -d $DB_NAME -f "create_table.sql"

#Chargement des données dans postgres
DATAPATH=$( cd $DATADIR ; pwd -P )
for YEAR in {2018..2022}
echo "[Réimport des données]"
DATAPATH=$( cd $DATA_DIRECTORY ; pwd -P )
for YEAR in $YEARS
do
sudo -u postgres psql -d dvf_$year_end$month -c "COPY dvf FROM '$DATAPATH/full_$YEAR.csv' delimiter ',' csv header encoding 'UTF8';"
printf "$YEAR ..."
sudo -u postgres psql -d $DB_NAME -c "COPY dvf FROM '$DATAPATH/full_$YEAR.csv' delimiter ',' csv header encoding 'UTF8';"
done

# Ajout d'une colonne et d'index - Assez long
sudo -u postgres psql -d dvf_$year_end$month -f "alter_table.sql"
echo "[Indexation]"
sudo -u postgres psql -d $DB_NAME -f "alter_table.sql"