Skip to content

Creating an OpenStreetMap Server

Jump to bottom
Joe Blankenship edited this page Jul 25, 2016 · 3 revisions

Creating an OpenStreetMap Server

Here is my process for creating an OSM server. This is an adaptation from switch2osm.org

Software Installation

Here we are going to use Ubuntu Server 14.04 LTS with PostgreSQL 9.5. We will later use PostGIS 2.x and H-store

If you have a fresh Ubuntu Server install, make sure the PostgreSQL they offer during installation is version 9.5. If not, install version 9.5 after the install. It is best to make sure everything is up to date before installing new packages:

sudo apt-get update
sudo apt-get upgrade

We want to make sure that all supporting libraries are installed:

sudo apt-get --no-install-recommends install git unzip curl build-essential software-properties-common

Next we want to install PostgreSQL and PostGIS:

sudo apt-get --no-install-recommends install postgresql-9.5-postgis-2.2 postgresql-contrib-9.5 proj-bin libgeos-dev

I also installed these libraries:

sudo apt-get --no-install-recommends install libdbg-pg-perl sysstat iotop ptop

Then we install the OSM software:

sudo add-apt-repository ppa:kakrueger/openstreetmap
sudo apt-get update
sudo apt-get --no-install-recommends install osm2pgsql osmctools

After all of these installations, check the versions to ensure they are correct:

proj ## version must be at least 4.8.0
psql --version ## version must be at least 9.5
grep 'default_version' '/usr/share/postgresql/9.5/extension/postgis.control' ## version must be at least 2.1
geos-config --version ## version must be at least 3.4.2
osm2pgsql --version ## version must be at least 0.85.0
osmconvert -h | head -n2 ## version must be at least 0.7T

Preparing to load the data

osm2pgsql must be adjusted to work with multiple processes via changes to the overcommit settings:

sudo tee /etc/sysctl.d/60-overcommit.conf <<EOF
# Overcommit settings to allow faster osm2pgsql imports
vm.overcommit_memory=1
EOF
sudo sysctl -p /etc/sysctl.d/60-overcommit.conf

For a temporary change:

sudo sysctl -w vm.overcommit_memory=1

Creating a database

For those in Windows, Mac, or Linux GUI environments, this can be done using pgAdminIII, but here we'll be using the psql console commands (which for Windows users can be done at the command prompt; Mac and Linux users can use terminal).

First, we will create users (from postgres user (NOT root), create a database, spatially enable the database with PostGIS, and install H-Store:

sudo -u postgres createuser -s $USER
createdb osm
psql -d osm -c 'CREATE EXTENSION hstore; CREATE EXTENSION postgis;'

Depending on the type of service environment, you may want to fine tune your server. See the Sources section for the link.

Next we want to download the style sheets to indicate which tags get imported into the database. If you have a specific tag structure in mind, adjust these files to reflect the end schema you want (see sources for links):

mkdir -p ~/osm
cd ~/osm
git clone https://github.com/gravitystorm/openstreetmap-carto.git

Loading the data

There are numerous ways to load the data depending on the size of the extracts you are loading. Two are presented below. Ensure you know the specifications of the server hardware so you can assign the appropriate resources to this process.

Loading an extract

Go to Geofabrik or other extract resource to download the area(s) you want to load into your database:

cd ~/osm
wget http://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf.md5
wget http://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf
md5sum -c liechtenstein-latest.osm.pbf.md5

Then run osm2pgsql to load that data into the database you created earlier (e.g., osm):

osm2pgsql --create -d database --slim --cache 1000 --number-processes 2 --hstore --style ~/osm/openstreetmap-carto/openstreetmap-carto.style --multi-geometry ~/osm/filename.osm.pbf

Replace the database name, style file, and pbf file path/name with your desired inputs. Adjust the variables to suite the server you are using and the manner in which you want osm2pgsql to manipulate the database and tables. See here for more details: http://wiki.openstreetmap.org/wiki/Osm2pgsql

Loading a Planet file or large extract

Make sure your server has enough resources - at least 16GB of RAM, 4 cores, and 500GB of free space - NOT a low end VM READ this entire section first before running

Go to PlanetOSM to download the latest weekly PBF file

cd ~/osm
wget http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf.md5
wget http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf
md5sum -c planet-latest.osm.pbf.md5

To update the dump:

osmupdate planet-latest.osm.pbf new_planet-latest.osm.pbf

Then run osm2pgsql to load that data into the database you created earlier (e.g., osm):

osm2pgsql --create -d database --slim --flat-nodes ~/osm/flat_nodes.bin -C 14000 --number-processes 4 --hstore --style ~/osm/openstreetmap-carto/openstreetmap-carto.style --multi-geometry ~/osm/filename.osm.pbf

Replace the database name, style file, and pbf file path/name with your desired inputs. Adjust the variables to suite the server you are using and the manner in which you want osm2pgsql to manipulate the database. See here for more details: http://wiki.openstreetmap.org/wiki/Osm2pgsql

Since this process takes so long to run, I recommend running this as NOHUP:

nohup osm2pgsql --create -d database --slim --flat-nodes ~/osm/flat_nodes.bin -C 14000 --number-processes 4 --hstore --style ~/osm/openstreetmap-carto/openstreetmap-carto.style --multi-geometry ~/osm/filename.osm.pbf &

OR disowning the process and pushing it to the background:

# Ctrl-Z to halt the process
jobs ## check the number of the osm2pgsql process. For this example, its 1
disown -h ## if osm2pgsql is the only process, otherwise tell it which process (e.g., 1)
bg 1 ## this starts the halted process 1 back up in the background.

The process should now run after you logout.

Create users

From you system user:

sudo su postgres

Then, go into postgres using the postgres user:

psql -d postgres -U postgres

At this time, create a user and revoke their public permissions:

CREATE USER user WITH PASSWORD 'password';
REVOKE ALL ON DATABASE osm FROM PUBLIC;
REVOKE ALL ON SCHEMA public FROM PUBLIC;

Then change to database in which you want to give the new user permissions:

\c osm

Then grant connection, select, and usage permissions for the user (read-only):

GRANT CONNECT ON DATABASE osm to user;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO user;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO user;
GRANT USAGE ON SCHEMA public to user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON SEQUENCES TO user;

The new user should then be able to read the tables in the database using their new credentials.

Known Issues

  • When attempting to run osm2pgsql, sometimes it will throw an error stating you don't have permission to access the file.
    • Try to use the absolute path from root to the file
    • Try to change the permissions of the folder (in linux its chmod [rwx per user] file or folder)
    • Try to use system user or a created user in PostgreSQL with read/write access to the database instead of postgres user
  • Tags are stored in hstore for quick indexing per unique object per table.
    • Make sure you setup your styles before hand as this is what the tables use for the schema.
    • Otherwise, you will have to parse out the tags data for the desired fields.

Sources (Many thank!)

Clone this wiki locally