Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running out of space when downloading the planet #418

Open
Shadow-shadown opened this issue Apr 15, 2024 · 3 comments · May be fixed by #420
Open

Running out of space when downloading the planet #418

Shadow-shadown opened this issue Apr 15, 2024 · 3 comments · May be fixed by #420

Comments

@Shadow-shadown
Copy link

I'm using this docker to download the planet
My system configuration is 10 core cpu and 128gb ram and 2.2TB storage

I'm running into the issue of no space in disk when I check it it shows 57% used

The error

  • chown -R renderer: /home/renderer/src/ /data/style/
  • '[' -f /data/style/scripts/get-external-data.py ']'
  • '[' -f /data/style/external-data.yml ']'
  • sudo -E -u renderer python3 /data/style/scripts/get-external-data.py -c /data/style/external-data.yml -D /data/style/data
    INFO:root:Starting load of external data into database
    Traceback (most recent call last):
    File "/data/style/scripts/get-external-data.py", line 311, in
    main()
    File "/data/style/scripts/get-external-data.py", line 182, in main
    os.makedirs(data_dir, exist_ok=True)
    File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
    OSError: [Errno 28] No space left on device: '/data/style/data'

The docker command used to run the container

docker run -d --shm-size="10G" -e DOWNLOAD_PBF=https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf -e PGPASSWORD=${PGPASSWORD} -e "OSM2PGSQL_EXTRA_ARGS=-C 40096" -e "FLAT_NODES=enabled" -v /osm-data:/data/database/ -v osm-tiles:/data/tiles/ tileserver import

The disk space

sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─oar_ubuntu-swap 253:1 0 3.9G 0 lvm [SWAP]
├─oar_ubuntu-home 253:2 0 9.8G 0 lvm /home
└─oar_ubuntu-root 253:4 0 85.3G 0 lvm /
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 2.7T 0 part
├─data-osm--data 253:0 0 2.1T 0 lvm /osm-data
└─data-osm--tiles 253:3 0 500G 0 lvm /osm-tiles

What am I doing wrong could someone help me out

@Istador
Copy link
Contributor

Istador commented Apr 15, 2024

docker run -d --shm-size="10G" -e DOWNLOAD_PBF=https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf -e PGPASSWORD=${PGPASSWORD} -e "OSM2PGSQL_EXTRA_ARGS=-C 40096" -e "FLAT_NODES=enabled" -v /osm-data:/data/database/ -v osm-tiles:/data/tiles/ tileserver import

NOT an explanation for the error, but -v osm-tiles:/data/tiles/ should be -v /osm-tiles:/data/tiles/ instead, if you want it saving to the extra partition that you created and that is mounted to /osm-tiles. Otherwise it saves to /var/lib/docker/volumes/....

(Unless you created the osm-tiles named volume manually with providing a mount point, but because you didn't seem to do that for /osm-data I assume you made a mistake here.)


Interestingly it says:

No space left on device: '/data/style/data'

/data/style/data is not part of the mounted volumes for /data/database/ or /data/tiles/ and is therefore part of the containers internal storage.

So the issue seems to be related to the root partition oar_ubuntu-root that holds the containers (/var/lib/docker/containers/...). Seemingly it doesn't have enough space to hold some temporary files downloaded in a later stage of the import (the main import of the planet.osm is already done).

Note: the ~75 GB planet-latest.osm.pbf was also downloaded into the container to /data/region.osm.pbf and resides on the root partition.

Note: you don't execute docker run with the --rm parameter, so it doesn't automatically delete containers and their internal storage automatically, so you need to do that manually.


An improvement to this image to prevent this error would be to add the following code to the run.sh after the osm2pgsql step of the import to clean up the temporarily downloaded files after the main import step:

# clean up downloaded files
if [ -n "${DOWNLOAD_PBF:-}" ]; then
    rm /data/region.osm.pbf
    if [ -n "${DOWNLOAD_POLY:-}" ]; then
        rm /data/region.poly
    fi
fi

@Shadow-shadown
Copy link
Author

Is there anyway that I can mount it to a volume or the temporary remove commands on top will be a better solution ..?

Istador added a commit to Istador/openstreetmap-tile-server that referenced this issue Apr 20, 2024
@Istador Istador linked a pull request Apr 20, 2024 that will close this issue
@Istador
Copy link
Contributor

Istador commented Apr 20, 2024

You could download the file before and then mount it as described in the readme.

    -v /absolute/path/to/luxembourg.osm.pbf:/data/region.osm.pbf \

Otherwise you'd need to modify the run.sh yourself and build a new image.

I created PR #420 to fix this, but I don't believe it will be merged anytime soon. The last update to this project was 10 months ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants