Efficiently uploading rasters to PostgreSQL

[dhardestylewis]

Note that with this command will create in-db rasters with tiles, which will consume ~2 times as much space as the original rasters.

TNRIS's entire collection of Lidar DEMs is ~3TB, so this consumes ~6TB.

This takes an estimated 27 hours to run over 223 cores (max 310.632 mB per process), 24 hours once for the first 265 sets of ~1200 tiles each, and 3 hours for the remaining 7 sets of ~20 tiles each.

for filename in $(ls /scratch/04950/dhl/single_seamless_elevation_buckets.csv.d/tnris_lidar_tiles_float_bts_notbad_absolutepath.csv.d/tnris_lidar_tiles_float_bts_notbad_absolutepath.csv*) ; do raster2pgsql -a -t 32x32 -F -q -e $(cat ${filename} | tr "\n" " ") public.best_tnris_tiles_in_db${filename: -3} > ${filename}.sql & done