Tile + deploy new commute raster data

[Robinlovelace]

Heads-up @nikolai-b, the new method of generating raster tiles works 1000+ times faster and seems to yield good results. See image below for London and the data here for the census scenario: https://github.com/npct/pct-scripts/releases/download/0.0.1/ras_bicycle_all_new_30.tif

Are you up for tiling and deploying this? Can do some tiling if you like. Should be much faster and lighter now, these have 30 m resolution. Can always tweak for better results (e.g. I think if the whole layer has some transparency it will be more useful, and we can always change the base resolution) but think this is looking good and is ready to test!

[Robinlovelace]

Just thinking @nikolai-b: perhaps the entire rnet build script should be in here, including this which generates the national build fast and with limited (<8GB) memory use:

wget https://github.com/npct/pct-outputs-national/raw/master/commute/lsoa/ras_bicycle_all.tif
gdal_calc.py -A ras_bicycle_all.tif --outfile=empty.tif --calc "A*0" --NoDataValue=0 # takes a few minutes
# gdal_translate -ot Int16 empty.tif empty16.tif
cp empty.tif empty1.tif empty2.tif empty3.tif empty4.tif empty5.tif
i=0
while (( i++ < 5 )); do
cp empty30.tif "empty30$i.tif"
done
gdalinfo empty.tif
ogrinfo rnet_all.gpkg
gdalwarp -tr 30 -30 empty.tif empty30.tif # about 10 times smaller
ls -hal | grep em

gdal_rasterize -burn -a bicycle -at rnet_all.gpkg empty30.tif # adds to existing layer
cp empty30.tif ras_bicycle_all_new_30.tif

You up for giving it a shot. The above does it for the bicycle layer but we need to do the others also.

[nikolai-b]

@Robinlovelace for the dutch tiles I'm only getting zero values, can you check if there is something wrong there? Also I've been tiling
ras_all_new_commute_10.zip

Do you want me to do anything with
ras_bicycle_all_new_10.zip
OR
ras_bicycle_all_new_30.tif

I'm assuming these are olc, right? I'm tiling ras_bicycle_all_new_10.zip but at the top you link to the .tif, let me know if that is correct. Ta

[Robinlovelace]

Just traveling back from Birmingham after visiting dad will give you a call when back. Will check go Dutch results. Do the results for other scenarios look ok?

[Robinlovelace]

In answer to your question, no everything should be on that big zip file which has 6 TIF files in, one for each scenario.

[nikolai-b]

Have a look at https://github.com/npct/pct-shiny/tree/new-ras,
I've put the tiles up there

[Robinlovelace]

Great work. The Census 2011 tiles work for sure. Tests on that branch suggest that the tiling has not been done for zoom levels greater than around 13, probably a good idea to test the tiles work before building all of them. Overall looks good:

[Robinlovelace]

Also looks like the intentional pixelisation at low zoom levels is not the same, at least in London:

[Robinlovelace]

Also @nikolai-b, regarding the question:

for the dutch tiles I'm only getting zero values, can you check if there is something wrong there?

I've checked the file ras_dutch_slc_all_new.tif I have on my computer and seems to have correct values:

can you check the results of the following command on your computer:

gdalinfo -approx_stats ras_dutch_slc_all_new.tif
Driver: GTiff/GeoTIFF
Files: ras_dutch_slc_all_new.tif
Size is 51892, 64034
Coordinate System is:
PROJCS["OSGB 1936 / British National Grid",
    GEOGCS["OSGB 1936",
        DATUM["OSGB_1936",
            SPHEROID["Airy 1830",6377563.396,299.3249646,
                AUTHORITY["EPSG","7001"]],
            TOWGS84[446.448,-125.157,542.06,0.15,0.247,0.842,-20.489],
            AUTHORITY["EPSG","6277"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4277"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",49],
    PARAMETER["central_meridian",-2],
    PARAMETER["scale_factor",0.9996012717],
    PARAMETER["false_easting",400000],
    PARAMETER["false_northing",-100000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","27700"]]
Origin = (136400.116390000010142,654396.785440000006929)
Pixel Size = (10.000000000000000,-10.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  136400.116,  654396.785) (  6d11'49.49"W, 55d42'40.23"N)
Lower Left  (  136400.116,   14056.785) (  5d40'35.19"W, 49d58' 3.81"N)
Upper Right (  655320.116,  654396.785) (  2d 3'56.26"E, 55d42'56.18"N)
Lower Right (  655320.116,   14056.785) (  1d33'40.33"E, 49d58'16.77"N)
Center      (  395860.116,  334226.785) (  2d 3'41.61"W, 52d54'18.51"N)
Band 1 Block=51892x1 Type=UInt16, ColorInterp=Gray
  Minimum=1.000, Maximum=7752.000, Mean=74.154, StdDev=167.485
  NoData Value=0
  Metadata:
    STATISTICS_APPROXIMATE=YES
    STATISTICS_MAXIMUM=7752
    STATISTICS_MEAN=74.154128859047
    STATISTICS_MINIMUM=1
    STATISTICS_STDDEV=167.48465194673
    STATISTICS_VALID_PERCENT=1.563

[nikolai-b]

@Robinlovelace thanks, yes you are right, the colour process terminated early, I'm redoing it now. As for the zoom level I put up the low res stuff first to check, the higher zoom levels are uploading now.