Dashboard

[bhousel]

Following up from a chat with @mojodna, @kamicut, and @jenningsanderson ..

Let's setup a dashboard to track activity. I'm thinking of things like

• num of buildings
• num of 3D buildings
• num of POIs
• num of parks
• num of addresses
• num of turn restrictions
• num of stop/give_way
• num of traffic_signals
• num of parking lots
• km of parking aisles / service roads / driveways
• km of footpaths / sidewalks / crossings
• km of roads with lanes
• notes closed

or maybe more obscure / scavenger hunt kinds of features?

• swimming pools
• manholes
• flagpoles
• bollards
• barriers/fences
• bike racks
• surveillance cameras
• bus shelters
• vending machines

Maybe we could send out swag to the winners.
cc @iandees @mikelmaron for thoughts.

[mikelmaron]

😍

Growth over time per feature?

Can we visualize on a map as well?

[mojodna]

I'd hoped to get this up and running more in advance of the conference, but we do have some live-updating data that @kamicut is going to hook up to an instance of Scoreboard soon.

An instance of the OSMesa stat server is running at http://ec2-34-201-2-29.compute-1.amazonaws.com/

This only tracks Missing Maps metrics at the moment (roads, waterways, buildings, POIs), but I may be able to update it (and backfill) to include stuff from ^^ that doesn't involve relations (because streaming augmented diffs don't include relation geoms yet).

Endpoints of interest:

• http://ec2-34-201-2-29.compute-1.amazonaws.com/users - list users
• http://ec2-34-201-2-29.compute-1.amazonaws.com/users/2772907 - "markscalf", Detroit's most prolific mapper
• http://ec2-34-201-2-29.compute-1.amazonaws.com/changesets - list changesets (this includes all changesets since replication started, not just ones that touched Detroit (but only includes calculations for those))
• http://ec2-34-201-2-29.compute-1.amazonaws.com/changesets/17710 - Detroit's first changeset
• http://ec2-34-201-2-29.compute-1.amazonaws.com/campaigns - hashtags (this includes all hashtags used since replication started, similar to changesets. it also includes extent_uris that don't reference anything)
• http://ec2-34-201-2-29.compute-1.amazonaws.com/campaigns/sotmus2018 - #sotmus2018

Endpoints that return lots of stuff return 25 per page and use ?page=<page number> to paginate.

The user endpoint includes extent_uri, e.g. user/2772907/{z}/{x}/{y}.mvt. This corresponds (or should) to a footprint tileset at http://ec2-34-201-2-29.compute-1.amazonaws.com/tiles/user/2772907/{z}/{x}/{y}.mvt (backed by s3://osm-detroit/footprints/user/user/2772907/{z}/{x}/{y}.mvt).

This tileset can be visualized using something similar to https://bl.ocks.org/mojodna/300441d3c581828f70290fbd4a5cea3d/ce25de3b113244edcdb8204d948d354a8d6f2eb0

Footprints exist for all OSM users who have edited in Detroit (or who edit while this is running). The tilesets are updated effectively immediately.

[mojodna]

Full history geometry tiles (at z15) are available from s3://osm-detroit/detroit-geom-tiles-20180930/15/{x}/{y}.mvt. These contain data through 9/30/18.

They can be visualized using something similar to https://bl.ocks.org/mojodna/49e42f58ab1ea6d560de584c1d1d3edf/5f7270d55e443e6cd462f658b7169fbdb04296e2

[mojodna]

Here's the underlying data (all paths relative to s3://osm-detroit/), should anyone want to have a play (pg_dump snapshot of aggregated statistics available upon request).

Raw

detroit-20180930.osh.pbf - Raw data (downloaded from Geofabrik, extracted using osmium w/ bbox: -83.2877,42.2555,-82.9105,42.4502)

Transcoded to ORC

detroit-changesets/ - Changesets involving Detroit
detroit-history/ - OSM history of Detroit

Aggregated statistics (ORC)

changeset-stats-20180930/ - OSMesa (Missing Maps-style) changeset stats
hashtag-stats-20180930/ - Hashtag stats
user-stats-20180930/ - User stats

These are subject to change; if I'm able to calculate any additional metrics (from above), I'll replace what's here.

Processed data

detroit-geoms-20180930/ - (ORC) Metadata + WKB geometries for all versions of all features. (Does not include turn restrictions, the 3D building in Detroit, or anything defined using recursive relations.)
tag-changes-20180930/ - (ORC) Tags added/removed between feature versions
detroit-geom-tiles-20180930/ - Full history MVTs @ z15
footprints/user/ - User footprint MVTs @ z14

Misc

augmented-diffs/ - augmented diff JSON, converted from Overpass output using overpass-diff-publisher. Compatible with OSMesa's AugmentedDiffSource, but likely useful in other environments. See state.yaml for the latest available sequence number.

AWS Glue will happily crawl each of the S3 prefixes containing ORC files and produce tables for use in Athena.

All of these datasets (included aggregated statistics in PostgreSQL) were generated (and are updated) using OSMesa.

[mojodna]

I re-ran the changeset statistics (and user rollup) with most of @bhousel's suggested metrics. The updated ORC files are in the same paths.

Here's the top-level roll-up:

road_km_added	road_km_modified	waterway_km_added	waterway_km_modified	roads_added	road edits	waterways_added	waterway edits	buildings_added	building edits	parking_aisle_km_added	parking_aisle_km_modified	parking_aisles_added	parking_aisle edits	service_road_km_added	service_road_km_modified	service_roads_added	service_road edits	driveway_km_added	driveway_km_modified	driveways_added	driveway edits	sidewalk_km_added	sidewalk_km_modified	sidewalks_added	sidewalk edits	lane_km_added	lane_km_modified	lanes_added	lane edits	cycleway_km_added	cycleway_km_modified	cycleways_added	cycleway edits	traffic_signs_added	traffic_sign edits	traffic_signals_added	traffic_signal edits	manholes_added	manhole edits	flagpoles_added	flagpole edits	bollards_added	bollard edits	surveillance_added	surveillance edits	vending_machines_added	vending_machine edits	pois_added	poi edits	parks_added	park edits	addresses_added	addresse edits	parking_added	parking edits	swimming_pools_added	swimming_pool edits	barriers_added	barrier edits	bike_parking_added	bike_parking edits	bus_shelters_added	bus shelter edits	crossings_added	crossing edits	changesets	users
18187.91363	44920.4253	190.1768178	132.478067	55900	104993	305	397	56084	11202	351.7725419	30.49025261	3336	1412	1425.669421	240.044993	10027	7963	147.4614282	27.87236877	1340	1748	2271.509313	12435.89917	9836	7540	864.601852	13101.62636	4553	28433	104.3654094	102.0677641	457	2361	137	118	174	4097	0	0	57	5	146	16	1	0	3	0	9520	6919	694	839	384696	4604	2892	2233	56	18	1623	630	45	1	33	15	6961	2686	19532	1007

Athena / Presto query:

select
  sum(road_km_added) road_km_added,
  sum(road_km_modified) road_km_modified,
  sum(waterway_km_added) waterway_km_added,
  sum(waterway_km_modified) waterway_km_modified,
  sum(roads_added) roads_added,
  sum(roads_modified) roads_modified,
  sum(waterways_added) waterways_added,
  sum(waterways_modified) waterways_modified,
  sum(buildings_added) buildings_added,
  sum(buildings_modified) buildings_modified,
  sum(parking_aisle_km_added) parking_aisle_km_added,
  sum(parking_aisle_km_modified) parking_aisle_km_modified,
  sum(parking_aisles_added) parking_aisles_added,
  sum(parking_aisles_modified) parking_aisles_modified,
  sum(service_road_km_added) service_road_km_added,
  sum(service_road_km_modified) service_road_km_modified,
  sum(service_roads_added) service_roads_added,
  sum(service_roads_modified) service_roads_modified,
  sum(driveway_km_added) driveway_km_added,
  sum(driveway_km_modified) driveway_km_modified,
  sum(driveways_added) driveways_added,
  sum(driveways_modified) driveways_modified,
  sum(sidewalk_km_added) sidewalk_km_added,
  sum(sidewalk_km_modified) sidewalk_km_modified,
  sum(sidewalks_added) sidewalks_added,
  sum(sidewalks_modified) sidewalks_modified,
  sum(lane_km_added) lane_km_added,
  sum(lane_km_modified) lane_km_modified,
  sum(lanes_added) lanes_added,
  sum(lanes_modified) lanes_modified,
  sum(cycleway_km_added) cycleway_km_added,
  sum(cycleway_km_modified) cycleway_km_modified,
  sum(cycleways_added) cycleways_added,
  sum(cycleways_modified) cycleways_modified,
  sum(traffic_signs_added) traffic_signs_added,
  sum(traffic_signs_modified) traffic_signs_modified,
  sum(traffic_signals_added) traffic_signals_added,
  sum(traffic_signals_modified) traffic_signals_modified,
  sum(manholes_added) manholes_added,
  sum(manholes_modified) manholes_modified,
  sum(flagpoles_added) flagpoles_added,
  sum(flagpoles_modified) flagpoles_modified,
  sum(bollards_added) bollards_added,
  sum(bollards_modified) bollards_modified,
  sum(surveillance_added) surveillance_added,
  sum(surveillance_modified) surveillance_modified,
  sum(vending_machines_added) vending_machines_added,
  sum(vending_machines_modified) vending_machines_modified,
  sum(pois_added) pois_added,
  sum(pois_modified) pois_modified,
  sum(parks_added) parks_added,
  sum(parks_modified) parks_modified,
  sum(addresses_added) addresses_added,
  sum(addresses_modified) addresses_modified,
  sum(parking_added) parking_added,
  sum(parking_modified) parking_modified,
  sum(swimming_pools_added) swimming_pools_added,
  sum(swimming_pools_modified) swimming_pools_modified,
  sum(barriers_added) barriers_added,
  sum(barriers_modified) barriers_modified,
  sum(bike_parking_added) bike_parking_added,
  sum(bike_parking_modified) bike_parking_modified,
  sum(shelters_added) shelters_added,
  sum(shelters_modified) shelters_modified,
  sum(crossings_added) crossings_added,
  sum(crossings_modified) crossings_modified,
  count(changeset) changesets,
  count(distinct uid) users
from changeset_stats_20180930

[mojodna]

Here's an Observable notebook that allows the full history geometry tiles to be explored: https://beta.observablehq.com/@mojodna/explorable-detroit

[mojodna]

Notebook with user contribution heatmaps: https://beta.observablehq.com/@mojodna/detroit-contributor-heatmaps

There are 2 limitations that I would love someone to address:

• lookups from display_name to uid (since the tiles are addressed by uid)
• prevent "Go" from needing to be clicked twice; something in removeSource() isn't actually removing the source in time.

[mojodna]

Limitations addressed.