Miovision Aggregation Improvements (ATR view, intersection_movements AFTER INSERT trigger, unacceptable_gaps)

[gabrielwol]

What this pull request accomplishes:

1. Remove miovision_api.volumes_15min (table) in place of view (volumes_15min_atr_filtered). Reduces duplicate storage (~75GB) and complexity in aggregation.
2. Add an AFTER INSERT trigger to intersection_movements to help aggregate and pad new movements into volumes_15min_mvt.
   • volumes/miovision/sql/function/function-add_intersection_movement_padding_values.sql: based on existing inserts into volumes_15min_mvt, except using new_rows (transition relation) instead of intersection_movements and without date limits.
3. Move unacceptable_gap removal from table inserts to filtered view: less complexity in adding and removing unacceptable_gaps.

Summary of files changed:
Many files have minor changes including renamed references, removal of ATR aggregation, big fluff (intersection_movements, volumes_15min_tmc).

Files with major changes:

• volumes/miovision/sql/function/function-add_intersection_movement_padding_values.sql: trigger function to add values to 15min aggregate table based on additions to intersection_movements. Based closely on existing aggregation but only for new intersection_movements.
• volumes/miovision/sql/table/create-table-movement_map_new.sql: a wide version of movement_map with entry/exit leg columns and directions more clearly labelled. Helps with ATR view joins.
• volumes/miovision/sql/views/create-view-volumes_15min_atr_filtered.sql: ATR view which replaces table.
• volumes/miovision/sql/views/create-view-volumes_15min_mvt_filtered.sql: add anti-join of unacceptable_gaps.

Issue(s) this solves:

• Closes Miovision: discrepencies between volumes_15min_mvt and unacceptable_gaps #868, Miovision: Should we filter unacceptable_gaps from volumes_15min_mvt via view instead of during insert? #879: filters unacceptable_gaps via view
• Progress towards closing issues around intersection_movements: Big list of miovision.intersection_movement inconsistencies and bugs #406, Need to monitor intersection_movements for missing high volume movements #855

What, in particular, needs to reviewed:

• Another option considered is adding movement map columns entry_dir, exit_leg, exit_dir to TMC table so ATR view wouldn't need to join to movement_map. Downside: more confusing TMC table, does not match current TMC table.
• Ok to anti join instead of set unacceptable_gaps to null in filtered view?
• New naming: match existing to reduce dependency changes or make contents more explicit?

	Old	New
TMC	volumes_15min_mvt	volumes_15min_mvt_unfiltered
	volumes_15min_mvt_filtered	volumes_15min_mvt_filtered
ATR	volumes_15min
	volumes_15min_filtered	volumes_15min_atr_filtered

What needs to be done by a sysadmin after this PR is merged

E.g.: these tables need to be migrated/created in the production schema.

[gabrielwol]

Testing the new ATR view for speed:

--test: 0.2 s with primary key
SELECT *
FROM miovision_api.volumes_15min_atr_filtered
WHERE
    intersection_uid = 6
    AND classification_uid = 1
    AND datetime_bin = '2024-06-25 12:00:00'
    AND leg <> LEFT(dir, 1);

--DR i0627 test using new view
--41s for original, 1:06 for view (1.3M rows)
SELECT
    volumes.intersection_uid,
    date_trunc('hour', volumes.datetime_bin) AS datetime_bin,
    volumes.leg,
    volumes.dir,
    classifications.classification,
    SUM(volumes.volume) AS volume
FROM miovision_api.volumes_15min_atr_filtered AS volumes
INNER JOIN miovision_api.classifications USING (classification_uid)
WHERE
    volumes.classification_uid NOT IN (2, 7)
    AND volumes.datetime_bin >= '2024-01-01'
    AND volumes.datetime_bin < '2024-05-22'
    AND volumes.intersection_uid IN (76, 88, 81, 85, 84, 87, 75, 83, 86, 49, 80, 36)
GROUP BY
    volumes.intersection_uid,
    classifications.classification,
    date_trunc('hour', volumes.datetime_bin),
    volumes.leg,
    volumes.dir;