Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

949 missing leg in miovision_api.open_issues join #950

Merged
merged 6 commits into from
Jul 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions volumes/miovision/sql/data_checks/select-open_issues.sql
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,25 @@
classification || ' (' || classification_uid || ')',
'All classifications'
) AS classification,
last_week_volume,
leg,
volume,
notes
FROM miovision_api.open_issues
WHERE last_week_volume > 0
WHERE volume > 0
ORDER BY uid
)

SELECT
NOT(COUNT(*) > 0) AS _check,
CASE WHEN COUNT(*) = 1 THEN 'There is ' ELSE 'There are ' END || COUNT(*)
|| ' open ended anomalous_range(s) with non-zero volumes in the last 7 days:'
|| ' open ended anomalous_range(s) with non-zero volumes within the last 7 days (or since AR began):'

Check notice on line 20 in volumes/miovision/sql/data_checks/select-open_issues.sql

View workflow job for this annotation

GitHub Actions / SQLFluff Lint

SQLFluff

LT05: Line is too long (105 > 100).
AS summ,
array_agg(
'uid: `' || ars.uid || '`, `'
|| ars.intersection || '`, `'
|| ars.classification || '`, '
|| 'volume: `' || ars.last_week_volume || '`, '
|| CASE WHEN ars.leg IS NOT NULL THEN '`' || ars.leg || ' leg`, ' ELSE '' END
|| 'volume: `' || to_char(ars.volume, 'FM9,999,999') || '`, '
|| 'notes: `' || LEFT(ars.notes, 80)
|| CASE WHEN LENGTH(ars.notes) > 80 THEN '...' ELSE '' END || '`'
) AS gaps
Expand Down
39 changes: 31 additions & 8 deletions volumes/miovision/sql/views/create-view-open_issues.sql
Original file line number Diff line number Diff line change
@@ -1,6 +1,23 @@
--DROP VIEW miovision_api.open_issues;
CREATE OR REPLACE VIEW miovision_api.open_issues AS

--DROP VIEW miovision_api.open_issues;

WITH alerts AS (
SELECT
ar.uid,
string_agg(DISTINCT alerts.alert, '; ') AS alerts
FROM miovision_api.anomalous_ranges AS ar
LEFT JOIN miovision_api.alerts
ON alerts.intersection_uid = ar.intersection_uid
AND alerts.start_time >= ar.range_start
AND (
alerts.end_time < ar.range_end
OR ar.range_end IS NULL
)
GROUP BY ar.uid
)

Check notice on line 20 in volumes/miovision/sql/views/create-view-open_issues.sql

View workflow job for this annotation

GitHub Actions / SQLFluff Lint

SQLFluff

LT01: Unnecessary trailing whitespace.
SELECT
ar.uid,
ar.intersection_uid,
Expand All @@ -13,11 +30,12 @@
WHEN ar.classification_uid IS NULL THEN 'All modes'
ELSE c.classification
END,
ar.leg,
ar.range_start::date,
(current_timestamp AT TIME ZONE 'EST5EDT')::date - ar.range_start::date AS num_days,
ar.notes,
SUM(v.volume) AS last_week_volume,
string_agg(DISTINCT alerts.alert, '; ') AS alerts
SUM(v.volume) AS volume,
alerts.alerts
FROM miovision_api.anomalous_ranges AS ar
--keep rows with null classification_uid
LEFT JOIN miovision_api.classifications AS c USING (classification_uid)
Expand All @@ -26,18 +44,19 @@
--find last week volume
LEFT JOIN miovision_api.volumes AS v
ON ar.intersection_uid = v.intersection_uid
--volume within the last 7 days and after AR started
AND v.datetime_bin >= ar.range_start
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so for example if we run this today, and the range was add yesterday,
the sum volume for last_week_volume would be summing up only 1 day of data

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because we wouldn't want to know the volume from before range_start, which would either be zero, or non-zero from a time before the issue started. I could rename it/change it to average daily volume if you think that's better?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yess, to reduce confusion if that ever happens

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decided to explain it with a column comment and also in the check_open_anomalous_ranges airflow alert.

--prune the partitions
AND v.datetime_bin >= current_date - interval '7 days'
AND (
ar.classification_uid = v.classification_uid
OR ar.classification_uid IS NULL
)
LEFT JOIN miovision_api.alerts
ON alerts.intersection_id = i.id
AND alerts.start_time >= ar.range_start
AND (
alerts.end_time < ar.range_end
OR ar.range_end IS NULL
ar.leg = v.leg
OR ar.leg IS NULL
)
LEFT JOIN alerts ON alerts.uid = ar.uid
WHERE
ar.problem_level <> 'valid-caveat'
--currently active
Expand All @@ -56,7 +75,8 @@
ar.classification_uid,
c.classification,
ar.range_start,
ar.notes
ar.notes,
alerts.alerts
ORDER BY
ar.intersection_uid,
ar.range_start,
Expand All @@ -68,5 +88,8 @@
(intersections.id, classifications.classification). anomalous_ranges.id col can be
used to link response back to table.''';

COMMENT ON COLUMN miovision_api.open_issues.volume
IS 'Volume recorded within the last 7 days or since anomalous_range began, whichever is later.'

ALTER TABLE miovision_api.open_issues OWNER TO miovision_admins;
GRANT SELECT ON TABLE miovision_api.open_issues TO bdit_humans;
2 changes: 2 additions & 0 deletions volumes/miovision/sql/views/create-view-volumes_daily.sql
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ CREATE VIEW miovision_api.volumes_daily AS (
ar.classification_uid = v.classification_uid
OR ar.classification_uid IS NULL
)
--omitting join to ar.leg implies that a single leg outage is
--sufficient to exclude entire intersection.
AND v.dt >= ar.range_start
AND (
v.dt < ar.range_end
Expand Down