Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Btdag #381

Open
wants to merge 90 commits into
base: master
Choose a base branch
from
Open

Btdag #381

Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
c4e3c05
#343 testing first DAG
Jan 8, 2021
534880f
#343 test run with new date
Jan 8, 2021
956bb45
#343 test run reader_history table
Jan 21, 2021
e9b24aa
343 test run reader history table again
Jan 22, 2021
d6e9700
#250 test dag for broken readers
Jan 26, 2021
bed366a
#250 test dag corrected
Jan 27, 2021
18d2e05
250 test dag updated again
Jan 27, 2021
39a0e85
250 test dag updated test
Jan 27, 2021
ca10bed
#250 test dag updated again
Jan 27, 2021
f1ce6b6
#317 readme file updated
Feb 5, 2021
8af7e0b
#343 blip pipeline fail check all in one DAG
Feb 19, 2021
d090db1
#343 blip pipeline fail check all in one DAG start date updated
Feb 19, 2021
87e2bf5
#381 SQL add in create_tables folder
Feb 26, 2021
03519de
#343 SQL for functions used in airflow added in functions folder
Feb 26, 2021
f89ef79
#381 Readme file updated for creating new routes after Cathy's comments
Mar 26, 2021
fffa6a8
#381 Readme file updated for creating new routes
Mar 26, 2021
eeb6b55
#326 Readme file updated for new tables after Raphael's comments
Apr 16, 2021
2f9209d
#343 blip pipeline fail check slack message in new format
Apr 16, 2021
21154c8
#343 blip pipeline fail check syntax corrected
Apr 23, 2021
df569f0
#343 blip pipeline check syntax update
Apr 23, 2021
9f0bbc3
#343 blip pipeline fail check syntax updated
Apr 30, 2021
98afd58
#343 blip pipeline connection corrected
Apr 30, 2021
9e85abf
#343 blip pipeline start_end corrected
Apr 30, 2021
4a2fb83
#343 blip pipeline ds corrected for yesterday
Apr 30, 2021
86bbc88
#343 blip pipeline ds corrected for yesterday
Apr 30, 2021
1b4848c
#343 blip pipeline ds corrected
Apr 30, 2021
10ae474
#343 blip pipeline ds updated
Apr 30, 2021
2547288
#343 blip pipeline updated
Apr 30, 2021
9a942bc
#343 blip pipeline syntax error corrected
May 7, 2021
ce236f9
#343 blip pipeline functions schema access updated
May 7, 2021
c016dbb
#343 blip pipeline functions updated
May 7, 2021
dc5771e
#343 blip pipeline ds updated with spaces
May 7, 2021
5dad382
#343 blip pipeline ds python operator updad
May 7, 2021
59649fe
#343 blip pipeline ds python operator updated
May 7, 2021
236282f
#343 blip pipeline ds python operator updated
May 7, 2021
2ec625f
#343 blip pipeline ds python operator updated
May 7, 2021
871df9f
#343 blip pipeline ds python operator updated
May 7, 2021
71849f3
#343 blip pipeline slack channel updated
May 14, 2021
7fa08df
#343 blip pipeline date in python operator fixed
May 14, 2021
f35e68d
#343 blip pipeline date in syntax error fixed
May 14, 2021
b247cb1
#343 blip pipeline syntax error fixed
May 14, 2021
7150d10
#343 blip pipeline syntax error fixed
May 14, 2021
b9f6b99
#343 blip pipeline quick fix for pipeline check
May 14, 2021
345b23b
#263 Bluetooth readme.md updated with lookup table addition
May 21, 2021
fdf7a9a
#317 Bluetooth update readme.md updated with liinks
May 21, 2021
1ccd39d
#343 empty list for broken reader test condition resolved
May 21, 2021
468c100
#250 functions changed to use bluetooth schema instead of personal s…
Jun 15, 2021
7ddb0b7
the detector_history_corrected table explaineaiained
Jun 15, 2021
3d96a7f
Offline bluetooth readers past one week.
webgisgeek Jun 18, 2021
508c0dd
#381 DAG name corrected
Jun 23, 2021
2ffa367
#343 DAG name corrected and PR feedback incorporated
Jul 5, 2021
7ce942d
Merge branch 'btdag' of https://github.com/CityofToronto/bdit_data-so…
Jul 5, 2021
55d5abb
Delete bt_read_history.py
webgisgeek Jul 12, 2021
c7dafa4
Delete bt_test_dag.py
webgisgeek Jul 12, 2021
3fdcb36
Delete check_brokenreaders.py
webgisgeek Jul 12, 2021
ee3a698
Updated broken_readers.sql
webgisgeek Jul 29, 2021
d40d1bf
Update reader_status_history.sql
webgisgeek Jul 29, 2021
c62a421
Update README.md
webgisgeek Jul 29, 2021
0db67ae
Update bluetooth_check_readers.py
webgisgeek Jul 29, 2021
5cc7bd1
#450 Update slack alerts to use variable
chmnata Feb 16, 2022
e0c27bd
#343 bugfix
radumas Feb 23, 2022
4f321c3
#381, #250 address comment for reader_status_history sql
chmnata Mar 21, 2022
9e321fb
#381, #250 update column name
chmnata Mar 22, 2022
4164e77
#385, #343 Simplify broken_readers.sql and add comments
chmnata Mar 22, 2022
dc763b6
#385 Add missing insert_report_date function
chmnata Mar 22, 2022
a7216eb
#343 Update insert report date sql used in the dag
chmnata Mar 23, 2022
a74c4dd
#343 Update pipeline check to use sqlcheckoperator and add comments
chmnata Mar 23, 2022
bd98f89
#385 update broken readers sql
chmnata Mar 23, 2022
e341086
#381, #343 Update broken reader task to alert and not kill self
chmnata Mar 24, 2022
e3647aa
#381, #317 update segment table to include length column
chmnata Mar 25, 2022
385a7f3
#131 Update blip api to use psycopg2 instead of pg
chmnata Mar 28, 2022
576b29d
#131 Updpg json encoder to use json dump
chmnata Mar 28, 2022
afacbcc
#131update comments from pg to psycopg2
chmnata Mar 29, 2022
c7cf1df
#131 update upsert sql
chmnata Mar 29, 2022
d36423a
#131fix that json mess in prepping upsert row
chmnata Mar 29, 2022
f06d4b9
#131 fix upsert sql for column name and got rid of included column
chmnata Mar 29, 2022
3353e20
#131 change from tuple to index for formatting analyses_pull_data
chmnata Mar 29, 2022
40b3ce0
#131 got rid of print() comments
chmnata Mar 29, 2022
65c4a6f
#131 fix upsert returned values to use inserting row
chmnata Mar 30, 2022
703dcab
#381 edit update readme
chmnata Mar 30, 2022
5d4d620
#381 change owner to bt_admins
chmnata Mar 30, 2022
342f739
#381 add st_linesubstring steps to readme
chmnata Mar 30, 2022
66a8c7b
#381 update create new segment to not use ST_reverse for geom
chmnata Mar 31, 2022
9b1a9c8
#381 update reader status history to use correct version of table
chmnata Apr 5, 2022
d034dfd
#381 move updating readme to main readme
chmnata Apr 5, 2022
277996a
#381 update reader_status_history function
chmnata Apr 6, 2022
5f40600
#381 update bt new segments table with new columns
chmnata Apr 7, 2022
6b6f826
#343 update check to not use case when
chmnata Apr 13, 2022
9c29252
#343 Update bt dag pipeline check task to be idempotent
chmnata May 6, 2022
7d1a558
#250 update function
chmnata Jun 6, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 116 additions & 26 deletions bluetooth/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,44 @@

## Table of Contents

- [Table of Contents](#table-of-contents)
- [1. Overview](#1-overview)
- [2. Table Structure](#2-table-structure)
- [Bluetooth - Bliptrack](#bluetooth---bliptrack)
- [Table of Contents](#table-of-contents)
- [1. Overview](#1-overview)
- [2. Table Structure](#2-table-structure)
- [Open Data Tables](#open-data-tables)
- [Live Feed](#live-feed)
- [Historical Data](#historical-data)
- [Geography](#geography)
- [Live Feed](#live-feed)
- [Historical Data](#historical-data)
- [Geography](#geography)
- [Internal Tables](#internal-tables)
- [Observations](#observations)
- [Filtering devices](#filtering-devices)
- [all_analyses](#all_analyses)
- [ClassOfDevice](#classofdevice)
- [3. Technology](#3-technology)
- [4. Bliptrack UI](#4-bliptrack-ui)
- [Observations](#observations)
- [Filtering devices](#filtering-devices)
- [all_analyses](#all_analyses)
- [ClassOfDevice](#classofdevice)
- [reader_history](#reader_history)
- [reader_locations](#reader_locations)
- [routes](#routes)
- [reader_status_history](#reader_status_history)
- [3. Technology](#3-technology)
- [4. Bliptrack UI](#4-bliptrack-ui)
- [Accessing Bliptrack](#accessing-bliptrack)
- [Terms](#terms)
- [Downloading travel time data](#downloading-travel-time-data)
- [Common Issues](#common-issues)
- [5. Bliptrack API](#5-bliptrack-api)
- [Terms](#terms)
- [Downloading travel time data](#downloading-travel-time-data)
- [Common Issues](#common-issues)
- [5. Bliptrack API](#5-bliptrack-api)
- [Pulling travel time data](#pulling-travel-time-data)
- [Under the Hood](#under-the-hood)
- [The `analysisId`](#the-analysisid)
- [6. Bliptrack API OD Data](#6-bliptrack-api-od-data)
- [Under the Hood](#under-the-hood)
- [The `analysisId`](#the-analysisid)
- [6. Bliptrack API OD Data](#6-bliptrack-api-od-data)
- [Start-End Data](#start-end-data)
- [Some notes on `measuredTime` and records](#some-notes-on-measuredtime-and-records)
- [Dictionary Structure](#dictionary-structure)
- [Some notes on `measuredTime` and records](#some-notes-on-measuredtime-and-records)
- [Dictionary Structure](#dictionary-structure)
- [Others Data](#others-data)
- [deviceClass and outlierLevel](#deviceclass-and-outlierlevel)
- [For the Start-End Data](#for-the-start-end-data)
- [For the Others Data](#for-the-others-data)
- [outliersLevel](#outlierslevel)
- [7. Adding New Segments to the Database](#7-adding-new-segments-to-the-database)
- [8. Open Data Releases](#8-open-data-releases)
- [For the Start-End Data](#for-the-start-end-data)
- [For the Others Data](#for-the-others-data)
- [outliersLevel](#outlierslevel)
- [7. Adding New Segments to the Database](#7-adding-new-segments-to-the-database)
- [8. Open Data Releases](#8-open-data-releases)

## 1. Overview

Expand Down Expand Up @@ -167,6 +172,91 @@ substring(cod::bit(24) from 17 for 6) as minor_device_class,
substring(cod::bit(24) from 12 for 5) as major_device_class
```

As a part of updating/modernizing Bluetooth Reader Tables [ISSUE #196](https://github.com/CityofToronto/bdit_data-sources/issues/196), the following four tables are prepared.
* `reader_history`
* `reader_locations`
* `routes`
* `reader_status_history`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is supposed to be introducing the new tables you've created, but this comment doesn't make sense where it is, which is under the ClassOfDevice heading.
This should be placed under the Internal Tables, and maybe move the sections for the individual tables above the ClassOfDevice section


#### reader_history

This is a table of bluetooth readers which have been installed at each locations at different times in the past. Readers that have been installed at any point of time in the past are listed in this table irrespective of whether the reader is still phycially present at the installed location or not. This table is the sum-total of all the readers irrespective of their current status.
`reader_history` table contains the following fields:

- `reader_history_id`: a unique ID

- `reader_id`: foreign key to `reader_locations`

- `serial_no_bluetooth`: This is a four digit number that is assigned to each bluetooth reader. This serial number corresponds to the **Zone** in the bliptrack table.

- `serial_no_wifi` : Some readers have both wifi and bluetooth sensors. Those readers which has the wifi serial number is available is populated in this field

- `date_installed` : Date the reader is installed at the location

- `date_uninstalled` : Date when reader is uninstalled from a location
radumas marked this conversation as resolved.
Show resolved Hide resolved

Except the `reader_history_id` and `reader_id` all other fields in this table has to be updated manually.

#### reader_locations

This is a table of all locations at which Bliptrack readers have been installed and are physically existing. The installed readers could be online or offline but has NOT been removed physically. Each intersection has only **one** reader that is assigned to a route/routes. Therefore, if there are more than one readers in a locations that have not been removed, such detectors are listed in the `reader_history` table. The function `reader_locations_dt_update` updates the field `date_last_received`. This table consists of the following fields:

- `reader_id`: Unique ID for a unique reader that corresponds to the `reader_id` in the `reader_history` table.

- `name`: For e.g. QN_SP (two characters for E/W street, two characters for N/S street. Whatever name is already existing has been retained for example, A, B, C or Beechwood, Castlefield etc has been retained)

- `int_id`: Centreline intersection id, for closest intersection or pseudo intersections in case of an expressway. A logical location closest to a reader that would be an intersection (Node)

- `date_active`: The date this detector was installed.

- `date_inactive`: NULL unless this field is updated manually to reflect the date when this reader is deemed inactive.

- `date_last_received`: Latest date when the `aggr_5min` table has data aggregated for this reader. This field is updated daily by the function `reader_locations_dt_update`.

- `project_name`: Name of the project by which the detector is installed.

- `geom` : Geometry of the location


#### routes

This is a table of all the routes that pass through the locations (which are either intersections or pseudo intersections) where readers are installed. It corresponds to a unique segments on which data is collected from the network of readers. For a two way street, routes are created for both directions such as Eastbound (EB) - Westbound (WB) or Northbound (NB) - Southbound (SB). In the City of Toronto, bluetooth readers are installed at various locations at different times by different projects. Thus, new routes were created accordingly as more readers are added in new locations. Easy way to create and update routes is [described here](https://github.com/CityofToronto/bdit_data-sources/blob/btdag/bluetooth/update/README.md).

The `routes` table has the following fields:

- `analysis_id`: analysis_id from the `bluetooth.all_analyses` table. For new routes that are added lately, new analysis_ids starting from 1600000 is assigned. _`all_analyses` table has to be updated to include these new routes for data aggregation_.

- `name` : name of the route. This generally contains a detail name explaining the route start and end points. For example, `DVP-J to DVP-I` is a route along Don Valley Parkway between detector **J** and **I**.

- `start_street_name` : This is the name of the street along which the route is created at the start point of the route.

- `start_cross_street` : The street that crosses the start street at the start point of the route.

- `start_reader_id`: Corresponding reader_id from the reader_locations table at the start point of the route.

- `end_street_name` : At times the route can start and end at different street name thus the name of the street along the route where the route ends.

- `end_cross_street` : This is the name of the street where the route ends.

- `end_reader_id`: Corresponding reader_id from the reader_locations table.

- `date_active` : The date when the `aggr_5min` table started aggregating data from this reader.

- `date_inactive` : The date when the reader stopped sending the readings. This field has to be updated manually as either one or both of the readers in a route may temporarily stop aggregating data for few days and come back again.

- `date_last_received`: Last day data on the route is aggregated. This field is updated everyday by the function `insert_report_date`. For the routes that are active, the last reported date will be yesterday.

- `geom` : geometry

#### reader_status_history

This is a table that logs the `last_active_date` for each reader daily. This table is used as a lookup table to identify the readers from which data aggregation did not occur aka `broken_readers` as of yesterday. This table is updated by the function `mohan.reader_status_history(date)`. The function runs daily in `Airflow`. The function `broken_readers(date)` depends on this look up table to identify the readers that were `not active` yesterday but were `active` the day before as `broken_readers`. It has the following four fields.

- `reader_id` : The reader_id unique for readers in each location.
- `last_active_date`: The latest date when the data from the reader was aggregated.
- `active` : Boolean true or false. If the data_aggregation for the reader occured yesterday, this field is true. Else false.
- `dt`: This is the same date that is used as the parameter in the function to update the table. As the function runs daily in airflow, this field contains the most recent date.

## 3. Technology

The BlipTrack sensors have two directional Bluetooth antennas and an omnidirectional WiFi antenna.
Expand Down
22 changes: 22 additions & 0 deletions bluetooth/readersdown/offline_readers.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
reader_id,bt_id,past_status,current_status,name,int_id,project_name,geom
135,4033,online,offline,Islington,13469826,RESCU,0101000020E61000007FFB3A70CEE053C01E1B81785DCF4540
175,5121,online,offline,Spanbridge,13455964,RESCU,0101000020E610000026E4839ECDD453C0137EA99F37DB4540
186,5122,online,offline,York Mills,13450302,RESCU,0101000020E61000005A8121AB5BD553C0B2BAD573D2DF4540
173,5124,online,offline,Royal York,13469715,RESCU,0101000020E6100000F5A276BF0AE053C0857B65DEAACF4540
96,5272,online,offline,CO_OS,13466116,Downtown Expansion,0101000020E610000000790A140CDB53C031CB434BC3D34540
69,5333,online,offline,Mt Pleasant,13454100,Cycle Tracks,0101000020E610000036C8242367D953C00DAB7823F3DC4540
89,5610,online,offline,Brenyon,13442793,NULL,0101000020E610000042CD902A8ACD53C0AA29C93A1CE74540
80,5612,online,offline,HB_QP,30021549,Downtown Expansion,0101000020E6100000AE2E02BE36D953C04AED45B41DD54540
25,5616,online,offline,Glengrove,13455032,NULL,0101000020E61000000F7F4DD6A8D953C0AD1397E315DC4540
77,5819,online,offline,BD_HP,13466332,NULL,0101000020E610000007600322C4DD53C00169FF03ACD34540
140,9018,online,offline,KN_SP,13467505,King Pilot,0101000020E6100000F163CC5D4BD953C07B32FFE89BD24540
162,9021,online,offline,QU_RO,13468436,King Pilot,0101000020E6100000868F882991DC53C0D673D2FBC6D14540
103,9039,online,offline,DU_DF,13466894,King Pilot,0101000020E61000004A7A185A9DDB53C0A034D42824D34540
131,9090,online,offline,HB_BA,13464943,Downtown Expansion,0101000020E610000042FC0D2439DA53C0FA43334FAED44540
98,9093,online,offline,CO_SP,13465489,Downtown Expansion,0101000020E61000009968EB3B9AD953C0EAEA8EC536D44540
74,9096,online,offline,BD_DF,13465260,Downtown Expansion,0101000020E6100000886F1750DEDB53C0C3EFA65B76D44540
183,9097,online,offline,WS_QP,13464585,Downtown Expansion,0101000020E6100000FACCFEE5FFD853C072005878F2D44540
133,9100,online,offline,HB_SP,13464674,Downtown Expansion,0101000020E61000001C171FB0BCD953C06A25BE24DFD44540
105,9279,online,offline,DU_JO,13464003,East Expansion,0101000020E610000067B4554964D553C0BF0CC68844D54540
15,9311,online,offline,C,13468814,NULL,0101000020E61000006C21C84109DB53C074EE76BD34D14540
87,9312,online,offline,BL_RU,13466753,NULL,0101000020E61000009FE5797077DE53C0DEC7D11C59D34540
31 changes: 31 additions & 0 deletions bluetooth/sql/create_tables/add_new_segments.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
CREATE table mohan.bt_segments_new AS (
WITH lookup AS (
SELECT analysis_id, from_id, origin.int_id AS source, to_id, dest.int_id AS target
FROM mohan.new_added_detectors
INNER JOIN mohan.bluetooth_nodes origin ON from_id = origin.bluetooth_id
INNER JOIN mohan.bluetooth_nodes dest ON to_id = dest.bluetooth_id
),
results AS (
SELECT *
FROM lookup
CROSS JOIN LATERAL pgr_dijkstra('SELECT id, source, target, cost FROM gis.centreline_routing_directional inner join gis.centreline on geo_id = id
where fcode != 207001', source::int, target::int, TRUE)
),
lines as (
SELECT analysis_id, street, direction, from_street, to_street, edge AS geo_id, geom
FROM results
INNER JOIN gis.centreline ON edge=geo_id
INNER JOIN mohan.new_added_detectors USING (analysis_id)
ORDER BY analysis_id
)
SELECT analysis_id, street, direction, from_street, to_street,
CASE WHEN geom_dir != direction THEN ST_reverse(geom)
ELSE geom
END AS
geom
FROM (
SELECT analysis_id, street, direction, from_street, to_street,
gis.twochar_direction(gis.direction_from_line(ST_linemerge(ST_union(geom)))) AS geom_dir,
ST_linemerge(ST_union(geom)) AS geom
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the table should also have a length column

FROM lines
GROUP BY analysis_id, street, direction, from_street, to_street) a)
37 changes: 37 additions & 0 deletions bluetooth/sql/functions/broken_readers.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
-- FUNCTION: bluetooth.broken_readers(date)

CREATE OR REPLACE FUNCTION bluetooth.broken_readers(
insert_dt date)
RETURNS void
LANGUAGE 'plpgsql'

COST 100
VOLATILE
AS $BODY$
begin
with x as (
select reader_id, last_active_date as active_date, active as active_two_days_before, dt as check_date
from bluetooth.reader_status_history
where dt = insert_dt - 1 and active IS TRUE
)
,y as (
select *
from bluetooth.reader_status_history
left join x using (reader_id)
where bluetooth.reader_status_history.active IS false and (bluetooth.reader_status_history.dt = insert_dt and last_active_date = insert_dt-2)

)
INSERT INTO bluetooth.broken_readers_log(reader_id, reader_name, check_date)
select reader_id, name, dt
from y
left join bluetooth.reader_locations using(reader_id)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This broken_readers_log is redundant. The same information can be gleaned from the reader_status_history table.
Also, this function doesn't return anything, it is only INSERTing, so the check in the DAG task won't do anything.

where active_two_days_before IS TRUE and active IS false
;
end; $BODY$;

ALTER FUNCTION bluetooth.broken_readers(date)
OWNER TO bluetooth;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bluetooth is not a group or a role, should be bt_admins


GRANT EXECUTE ON FUNCTION bluetooth.broken_readers(date) TO PUBLIC;

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment to describe what this function is doing

GRANT EXECUTE ON FUNCTION bluetooth.broken_readers(date) TO bluetooth;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be either bt_bot or bt_admin_bot

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

going to put it as bt_bot as of now

113 changes: 113 additions & 0 deletions bluetooth/sql/functions/reader_locations_dt_update.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
-- FUNCTION: bluetooth.reader_locations_dt_update(date)

-- DROP FUNCTION bluetooth.reader_locations_dt_update(date);

CREATE OR REPLACE FUNCTION bluetooth.reader_locations_dt_update(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is included in reader_status_history I think this can be deleted

insert_value date)
RETURNS void
LANGUAGE 'plpgsql'

COST 100
VOLATILE
AS $BODY$
begin
--return query
with x AS
(select DISTINCT(analysis_id),
CASE
WHEN MAX (datetime_bin::date)>= (insert_value - 1) then (insert_value - 1)
ELSE MAX (datetime_bin::date)
END
as last_reported,
CASE
WHEN MAX (datetime_bin::date)>= (insert_value-1) then 'True'::text
else 'False'::text
END
AS route_status
,CASE
WHEN max(aggr_5min.datetime_bin::date) >= (insert_value::date - 1) THEN insert_value::date - 1
ELSE max(aggr_5min.datetime_bin::date)
END AS dt
from
bluetooth.all_analyses
LEFT JOIN bluetooth.aggr_5min USING (analysis_id)
GROUP BY analysis_id)
, y AS (
SELECT all_analyses.analysis_id,
(all_analyses.route_points -> 0) ->> 'id'::text AS start_route_point_id,
(all_analyses.route_points -> 0) ->> 'name'::text AS start_detector,
(all_analyses.route_points -> 1) ->> 'id'::text AS end_route_point_id,
(all_analyses.route_points -> 1) ->> 'name'::text AS end_detector
FROM bluetooth.all_analyses
)
, z AS (
SELECT x.analysis_id,
x.last_reported,
x.route_status,
x.dt,
y.start_route_point_id,
y.start_detector,
y.end_route_point_id,
y.end_detector
FROM x
LEFT JOIN y USING (analysis_id)
)
, b AS (
SELECT z.analysis_id,
z.start_detector AS detector_name,
z.route_status,
z.last_reported
FROM z
)
, c AS (
SELECT b.analysis_id,
b.detector_name,
b.route_status,
b.last_reported
FROM b
UNION
SELECT z.analysis_id,
z.end_detector AS detector_name,
z.route_status,
z.last_reported
FROM z
)
, active AS (
SELECT DISTINCT c.detector_name,
max(c.last_reported) AS max,
c.route_status,
bluetooth.detectors_history_final.reader_id as id
FROM c
LEFT JOIN bluetooth.detectors_history_final ON c.detector_name = bluetooth.detectors_history_final.read_name::text
WHERE c.route_status = 'True'::text
GROUP BY c.route_status, c.detector_name, bluetooth.detectors_history_final.reader_id
), final as (
SELECT DISTINCT c.detector_name,
max(c.last_reported) AS max,
c.route_status,
bluetooth.detectors_history_final.reader_id
FROM c
LEFT JOIN bluetooth.detectors_history_final ON c.detector_name = bluetooth.detectors_history_final.read_name::text
WHERE c.route_status = 'False'::text AND NOT (c.detector_name IN ( SELECT active.detector_name
FROM active))
GROUP BY c.route_status, c.detector_name, bluetooth.detectors_history_final.reader_id
UNION
SELECT active.detector_name,
active.max,
active.route_status,
active.id
FROM active)
, date_refresh as(
SELECT max(max) as date_last_received, reader_id
from final
where reader_id in (select reader_id from bluetooth.reader_locations)
group by reader_id
order by reader_id)

UPDATE bluetooth.reader_locations
set date_last_received = (SELECT date_last_received from date_refresh where date_refresh.reader_id = bluetooth.reader_locations.reader_id)
;
end; $BODY$;

ALTER FUNCTION bluetooth.reader_locations_dt_update(date)
OWNER TO bluetooth;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to bt_admins

Loading