Btdag #381
Btdag #381
Conversation
dags/check_brokenreaders.py
Outdated
|
|
||
| def bad_readers(con, run_date): | ||
| with con.cursor() as cursor: | ||
| select_query = """SELECT reader from mohan.broken_reader()""" |
There was a problem hiding this comment.
shouldn't the date be incorporate as a parameter to this function?
Also the sql for this function isn't in this branch
There was a problem hiding this comment.
forgot about SQL altogether. Will add it to the branch too. Yes date should be a parameter. Will change that
dags/check_brokenreaders.py
Outdated
| logging.basicConfig(level=logging.DEBUG) | ||
|
|
||
| #To connect to pgadmin | ||
| CONFIG = configparser.ConfigParser() |
There was a problem hiding this comment.
I should set up a bt bot account for Airflow so then the credentials would be imported
https://github.com/CityofToronto/bdit_data-sources/blob/master/dags/pull_wys.py#L59-L60
dags/check_brokenreaders.py
Outdated
| import configparser | ||
|
|
||
| LOGGER = logging.getLogger(__name__) | ||
| logging.basicConfig(level=logging.DEBUG) |
There was a problem hiding this comment.
Airflow is already taking care of the logging configuration, the code should simply call logging.info, etc, further down.
dags/check_brokenreaders.py
Outdated
| :red_circle: A bt Reader may be broken! | ||
| *Task*: {task} | ||
| *Dag*: {dag} | ||
| *Execution Time*: {exec_date} | ||
| *Log Url*: {log_url} |
There was a problem hiding this comment.
This should be updated to our more recent Slack message style.
See for example
https://github.com/CityofToronto/bdit_data-sources/blob/master/dags/pull_wys.py#L28-L44
dags/bt_read_history.py
Outdated
|
|
||
| ## Postgres update bluetooth reader_status_history DAG | ||
| # Task to update the reader_status_history table daily | ||
| bt_status_history = PostgresOperator(sql='''SELECT * from mohan.reader_status_history(current_date)''', |
There was a problem hiding this comment.
This isn't going to work because current_date isn't a template parameter.
See for example https://github.com/CityofToronto/bdit_data-sources/blob/master/dags/refresh_wys_monthly.py#L72 and our documentation "How to use {{ ds }}"
In short it should just be {{ ds }} here
|
@webgisgeek Hi Mohan, the readme is pretty clear, I edited a few minor things but there were some things that I didn't understand:
|
bluetooth/update/README.md
Outdated
| ``` | ||
| Draws lines using the reader locations. This is the result. | ||
| Check that correct intersections are returned from this query especially for oblique intersections with an offset. If required, correct the intersection_id and geom for such intersections and finalize the table `mohan.bluetooth_nodes`. An example is shown below. The intersection between Dundas St W, Roncesvalles Ave and Boustead Ave is oblique with an offset. The BT reader 5263 (red dot) picked up intersection_id `13466305` (greed dot) as nearest intersection to the reader. But the planned route does not go through this intersection_id. The correct intersection_id for this reader is `113466258`. The intersection_id and geom for this redear was therefore corrected manually. |
| INTO rliu.new_bt_coflate1000 | ||
| FROM conflation | ||
| SELECT analysis_id, street, direction, from_street, to_street, | ||
| CASE WHEN geom_dir != direction THEN ST_reverse(geom) |
There was a problem hiding this comment.
not sure if this is needed because we are using centreline with direction as a base network for which the direction of the line is already flipped. This might result in double flipped geometries if the dirs didnt match because of a NB vs WB comparison
bluetooth/update/README.md
Outdated
|
|
||
|  | ||
| ## Things to note | ||
| Geostatistical lines and planning boundaries need to be avoided while pgrouting. |
There was a problem hiding this comment.
also railways and rivers and stuff
bluetooth/update/README.md
Outdated
|
|
||
| ## Cleanup, Warnings, and Alternative Methods | ||
| ## Validating Output | ||
| Validate the length of the segments with length ST_length(geom) and direction using gis.direction_from_line(geom) functions.If the detectors are located very close to the centerline intersections, it is not necessary to do the centreline cutting. If any bluetooth detectors are not located at the start or end point of a centreline, we will need to cut the centreline using ST_linelocatepoint() as explained in [here.](https://github.com/CityofToronto/bdit_data-sources/issues/234) . |
There was a problem hiding this comment.
I think it would be helpful to include the steps to cut the line here in the readme instead of linking the issue.
|
|
||
| At this point, the layer has multiple entries containing centreline segments for each analysis_id. Instead, we need a single line/entry for each analysis_id. This query concatenates the individual centreline segments into one line. The main function doing this work is [`ST_LineMerge()`](https://postgis.net/docs/ST_LineMerge.html) | ||
| The new routes table is now ready to append to the existing routes table. |
There was a problem hiding this comment.
The bluetooth.segments table has the columns:
- segment_name ,
- analysis_id,
- street ,
- direction,
- start_cross ,
- end_cross ,
- length ,
- bluetooth ,
- wifi ,
- geom ,
- duplicate ,
- reversed ,
- end_street
So its missing:
- segment_name , bluetooth, widi, duplicate, revered, end_street
| @@ -0,0 +1,40 @@ | |||
| # Bluetooth reader's status | |||
There was a problem hiding this comment.
Maybe this can go under the readme?
|
List of modified things:
DAG: README Things that needs to be fixed:
|
|
Looking to move the |
|
|
||
| -- Combine active and deactive detector | ||
| , detector_status as ( | ||
| SELECT DISTINCT detector_list.detector_name, |
There was a problem hiding this comment.
should select distinct on reader_id instead of detector_name, because we are inserting reader_id's status at the end of the query and multiple detector_names can be refer to one reader_id
-- return 216 rows
select distinct reader_id, read_name from mohan.detectors_history_corrected
-- return 178 rows
select distinct reader_id from mohan.detectors_history_corrected
|
|
||
| INSERT INTO bluetooth.reader_status_history (reader_id, last_active_date, active, dt) | ||
|
|
||
| SELECT DISTINCT reader_id, max(last_reported), route_status, dt |
There was a problem hiding this comment.
I don't think this select statement is necessary because the previous cte already remove possible dups
|
|
||
| SELECT DISTINCT reader_id, max(last_reported), route_status, dt | ||
| FROM detector_status | ||
| WHERE reader_id IS NOT NULL |
There was a problem hiding this comment.
there is no null reader_id in the detector_history table
|
Working on the new segments tables and adding the missing columns: segment_name , bluetooth, wifi, duplicate, reverse, end_street |
|
Just in case future spelunkers come here. |
|
In bluetooth.routes_temp, there are 61 routes out of 402 with no data in
But it does seems like some routes with data do have coverage on those routes with no data. (Green dotted lines are the ones without data and grey lines are the ones with data) Ones that still have coverage (or partial coverage):
|
|
Checking the distinct status in reader_history table,
The following readme explained the meaning behind each status, kindddd of feels like the below 3 is confusing or just keeping a log of duplicated data/data of human error.
|
What this pull request accomplishes:
Issue(s) this solves:
documentation on creating new bt routes
What, in particular, needs to reviewed: