Scrape schedule versions

[dcjohnson24]

Description

A first attempt at scraping the schedule versions from transitfeeds.com. See #30 and #36

Type of change

• Bug fix
• New functionality
• Documentation

How has this been tested?

Locally. Calling the function create_schedule_list(5, 2022) will re-create the original list of schedule_feeds, in addition to the schedule versions and their date ranges up to the current version.

In [9]: create_schedule_list(5, 2022)
INFO:root: Searching page 1
INFO:root: Searching page 2
INFO:root: Found schedule for May 2022
INFO:root: Adding schedule for May 7, 2022
Out[9]: 
[{'schedule_version': '20220507',
  'feed_start_date': '2022-05-20',
  'feed_end_date': '2022-06-02'},
 {'schedule_version': '20220603',
  'feed_start_date': '2022-06-04',
  'feed_end_date': '2022-06-07'},
 {'schedule_version': '20220608',
  'feed_start_date': '2022-06-09',
  'feed_end_date': '2022-07-08'},
 {'schedule_version': '20220709',
  'feed_start_date': '2022-07-10',
  'feed_end_date': '2022-07-17'},
 {'schedule_version': '20220718',
  'feed_start_date': '2022-07-19',
  'feed_end_date': '2022-07-29'},
 {'schedule_version': '20220730',
  'feed_start_date': '2022-07-31',
  'feed_end_date': '2022-08-10'},
 {'schedule_version': '20220811',
  'feed_start_date': '2022-08-12',
  'feed_end_date': '2022-08-12'},
 {'schedule_version': '20220813',
  'feed_start_date': '2022-08-14',
  'feed_end_date': '2022-08-16'},
 {'schedule_version': '20220817',
  'feed_start_date': '2022-08-18',
  'feed_end_date': '2022-09-07'},
 {'schedule_version': '20220908',
  'feed_start_date': '2022-09-09',
  'feed_end_date': '2022-09-17'},
 {'schedule_version': '20220918',
  'feed_start_date': '2022-09-19',
  'feed_end_date': '2022-09-28'},
 {'schedule_version': '20220929',
  'feed_start_date': '2022-09-30',
  'feed_end_date': '2022-10-06'},
 {'schedule_version': '20221007',
  'feed_start_date': '2022-10-08',
  'feed_end_date': '2022-10-11'},
 {'schedule_version': '20221012',
  'feed_start_date': '2022-10-13',
  'feed_end_date': '2022-10-19'},
 {'schedule_version': '20221020',
  'feed_start_date': '2022-10-21',
  'feed_end_date': '2022-10-22'}]

[lauriemerrell]

Sorry to always nitpick comments but they are super helpful to me!

[lauriemerrell]

Just checking here - "first" means "the one that appears first in the order of display on the Transitfeeds site, which was actually scraped later"? (That would be the correct implementation IMO, just wasn't sure whether "first" means chronologically or in display order which is reverse chronological)

[dcjohnson24]

Yes "first" here means the first in the order of display on Transitfeeds.

date_list appears as

[...'14 September 2021', '7 September 2021', '1 September 2021',
  '1 September 2021', '1 September 2021', '1 September 2021',
  '1 September 2021', '2 August 2021', '15 June 2021',..]

So taking the first will take the latest version. I've added a note on that in the comments.