Home

Regular tasks

You should read the overview and documentation below before performing these tasks. They are only at the top of the page for easy reference by regular users.

• Tasks: Data maintenance
• Tasks: Represent CSV Schema
• Tasks: Request boundaries
• Tasks: Respond to emails
• Tasks: Miscellaneous
• Tasks: Updating Census data
• Troubleshooting

Overview

Browse the Represent website and read its API documentation to start.

At a high level, the represent-canada Django project and its three Django apps run represent.opennorth.ca and rarely change. Most of the work is feeding them data for boundaries, representatives, and postcodes.

The service is heavily automated and can be maintained by a single highly trained developer. The service's data should be correct, complete, consistent, timely, and from authoritative sources. Many checks exist to ensure these service standards are maintained.

Data sources

The Django project imports boundary data by cloning two repositories: one public and one private. Read Tasks: Request boundaries for more.

The Django project imports representative data by requesting the data_url of each representative set. The data_url points to scrapers_ca_app, which runs on Heroku.

scrapers_ca_app wraps scrapers-ca, which contains HTML and CSV scrapers. HTML scrapers are more expensive to maintain, so we regularly request municipalities with open data catalogs to publish CSV data.

In rare cases, we work with external partners to manually fill a Google Sheet whose headers match our CSV Schema. We do this rarely, because our automated systems can't determine when that manual data goes stale and because the source is non-authoritative. We have done this for provincial elections and for major users who want to run a large campaign against multiple municipalities; in the latter case, we create a representative set with a slug like campaign-set-# to indicate to other users that it is an irregular representative set. The Google Sheets are typically imported using CSV scrapers in scrapers-ca, in which we track the contact person and the creation date to measure timeliness. We collect the Google Sheets into a folder.

Other pieces

Open Civic Data Division Identifiers (OCD-ID) hold many pieces together, as described here. Without OCD-IDs, it would be difficult to maintain a consistent and coherent database.

We have also created API clients, including a Ruby client, Drupal module and WordPress plugin. Since the API has not changed in years, these have required no maintenance.

Documentation

Read the documentation of:

• Represent API
• represent-canada: the represent.opennorth.ca Django project, which depends on:
  • represent-boundaries: Django app
  • represent-postcodes: Django app
  • represent-reps: Django app
  • represent-canada-data: public boundary data
  • represent-canada-private-data: private boundary data
• scrapers_ca_app: the scrapers.herokuapp.com Django project, which uses:
  • scrapers-ca

Deployment

In addition to the instructions below, add a new maintainer as a collaborator on Heroku.

Read the deployment instructions of:

• represent-canada on dedicated servers
• scrapers_ca_app on Heroku (scrapers)

Other deployments are:

• scrapers-ca-ruby on Heroku (scrapers-ruby)
  • Publishes MP's Twitter accounts, used by the scrapers-ca ca scraper