Clean socrata data

[galbwe]

• In the data scraped from socrata api:
  1. names should not be in all caps
  2. nonprofits that have obvious religious affiliations based on their names should be filtered out
• cleaning can be done as part of the find_socrata_api_leads task (in tasks.py), or can be done in a separate task in tasks.py that runs after find_socrata_api_leads

[Nova791]

Clean Socrata Data

[kaleeaswari]

@galbwe Can I get this assigned?

[galbwe]

Hey @kaleeaswari if you can focus on the Colorado Non-profits (CNP) data, that would be the most helpful. We currently aren't using the socrata dataset because it was missing most of the relevant information for the app.

The files you will want to look at are scrape_CNP.py and tasks.py. A good start would be to filter out the records that are missing all of the fields needed for the the table on the homepage. There are other things to do like making sure strings are formatted consistently, and removing records for nonprofits with obvious religious affiliations.

[kaleeaswari]

Mandatory fields to consider a lead : Name, Contact, Website, SocialMedia. Is that correct?

@galbwe

[galbwe]

I think they all have the Name field. If a lead is missing Contact, Website, Facebook, Twitter, Instagram, and LinkedIn, then it should be dropped.