Update to use new data

[ghing]

Hannah Cushman at datamade pointed out that the state released new IUCR codes. We should update this library to reflect the new codes.

[hancush]

Hi!

This (.xlsx) looks like the best and most recent resource via the Illinois State Police, dated December 29, 2016.

I did some quick and dirty munging, looks like there are about 75 codes in the update that aren't in the lib's data (.csv). (Probably a good idea to cq this. 😅)

The statutes may take more time to reconcile because I'm not sure we can assume they're in the same format.

[ghing]

@hancush Thanks! Do we have to worry about IUCRs mapping to different ILCS codes? Or is this something that we should verify?

[hancush]

Yes, in some cases the ILCS code may have been updated or repealed. Luckily old IUCR codes are "retained for historical purposes" in the data.

Here's the key provided by ISP (emphasis added):

The statutes associated with many offense codes have been repealed over the past six years. These offense codes have been ‘retired’ and retained for historical purposes only. ‘Retired’ offense codes will never be reassigned to a different offense. ‘Retired’ offense codes are denoted with the phrase ‘retained for historical purposes’ in the ILCS reference field.

Some offense codes had very minor changes to either the name of the offense, or the ILCS reference. These changes are denoted in bold black font. The entire ILCS reference was placed in bold though the change may have been as minor as adding or removing a suffix.

In only a few instances an offense code was both re-named and the ILCS reference updated. This only occurred when the revised offense mirrored the previous offense.
Example:
4410 previous Destruction of Draft Card
4410 revised Mutilation of Draft Card

Additional offense codes have been created and are represented in blue font.

Not every ILCS statute is represented in the revised code table. It will continue to be the practice of the I-UCR Program to refrain from creating an offense code for an offense that would never be reported in any of the I-UCR program reporting elements. Agencies are encouraged to create in-house codes, using the 6000 series or higher, for offenses outside the realm of UCR reporting.

Agencies are encouraged to begin use of the revised offense codes immediately.

(The blue/bolding is in the source .xlsx I linked above.)

[hancush]

Did a full diff of the existing and updated codes. This output is all codes where the ILCS statute is different in the update than it is in the existing data. Some just get more specific (including (a)/(b), etc.) and some are entirely different statutes. I can sort thru by hand.

(Makefile and comparison script plus source data also in that gist.)

[hancush]

Reading back over the notes, I had some thoughts.

If the primary objective of this lib is to convert ILCS statutes to IUCR codes, preserving historical code/statute pairs as we update the data is in our best interest.

I propose adding two columns to the underlying data:

• year_published – Year the source data was published.
• year_retired – If a code is retired, year the data indicating its retirement was published, i.e. these codes would have a year_retired value of 2016. Null if code is active as of the most recent update.

We could also update the API, taking a leaf from census' book by returning results from the most recent update by default but allowing a year to be passed for older data (h/t @fgregg).

Does this seem reasonable, @ghing?