Migrate enrich_date and enrich_earliest_date enrichments

[amywieliczka]

Should be migrated as methods on the Record class in metadata_mapper/mapper.py

Related to #196

[amywieliczka]

I performed some analysis of the usage of enrich_earliest_date and enrich_date enrichment functions, and then migrated these enrichments from calisphere-legacy-harvester/dpla-ingestion as part of this PR: #1090

Next steps:

• This is simply a theoretical migration of the enrichments, and no harvests have actually been run through these migrated enrichment functions. We need to actually harvest some collections in a local dev environment to make sure the code actually works. Unfortunately, my aws-mwaa-local-environment has been borked, and I haven't been able to get a dev environment working again to test out these code updates.
• Once we know the code works, then we also need to make sure the code does what it's supposed to. As far as I can tell, we can either:
  • Get the validator working again (may need to resolve whatever error is thrown in MWAA) against Solr (beanstalk solr is still alive and kicking) and Couch (may need to start up ingest-couch-prd, ingest-front-prd) so that we can validate the facet_decade, sort_date_start, and sort_date_end fields for records - maybe even get this working in deployed mwaa, so @aturner and @christinklez can validate these fields for a handful of 5-10 sample collections
  • Run 5-10 collections through a re-harvest process and QA those collections on calisphere-stage with a special eye for the decade facet, date metadata value, and sort_date_start/sort_date_end usage using sort.
  • Maybe write a one-off date validation script to compare these handful of date-related fields (date, facet_decade, sort_date_start, sort_date_end), but the validator already does a really good job of comparing records, and already is designed to talk to Solr.
• Once we've determined that the code mostly works and mostly does what it's supposed to (even if it's not an exhaustive test), then we need to determine either a) a migration path to update records already in the index or b) a re-harvesting plan to progressively pick up decade data for harvests on re-harvest of the collections.
  • Probably a re-harvesting plan is best, following standard QA practices that include a special glance at the decade facet for collections as we re-harvest them. Could possibly even just restart harvests from the most recently fetched metadata by clearing the mapping tasks onward in the harvest DAG - this would prevent inadvertently refreshing anyone's metadata that they did not want refreshed yet.
• [No further work needed] Update the Calisphere Prod UI to facet and filter on the facet_decade field instead of the date field (Stage ucldc/public_interface#405)
• [No further work needed] Confirmed that Calisphere UI already uses sort_date_start and sort_date_end to support the sort options - no changes needed for this, though as more records have these fields, we'll get better sorts.

[amywieliczka]

Collection Date Samples - issue #1093 - https://docs.google.com/spreadsheets/d/1JNw7ynxCJS8d4f5W4Glwwqx0qMm8ss-aB_GpjVxN1dI/edit?gid=0#gid=0

[amywieliczka]

Completed since last update:

• Got the code actually working and running in my local mwaa runner (fixed my local mwaa runner)
• Got the validator working again
  • Made validator conditional on the presence of UCLDC_SOLR_URL and UCLDC_COUCH_URL
  • Commented out is_shown_at and is_shown_by validations (these validated against couch, which is no longer up)
• Ran the validator for 27 test collections - filter on Calisphere Solr Mapper Type = No
• Using those 27 test collections, created a set of test cases to enrich_date, enrich_earliest_date, get_facet_decades, make_sort_dates, and unpack_display_date. To run these tests, from within the rikolti folder, run pytest metadata_mapper

Next steps:

• Code Review of the PR
• Deployment to MWAA (this will involve a new pip install & an update to startup.sh) & verify conditional validation logic is working well
• Create a re-harvesting plan to progressively pick up and QA date data. (Could potentially re-harvest from fetched metadata stored in our s3 bucket)

Later steps

• Create documentation describing the kinds of dates Rikolti can parse, date best practices and recommendations - for contributors #1106 - given the highly conditional and varied paths date parsing takes, this is prohibitively difficult at the moment.