Write scripts to run basic text reuse pipeline

[mnaydan]

I/O

• Generate a minimal Chadwyck-Healey text corpus file (.jsonl) (limited to poems in the test set)
  • Write script
  • Test script output (i.e. jsonl file)
  • Create initial .jsonl file for the Chadwyck-Healey test set.
• Build minimal ppa page corpus using old OCR for Gale volumes.

Passim-specific

• Initial script(s) for running passim
  • Write script to convert input text .jsonl files to a passim-suitable format
  • Initial bash script for running running passim
  • Write script for standardizing passim output
    • Minimal output
    • Optionally include original/aligned text excerpts for manual evaluation

Evaluation

• Get passim results for test set with default parameters
  • "Raw" passim output (i.e. various files within a specific (top-level) output directory)

[mnaydan]

@laurejt I am testing the jsonl. Could you please reply with a comment here specifying the acceptance criteria? Are new lines important? What else am I looking for?

[laurejt]

@mnaydan The immediate goals of testing the jsonl is to exam the "text" field and confirm that it corresponds to the "full text" of the poem. I'm not sure how best to compare this beyond finding an external copy from somewhere else and checking that it "looks" right.

[mnaydan]

@laurejt thank you, that is helpful! I used Visual Studio Code and json-lines-viewer.preview to read the jsonl, and spot checked a dozen or so poems. The text field does correspond to the full text of the poem as I would expect it, so I would consider this "tested" and "accepted."

[jerielizabeth]

Issues split out from this task

• Integrate Chadwyck-Healey corpus building code into corppa #83
• Integrate passim-related code into corppa #84
• Refine Passim methods and parameters #72
• write evaluation scripts for comparing reference annotations and model annotations  #71