Add batch processing and report summaries

[rfdj]

In this PR, we added several features:

• batch processing of entire directories
• produce a list of the differences between GT and OCR and their number of occurrences. This list is shown in both the JSON and the HTML reports (sortable).
• parsing sets of generated JSON reports, calculating average scores and aggregating a final list of differences. A threshold option was added to prevent the HTML report becoming too large to open.

The README has been updated accordingly.
This allows us to easily evaluate OCR results in a more systematic way. Please let us know if it's of any interest to you.

[mikegerber]

Hi Ruud,

thanks for the PR! Very good ideas!

I think I am going to add this, but I also believe it needs some more work:

A)

• The average CER calculated here is what I would call a macro average and, depending on the intended use case, does not weigh in the length of the texts ("micro average"). This is the reason why the JSON output includes the length so this can be calculated.
• It also does not account for possible "infinite CER" cases (may happen when len(GT)==0)
• We also need support for summarizing inside and over OCR-D workspaces

B)

• the list of differences may fit Feature request: list with error frequencies in report #18 but I need to look into it further

I'll work on A first (possibly leaving out OCR-D for now)

[mikegerber]

Working in https://github.com/qurator-spk/dinglehopper/tree/pr-83 (can't push to INL:feat/batch-processing, it seems).

[mikegerber]

@rfdj Could you check if you can "Allow edits from maintainers" for this PR? (It may not be possible due to https://github.com/orgs/community/discussions/5634, though)

[mikegerber]

I've fixed a bug in dinglehopper-summarize when the reports do not contain any difference statistics here:

7c323e1

[mikegerber]

@rfdj Because I'm going on vacation I decided to merge the PR as is (to not block it for more weeks) and will take care of the points I mentioned after my vacation :)

Thanks for the contribution, I think this will be useful for the users!

[rfdj]

I was away for a few days myself, but let me know if I can still be of assistance somewhere when you come back.

[mikegerber]

I've noticed that I had a small fix for this in 7c323e1 (= f077ce2 in master) that I hadn't merged yet and did so today. (Summarizing threw an Exception if the reports didn't have the difference stats.)