bug-1921849: support elasticsearch 8

[relud]

use ELASTICSEARCH_MODE=PREFER_NEW to make the webapp use es8 and the processor write to both es 1.4 and es8

[willkg]

We're going to do this PR in two passes. This is a code read pass.

While you're fixing things I brought up, I'll spend some time going through some manual testing for things I'm wondering about.

Then after you make changes, I'll read through those and add anything that came up in manual testing.

[relud]

this section removes fields that cause document_parsing_exception and retries the document. This seems like an odd choice given that it happens after value fixing occurs, which should already be preventing the three types of failure we catch here. The only way i can think of to reach this code block in production is if we are writing to a field not in our mapping.

[willkg]

It used to be the case that it wrote all the data into Elasticsearch even if it wasn't in the mapping. That way when they add new fields to the crash report, they'd get indexed even if Socorro didn't explicitly have support for it. While the intentions were good, that was terrible so I changed it such that it only indexes what's defined in super search fields and in the mapping.

[willkg]

@relud Can you pull in the fix in PR #6813? Then I can go through this PR again.

[relud]

@relud Can you pull in the fix in PR #6813? Then I can go through this PR again.

done

[willkg]

I went through and tested the issues I raised in the previous review.

1. bin/process_crashes.sh works now.
2. Things correctly wait for both es and legacy_es containers to start up.
3. TopCrashers works now.

I wrote up bug 1933824 about a curiosity I hit when uploading some crash report dump files to gcs emulator. That's not related to these changes.

My only issue is that I'm not sure why you added four new metrics to statsd_metrics.yml rather than just the one I was hitting issues with.

Everything else looks fine as far as I can tell.

r+wc