HTTP2 API calls from prefect server breaking flow runs with large amounts of async tasks

[2tony2]

Bug summary

We have a production workflow that will perform massive backfills againt an external api.
For this it generates up to tens of thousands of asynchronous task runs in a flow run.

A lot of flow runs were suddenly crashing last week at random intervals with a keyerror which seems to be related to the following #11660.

We're running prefect 2.19>.

When we set the env variable to turn of HTTP2 for the prefect API PREFECT_API_ENABLE_HTTP2 = False this completely resolves the random crashes.

Just wanted to raise this because it is definitely strange to suddenly have to manually disable prefect's HTTP2 API to prevent crashes.

Version info (prefect version output)

we're running prefect 2.19.X

Additional context

No response

[williamjamir]

Looks like on Prefect 3 HTTP2 is disable by default: #15311