Too many scroll contexts in Elasticsearch due to scroll requests from ES fetchers

[lahsivjar]

APM Server version (apm-server version): All APM servers using ES fetchers

Description of the problem including expected versus actual behavior: Too many scroll contexts are getting created in Elasticsearch. Looking at the code, we don't seem to be clearing the scroll context when the refresh encounters an error which could lead to scroll contexts piling up.

Steps to reproduce:

We can probably observe this locally by simulating an error in the metadata fetcher's refresh call (ref).

Provide logs (if relevant):

In a deployment reporting this error a lot of refresh cache error: context deadline exceeded can be observed.

[lahsivjar]

Putting it as high impact and urgency because, IIANM, APM-Server hogging all the scroll context limit in Elasticsearch might cause other search requests relying on scroll context to start failing - this is based on my understanding so far.

[kruskall]

Thanks for opening an issue! 🙇

Not to argue impact/urgency but it's worth adding that according to https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#clear-scroll search context are removed after the scroll timeout (which we are setting) so they shouldn't leak forever

[lahsivjar]

search context are removed after the scroll timeout (which we are setting) so they shouldn't leak forever

Fair point, I think there is still a missing link here as it doesn't make sense why the scroll contexts will accumulate if they should timeout.

[kruskall]

PR merged