Introduce profile-guided optimization to APM Server

[1pkg]

Context

Go compiler added support for profile-guided optimization (PGO) since Go 1.20, thus allowing to build further optimized Go binaries using CPU profiles. Depending on the workload, optimizations and the profile quality one can expect to extract somewhere from 2-14% of performance gains according to go.dev/blog which is quite meaningful. Further more for CPU bound services, in general, the expected results should be somewhere in the upper bound of the expected performance gains.

Requested Changes

We should capture this performance improvement in APM Server.

We should consider how we will collect the profiles, initially we could leverage existing benchmarks workflow to collect CPU profiles which will be included into the future PGO builds.

We should consider how we will store profiles and include them into the builds, the simplest and most convenient approach to commit them directly in the source code for easy distribution and repeatable builds.

Additional Links

go.dev/doc/pgo
go.dev/blog/pgo

[1pkg]

The existing benchmarks in APM Server are highly coupled to the ElasticSearch stack, making them over indexing ES performance into the benchmark results. This could lead to a large distribution between individual benchmark results, up to 20%, and make them hard to measure and reason about. This is a particular problem for PGO, since PGO only adds an incremental gain to the performance. In my synthetical local test when benchmarking an isolated instance of APM Server the result was an average 5% increase in the throughput. These results are really hard to observe when the existing benchmark hide them between ES performance.

The solution to this problem would be to use a separate second set of benchmark workflow that targets a http API stub instead of real ES instance. This way we should be able to sufficiently isolate APM Server performance from underlying ES performance.