Profile-Guided Optimization (PGO) benchmark result

[zamazan4ik]

Hi!

I do ongoing PGO research on different applications (including many parsers like JSON parsers) - all results are available at https://github.com/zamazan4ik/awesome-pgo . I performed some PGO benchmarks on the struson too and want to share my results here.

Test environment

• Fedora 39
• Linux kernel 6.7.6
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.76
• struson version: main branch on commit 3f5de617c4245baf8f0d78c3af560c73446bdeb7
• Disabled Turbo boost for improving consistency across runs

Benchmark

Release benchmarks are done with tasket -c 0 cargo bench, PGO training phase with tasket -c 0 cargo pgo bench, PGO optimization with tasket -c 0 cargo pgo optimize bench.

taskset -c 0 is used for better benchmark consistency. All PGO-related routines are done with cargo-pgo. All benchmarks are done on the same machine, with the same hardware/software during runs, with the same background "noise" (as much as I can guarantee, of course).

Results

Here are the results:

• Release: https://gist.github.com/zamazan4ik/dd60b963378e81c85a4aa7533a8f3807
• PGO-optimized: https://gist.github.com/zamazan4ik/c505fe6a2d12c4ad81557d00f34ccbfd
• (just for reference) PGO instrumented: https://gist.github.com/zamazan4ik/297c84304d45564c15fab6882f111c34

At least in the provided by project benchmarks, there are measurable improvements in many cases. However, also there are some regressions.

If you think that results are helpful, maybe mentioning using PGO with struson somewhere in the README will be a good thing to do.

[Marcono1234]

Thanks a lot for this information! I will have to read a bit to become familiar with this but it sounds quite interesting. At the moment I think there are likely some code changes I could do first to improve performance. And I will also need to take a look at how users are using Struson to be able to better understand where users consider Struson to be useful; the current benchmarks might not be realistic enough. So I am not sure how much impact PGO has on real user data.

Just to make sure I am not misunderstanding this topic; this is nothing I can / should set up for Struson itself (and which then automatically applies to users of the library), but you are suggesting that I recommend in the README that users should give PGO a try for their applications using Struson, since it could speed up their applications. Right?

I will probably not add it to the README at the moment because speed is not the main focus yet, but I will keep this information about PGO in mind for when this library has evolved a bit more.

[zamazan4ik]

Just to make sure I am not misunderstanding this topic; this is nothing I can / should set up for Struson itself (and which then automatically applies to users of the library), but you are suggesting that I recommend in the README that users should give PGO a try for their applications using Struson, since it could speed up their applications. Right?

Yep, you are right!

I will probably not add it to the README at the moment because speed is not the main focus yet, but I will keep this information about PGO in mind for when this library has evolved a bit more.

Thank you!