startup RSS increase in both native and JVM mode

[franz1981]

Describe the bug

Using https://github.com/quarkusio/quarkus-quickstarts/tree/development/getting-started startup tests in native-mode (but it applies to JVM mode well) with a 4 cores machine:

80ea941	0782ecc	2d19846
41.600 MB	42.236 MB	43.960 MB

Which trend is pretty obvious in our perf CI is (still native image case):

The RSS X CPU part is related how many cores the machine starting quarkus is using (if via docker --cpus=X options or can be done manually via taskset or numactl).

The actual measurement related RSS can be performed via a script like this, since we can focus on the case RSS 4 (which allow to be run from laptops too):

taskset -c 0,1,2,3 java -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xmx128m -Dquarkus.vertx.event-loops-pool-size=2 -Dquarkus.vertx.worker-pool-size=4  -jar target/quarkus-app/quarkus-run.jar &
q_pid=$!
sleep 4
echo "RSS:"
sudo pmap -x ${q_pid} | grep total | awk '{print $4}' | sed 's/[^0-9]*//g'
echo "VMRSS:"
sudo cat /proc/${q_pid}/status | grep VmRSS | awk '{print $2}'
kill -SIGTERM ${q_pid}

In case docker is used, retrieving the PID can be done via

docker ps --format="{{.ID}}" | xargs docker inspect | jq '.[].State.Pid'

which could add a filtering on the docker name instance via

`--filter="name=${{QUARKUS_NAME}}`

if the container has some specific name assigned.
If these info to reproduce it are not sufficient I can create a repository for convenience.

[quarkus-bot]

/cc @zakkak (native-image)

[geoand]

Thanks for opening this!

Do you perhaps have any flamegraphs or other measurements that could indicate what is causing the issue?

[franz1981]

No clues, nor I have any cycles to do it :"/
I have shared already some information with @radcortez that allocations flame graphs doesn't show anything relevant, in JVM mode.
A possible way to troubleshoot this would be to use the analysis from @galderz at https://quarkus.io/blog/native-startup-rss-troubleshooting/ - especially collecting the jdk.ObjectAllocationInNewTLAB event which now should work on native image

[geoand]

👌🏽

[franz1981]

@rsvoboda too: in case you can run the Quarkus start/stop tests on the mentioned commits, to check if it shows something there as well.
In our perf CI it doesn't seem to affect the first time to request RSS, just startup (no requests) which is a bit of a "silly" scenario.

[radcortez]

I'm also going to have a look into it.

[gsmet]

How did we measure what we have on the home page of quarkus.io because it says 12 MB of RSS after first request for a REST application (which the getting-started is)?

Because I see we are at 40MB+ here even for 1 CPU.

[franz1981]

I suppose It was a very old Quarkus version (pre-resteasy reactive? Likely!) - and native image has changed as well too, since is now fatter than has ever been (there are few articles about it from us well, including some related vulnerability mitigations, too).
We are working (perf team), with the Quarkus managers, to talk about this and other performance-related topics.

[mkouba]

How did we measure what we have on the home page of quarkus.io because it says 12 MB of RSS after first request for a REST application (which the getting-started is)?

This info is IMO very outdated. Ideally, we should have this automated, i.e. measure the RSS consumption for every new release but that would take quite some work I guess 🤷.

[franz1981]

Agree @mkouba , this is indeed one among many of the topic for

We are working (perf team), with the Quarkus managers, to talk about this and other performance-related topics.

Since it requires some consensus on what/why/who