Reduce memory footprint of pass 1 heapdump loading

[eclipsewebmaster]

| --- | --- |
| Bugzilla Link | 571331 |
| Status | NEW |
| Importance | P3 enhancement |
| Reported | Feb 19, 2021 03:58 EDT |
| Modified | May 13, 2022 08:54 EDT |
| Version | 1.11 |
| Reporter | Jason Koch |

Description

The Identifier index created during Pass 1 parsing currently uses a single linear array, and grows as required by allocating arrays at 1.5x and copying all values. A consequence of this is that, for a given memory footprint (eg: 1GB + one object), the required parsing space is (1 + 1*1.5 =) 2.5GB of heap.

Instead of growing by 1.5x, it would use less memory during Pass 1 parsing if memory was allocated and extended as needed, with no copying to an extended array.

[eclipsewebmaster]

By Jason Koch on Feb 19, 2021 04:11

Created attachment 285606
Identifier Index patch

:notepad_spiral: 571331_pass1_heap.patch

[eclipsewebmaster]

By Andrew Johnson on Feb 19, 2021 11:20

Thanks for the contribution.

1. Is this a real problem in that is pass 1 the limiting factor for being able to successfully parse a dump, or will a later stage always
   fail first - e.g. the dominator tree which allocates at least 6 int arrays each of size of the number of identifiers?
2. The parallel sort is replaced by a special Quick sort - is this significantly slower?
3. Could a pathological / malicious dump trigger n-squared sort time?

[eclipsewebmaster]

By Jason Koch on Mar 25, 2021 11:02

Andrew, this is a lead-in to some follow-up changes; the goal being that we can likely eliminate the pass1 static footprint requirement at initial parse time. The second chunk is allowing pass1 parsing to be shifted to disk before starting pass2. Net outcome is a reduction of the fixed pass1 memory requirements during parsing. The main challenge is doing it without breaking the API.

I'm happy to withdraw this, I think there's probably a better way to tackle it by deferring to some alternative IO model. This would be something like writing directly through to disk for some of the Array* models but this needs to be balanced against IO costs. I've yet to really think about this in detail but it seems like this would allow for significantly reduced parsing memory requirements.

[eclipsewebmaster]

By Andrew Johnson on May 13, 2022 08:54

I made a basic change to address some of this problem:
https://git.eclipse.org/c/mat/org.eclipse.mat.git/diff/plugins/org.eclipse.mat.parser/src/org/eclipse/mat/parser/index/IndexWriter.java?id=23f80cc96a622bdc1dc886fdab5cbc69c9f3c945\
This uses an ArrayLongBig to accumulate the longs in a series of arrays.
Before a sort the values are then copied to a long[], so it does require approximately 2N values, rather than up to 2.5N for before this change.
The attachment 285606 shows how it could be done in N values, but with
a more complex sort of the subarrays.

I notice that the class library for Java 17 for Arrays.parallelSort() makes
a copy of the array, whereas Java 11 did not. This means that even if the
conversion of ArrayLongBig to long[] were done via a file so that only N values
were kept in memory at once, the parallelSort would require another N.

If the heap space for Identifier index is still a problem in pass 1 then we can look at this patch again.