Skip to content

Commit

Permalink
[Enhancement] Opt memory tracker for FE (#53055)
Browse files Browse the repository at this point in the history
## Why I'm doing:
`SizeEstimator.estimate` will calculate the retained size of the object, that is the size of the param object and all the referenced objects. This will lead to repeated counting when the same object is referenced by many objects, and result in very high resource usage when the param object refers to network object or GlobalMetaStore.

## What I'm doing:
Change to `ClassLayout` to only calculate the shadow size of the object.

Signed-off-by: gengjun-git <[email protected]>
  • Loading branch information
gengjun-git authored Nov 22, 2024
1 parent 9627535 commit 4f60313
Showing 1 changed file with 15 additions and 2 deletions.
17 changes: 15 additions & 2 deletions fe/fe-core/src/main/java/com/starrocks/memory/MemoryTrackable.java
Original file line number Diff line number Diff line change
Expand Up @@ -15,26 +15,39 @@
package com.starrocks.memory;

import com.starrocks.common.Pair;
import org.apache.spark.util.SizeEstimator;
import org.openjdk.jol.info.ClassLayout;

import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public interface MemoryTrackable {
// The default implementation of estimateSize only calculate the shadow size of the object.
// The shadow size is the same for all instances of the specified class,
// so using CLASS_SIZE to cache the class's instance shadow size.
// the key is class name, the value is size
Map<String, Long> CLASS_SIZE = new ConcurrentHashMap<>();

default long estimateSize() {
List<Pair<List<Object>, Long>> samples = getSamples();
long totalBytes = 0;
for (Pair<List<Object>, Long> pair : samples) {
List<Object> sampleObjects = pair.first;
long size = pair.second;
if (!sampleObjects.isEmpty()) {
totalBytes += (long) (((double) SizeEstimator.estimate(sampleObjects)) / sampleObjects.size() * size);
long sampleSize = sampleObjects.stream().mapToLong(this::getInstanceSize).sum();
totalBytes += (long) (((double) sampleSize) / sampleObjects.size() * size);
}
}

return totalBytes;
}

default long getInstanceSize(Object object) {
String className = object.getClass().getName();
return CLASS_SIZE.computeIfAbsent(className, s -> ClassLayout.parseInstance(object).instanceSize());
}

Map<String, Long> estimateCount();

// Samples for estimateSize() to calculate memory size;
Expand Down

0 comments on commit 4f60313

Please sign in to comment.