WPE 2.38: Out of memory error while streaming any content with WebInspector open #1335

janani-balasubramanian-infosys · 2024-05-14T04:33:55Z

In RPi, there is a significant increase in memory consumption of WPEThunder when Webinspector is started along with the streaming of any content. But no crash is seen because there is no limit set for WPEThunder and We were able to stream for more than 1.5 hours. But, the memory max usage rapidly increases. Below are the observations in RPi.

root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.limit_in_bytes
9223372036854771712

root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
531464192
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
544538624
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
550207488
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
561790976
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
584716288
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
620036096
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
635936768
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
646520832
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
655519744
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
695853056
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
702447616
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
711151616
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
721469440
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
737206272
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
835309568
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
838696960

Every memory usage values are noted within 1-2 mins.

Scenario to reproduce

Start any video streaming app
Open WebInspector
Play any VOD
Notice the memory consumption

janani-balasubramanian-infosys · 2024-05-17T04:00:23Z

Below is an additional observation made by making the JSC_criticalGCMemoryThreshold=0.10 .

With this change, the playback was running for almost 45 mins. In this, for the first 30 mins the memory increase was stable. But after 30 mins, the memory increase is very fast.

Image tested:

root@raspberrypi4-64-rdk-android-mc:~# cat /version.txt
imagename:rdk-generic-mediaclient-wpe-image_FBT_TDK_nightly-20240503-dunfell_20240503043146
BRANCH=nightly-20240503-dunfell
YOCTO_VERSION=dunfell
VERSION=nightly-20240503-dunfell.05.03.24
SPIN=0
BUILD_TIME="2024-05-03 04:31:46"
WPE_WEBKIT_VERSION=2.38.4+gitAUTOINC+fc1703ed69-r4
WPEFRAMEWORK-VERSION=R4.2
Generated on Fri May 03 04:31:46 UTC 2024
JENKINS_JOB=0
JENKINS_BUILD_NUMBER=0

calvaris · 2024-09-25T09:22:27Z

Could you please test with latest wpe-2.38 ? I landed several build fixes.

suresh-khurdiya-infosys · 2024-09-26T03:45:52Z

thanks @calvaris ,
We reviewed the last 4 to 6 months patches and we don't see any patch related memory pressure, web inspector side.
Do you have few patch in your mind which we may fix this issue?

calvaris · 2024-09-26T05:09:05Z

Yes, all patches merged during merge commit 0cb500e .

hridhya-narayanan-infosys · 2024-10-04T12:54:47Z

@calvaris ,

I have pulled the latest wpe-2.38 code and tested the scenario on RPI4 device.
CNN App was launched and tested video playback while keeping the web inspector open and monitored memory usage.

Observation
I could see memory is getting increased and after some time about 35 mins, I could see WebKitBrowser is deactivated by reason MemoryExceeded.

2024-10-04T11:23:20.509Z WPEFramework[1696]: [SysLog]:[Fatal]: FORCED Shutdown: WebKitBrowser by reason: MemoryExceeded.

root@raspberrypi4-64-rdk-android-mc:~# while [ 1 ]; do
> cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
> sleep 60
> done
373530624
496242688
560435200
618070016
636719104
636719104
636719104
636719104
636719104
640253952
640622592
640655360
657563648
657563648
657563648
657563648
657563648
657563648
660316160
660316160
660643840
661180416
661180416
662335488
662335488
662335488
662335488
662335488
721186816
721186816
721186816
739287040
776536064
835862528
856977408

From Controller UI console I could see below logs:

WebKitBrowser"data: {url: https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#app:com.metrological.app.CNN, loaded: false}
"WebKitBrowser"data: {url: https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#app:com.metrological.app.CNN, loaded: true, httpstatus: 200}
"WebKitBrowser"data: {url: https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#app:com.metrological.app.CNN, loaded: true}
"Monitor"data: {callsign: WebKitBrowser, action: Deactivate, reason: MemoryExceeded}
"Monitor"data: {callsign: WebKitBrowser, action: Activate, reason: Automatic}
"WebKitBrowser"data: {state: deactivated, reason: MemoryExceeded}
"WebKitBrowser"data: {state: activated, reason: Automatic}
"WebKitBrowser"data: {url: about:blank, loaded: false}
"WebKitBrowser"data: {suspended: false}
"WebKitBrowser"data: {url: about:blank, loaded: true, httpstatus: -1}
"WebKitBrowser"data: {url: about:blank, loaded: true}

logs:
wpeframework.log

Without webinspector open I could see memory usage(/sys/fs/cgroup/memory/memory.max_usage_in_bytes) is around 461193216 after 40 mins of CNN playback

Image tested:

root@raspberrypi4-64-rdk-android-mc:~# cat /version.txt 
imagename:rdk-generic-mediaclient-wpe-image_FBT_rdk-next_20241004052558
BRANCH=rdk-next
YOCTO_VERSION=dunfell
VERSION=6.1.10.04.24
SPIN=0
BUILD_TIME="2024-10-04 05:25:58"
WPE_WEBKIT_VERSION=2.38.6+gitAUTOINC+030740eeae-r4
WPEFRAMEWORK-VERSION=R4.4.1
Generated on Fri Oct 04  05:25:58 UTC 2024
JENKINS_JOB=0
JENKINS_BUILD_NUMBER=0
root@raspberrypi4-64-rdk-android-mc:~#

hridhya-narayanan-infosys · 2024-10-16T09:44:41Z

Any update on this issue?

pgorszkowski-igalia · 2024-12-04T10:03:20Z

@hridhya-narayanan-infosys : I am able to reproduce the problem and I am investigating the reason of the problem.

pgorszkowski-igalia · 2024-12-06T08:57:43Z

@hridhya-narayanan-infosys : the problem is also visible in upstream (in specific situation). I created a fix in upstream: WebKit/WebKit#37536 which is waiting for review and merge.
In the meantime I will prepare a downstream patch which you can apply and test.

pgorszkowski-igalia · 2024-12-06T09:03:42Z

@hridhya-narayanan-infosys : The downstream patch you can find here: 39f3e68, please use it and test.

hridhya-narayanan-infosys · 2024-12-06T09:40:51Z

@pgorszkowski-igalia thanks. I will test the issue with above patch.

hridhya-narayanan-infosys · 2024-12-12T11:15:14Z

@pgorszkowski-igalia , I have tested the issue with above patch on RPI 4 device. I could see good improvements.
The CNN video played without any issue for about 3 hours. After this WebKitBrowser is deactivated by reason MemoryExceeded.

Currently checking on the behaviour with other apps as well.

hridhya-narayanan-infosys · 2024-12-18T08:30:54Z

Tested PlutoTV App(https://app-virginmedia360.pluto.tv/) on RPI4 device and we could observe crash around 30 mins.

Testing with other apps is in progress.

pgorszkowski-igalia · 2024-12-18T08:34:02Z

@hridhya-narayanan-infosys : the crash is because of exceeding the memory limit?

pgorszkowski-igalia · 2024-12-18T08:38:20Z

BTW: CNN app is unavailable, I use link: https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#app:com.metrological.app.CNN, on RPi it hangs on CNN logo.

hridhya-narayanan-infosys · 2024-12-18T08:40:07Z

yes the crash is due to memory exceeded. From logs I could see below:

Dec 13 08:53:50 raspberrypi4-64-rdk-android-mc WPEFramework[1694]: [Plugin_Monitor]:[Monitor.h:627]:[MonitorObject]:[Error]: Status MetaData Exceeded. 627
Dec 13 08:53:50 raspberrypi4-64-rdk-android-mc WPEFramework[1694]: [SysLog]:[Fatal]: FORCED Shutdown: WebKitBrowser by reason: MemoryExceeded.
Dec 13 08:53:50 raspberrypi4-64-rdk-android-mc WPEFramework[1694]: [Plugin_Monitor]:[Monitor.h:813]:[ProxyObject<WPEFramework::Core::Service<WPEFramework::Plugin::Monitor::MonitorObjects> >]:[Error]: Restarting WebKitBrowser again because we detected it misbehaved.

Attached logs:
pluto-tv-logs.txt

pgorszkowski-igalia · 2024-12-18T08:40:39Z

With https://app-virginmedia360.pluto.tv/ I get "We're having some trouble... We're currently working on this issue. Please reload the Pluto TV app." , but reloading gives the same result:(

pgorszkowski-igalia · 2024-12-18T08:50:02Z

yes the crash is due to memory exceeded. From logs I could see below:

Dec 13 08:53:50 raspberrypi4-64-rdk-android-mc WPEFramework[1694]: [Plugin_Monitor]:[Monitor.h:627]:[MonitorObject]:[Error]: Status MetaData Exceeded. 627
Dec 13 08:53:50 raspberrypi4-64-rdk-android-mc WPEFramework[1694]: [SysLog]:[Fatal]: FORCED Shutdown: WebKitBrowser by reason: MemoryExceeded.
Dec 13 08:53:50 raspberrypi4-64-rdk-android-mc WPEFramework[1694]: [Plugin_Monitor]:[Monitor.h:813]:[ProxyObject<WPEFramework::Core::Service<WPEFramework::Plugin::Monitor::MonitorObjects> >]:[Error]: Restarting WebKitBrowser again because we detected it misbehaved.

Attached logs: pluto-tv-logs.txt

From your logs I see that the memory usage is huge at the beginning, are logs from the last phase of problem? In further logs I see that you use AAMP-PLAYER for video playback - AAMP-PLAYER is something separate than WPE (it is a separate lib). Does the problem occur with or without this AAMP-PLAYER?

hridhya-narayanan-infosys · 2024-12-18T09:06:55Z

Regarding the logs, yes those are the logs from last phase before the crash.

We didn't use AAMP player.
We have launched WebKitBrowser Plugin from Controller UI and then loaded the PlutoTv app url and tested.

pgorszkowski-igalia · 2024-12-18T12:27:00Z

There are a lot of logs with:

WPEWebProcess[453]: [AAMP-PLAYER]

Are you sure that PlutoTV does not use it? via UVE js bindings?

Do you know what kind of playback is used by PlutoTV? regular one or adaptive (mpeg-dash or hls)?

hridhya-narayanan-infosys · 2024-12-20T12:54:45Z

@pgorszkowski-igalia ,

In the earlier logs shared we had opened a HTML5 app(Xumo) which used AAMP JS bindings before launching PlutoTV app. So the log was showing the AAMP-PLAYER related logs.

Pluto TV App has been tested once again. These are the logs.
wpeframework.log

PlutoTV App uses HLS playback and also from JS code of PlutoTV App I couldn't see usage of Aamp player UVE APIs.

pgorszkowski-igalia · 2024-12-20T13:23:05Z

Do you observe similar behaviour in other apps which use HLS?

janani-balasubramanian-infosys changed the title ~~WPE 2.38: Out of memory error when running an App with WebInspector open~~ WPE 2.38: Out of memory error while streaming any content with WebInspector open May 14, 2024

modeveci assigned pgorszkowski-igalia Nov 21, 2024

modeveci added the wpe-2.38 label Nov 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WPE 2.38: Out of memory error while streaming any content with WebInspector open #1335

WPE 2.38: Out of memory error while streaming any content with WebInspector open #1335

janani-balasubramanian-infosys commented May 14, 2024 •

edited

Loading

janani-balasubramanian-infosys commented May 17, 2024 •

edited

Loading

calvaris commented Sep 25, 2024

suresh-khurdiya-infosys commented Sep 26, 2024

calvaris commented Sep 26, 2024

hridhya-narayanan-infosys commented Oct 4, 2024 •

edited

Loading

hridhya-narayanan-infosys commented Oct 16, 2024

pgorszkowski-igalia commented Dec 4, 2024

pgorszkowski-igalia commented Dec 6, 2024

pgorszkowski-igalia commented Dec 6, 2024 •

edited

Loading

hridhya-narayanan-infosys commented Dec 6, 2024

hridhya-narayanan-infosys commented Dec 12, 2024

hridhya-narayanan-infosys commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024

hridhya-narayanan-infosys commented Dec 18, 2024 •

edited

Loading

pgorszkowski-igalia commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024

hridhya-narayanan-infosys commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024 •

edited

Loading

hridhya-narayanan-infosys commented Dec 20, 2024

pgorszkowski-igalia commented Dec 20, 2024

WPE 2.38: Out of memory error while streaming any content with WebInspector open #1335

WPE 2.38: Out of memory error while streaming any content with WebInspector open #1335

Comments

janani-balasubramanian-infosys commented May 14, 2024 • edited Loading

janani-balasubramanian-infosys commented May 17, 2024 • edited Loading

calvaris commented Sep 25, 2024

suresh-khurdiya-infosys commented Sep 26, 2024

calvaris commented Sep 26, 2024

hridhya-narayanan-infosys commented Oct 4, 2024 • edited Loading

hridhya-narayanan-infosys commented Oct 16, 2024

pgorszkowski-igalia commented Dec 4, 2024

pgorszkowski-igalia commented Dec 6, 2024

pgorszkowski-igalia commented Dec 6, 2024 • edited Loading

hridhya-narayanan-infosys commented Dec 6, 2024

hridhya-narayanan-infosys commented Dec 12, 2024

hridhya-narayanan-infosys commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024

hridhya-narayanan-infosys commented Dec 18, 2024 • edited Loading

pgorszkowski-igalia commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024

hridhya-narayanan-infosys commented Dec 18, 2024

pgorszkowski-igalia commented Dec 18, 2024 • edited Loading

hridhya-narayanan-infosys commented Dec 20, 2024

pgorszkowski-igalia commented Dec 20, 2024

janani-balasubramanian-infosys commented May 14, 2024 •

edited

Loading

janani-balasubramanian-infosys commented May 17, 2024 •

edited

Loading

hridhya-narayanan-infosys commented Oct 4, 2024 •

edited

Loading

pgorszkowski-igalia commented Dec 6, 2024 •

edited

Loading

hridhya-narayanan-infosys commented Dec 18, 2024 •

edited

Loading

pgorszkowski-igalia commented Dec 18, 2024 •

edited

Loading