Webview being hidden by Unity UI on MacOS Catalina 10.15.7

[Wave1]

Upgraded my Mac to Catalina 10.15.7 and was finding my project was crashing when loading into Unity due to disabling Metal support, so upgraded to latest gree-webview version with metal support to get around this problem.

I now find that when running in the Unity Editor the webview is not visible and seems to be hidden behind my Unity UI drawn with a screen space overlay canvas. If I disable my game object with the canvas then the webview display can be seen, so it clearly is rendering correctly but somehow being drawn first or perhaps hidden due to depth buffer or something like that?

For now my workaround is to run using the 'separated' option which appears to work OK, but would ideally like to get it working in the Game view as it's easier to test my app that way.

I've tried the sample code and that works well so it seems something in my app could be to blame? Anything you can think of that might cause this behaviour? I do create two instances of the webview at startup in my app, although I have tried reducing this to one and it made no difference.

[KojiNakamaru]

For the editor, a webview is rendered with GUI.DrawTexture() in OnGUI(). The original implementation however didn't consider Event.current.type so its behaviour was redundant (the webview may be drawn repeately in one frame) and difficult to understand. I've refactored the code in #600 and updated binaries. This could not solve your issue, but could you please try the latest? Then, depending on the result, I'll further track down the issue.

BTW, when you use "separated", can you see a translucent white rectangle on screen? If yes, then texture created/updated at

(latest code)

unity-webview/plugins/WebViewObject.cs

Lines 1014 to 1034 in ac144eb

	if (webView == IntPtr.Zero || !visibility)
	return;
	bool refreshBitmap = (Time.frameCount % bitmapRefreshCycle == 0);
	_CWebViewPlugin_Update(webView, refreshBitmap);
	if (refreshBitmap) {
	{
	var w = _CWebViewPlugin_BitmapWidth(webView);
	var h = _CWebViewPlugin_BitmapHeight(webView);
	if (texture == null || texture.width != w || texture.height != h) {
	texture = new Texture2D(w, h, TextureFormat.RGBA32, false, true);
	texture.filterMode = FilterMode.Bilinear;
	texture.wrapMode = TextureWrapMode.Clamp;
	}
	}
	_CWebViewPlugin_SetTextureId(webView, texture.GetNativeTexturePtr());
	_CWebViewPlugin_SetCurrentInstance(webView);
	#if UNITY_4_6 || UNITY_5_0 || UNITY_5_1
	GL.IssuePluginEvent(-1);
	#else
	GL.IssuePluginEvent(GetRenderEventFunc(), -1);
	#endif

(last code)

unity-webview/plugins/WebViewObject.cs

Lines 1066 to 1084 in e0e11db

	bool refreshBitmap = (Time.frameCount % bitmapRefreshCycle == 0);
	_CWebViewPlugin_Update(webView, refreshBitmap);
	if (refreshBitmap) {
	{
	var w = _CWebViewPlugin_BitmapWidth(webView);
	var h = _CWebViewPlugin_BitmapHeight(webView);
	if (texture == null || texture.width != w || texture.height != h) {
	texture = new Texture2D(w, h, TextureFormat.RGBA32, false, true);
	texture.filterMode = FilterMode.Bilinear;
	texture.wrapMode = TextureWrapMode.Clamp;
	}
	}
	_CWebViewPlugin_SetTextureId(webView, texture.GetNativeTexturePtr());
	_CWebViewPlugin_SetCurrentInstance(webView);
	#if UNITY_4_6 || UNITY_5_0 || UNITY_5_1
	GL.IssuePluginEvent(-1);
	#else
	GL.IssuePluginEvent(GetRenderEventFunc(), -1);
	#endif

is somehow not updated correctly with the webview's content in "non separated" mode.

[Wave1]

Thanks for looking into this so quickly. Unfortunately that change hasn't fixed the problem but it does give me some further information you may find useful.

If I edit the GUI.DrawTexture call in WebViewObject.cs to:

GUI.DrawTexture(rect, texture, ScaleMode.ScaleToFit, false);

The screen flickers where the webview should be drawn. I also tried modifying this code to use a Graphics.DrawTexture call instead, and was only able to draw a black rectangle, which perhaps suggests the texture hasn't been created properly.

To answer your question about running in separated mode, yes, there is a rectangle drawn where the webview should be. It has the effect of darkening my UI so it would appear it is being drawn over the UI. The images drawn in the separated windows appear to be correct.

[KojiNakamaru]

If you use separated: true, do you see a rectangle as below, without flickering?

The plugin updates the bitmap by 0x80808080 for separated: true and with the webview snapshot for separated: false,

unity-webview/plugins/Mac/Sources/WebView.mm

Lines 619 to 645 in ac144eb

	if (window != nil) {
	memset([bitmap bitmapData], 128, [bitmap bytesPerRow] * [bitmap pixelsHigh]);
	inRendering = NO;
	} else {
	NSBitmapImageRep *bmpRep = bitmap;
	// memset([bitmap bitmapData], 0, [bitmap bytesPerRow] * [bitmap pixelsHigh]);
	// [webView cacheDisplayInRect:webView.frame toBitmapImageRep:bitmap];
	WKWebView *wkv = webView;
	WKSnapshotConfiguration *cfg = [WKSnapshotConfiguration new];
	//cfg.snapshotWidth = [NSNumber numberWithLong:[bitmap pixelsWide]];
	//cfg.rect = webView.frame;
	[wkv takeSnapshotWithConfiguration:cfg
	completionHandler:^(NSImage *nsImg, NSError *err) {
	if (err == nil) {
	NSGraphicsContext *ctx = [NSGraphicsContext graphicsContextWithBitmapImageRep:bmpRep];
	[NSGraphicsContext saveGraphicsState];
	[NSGraphicsContext setCurrentContext:ctx];
	[nsImg drawAtPoint:CGPointZero
	fromRect:CGRectMake(0, 0, [bmpRep pixelsWide], [bmpRep pixelsHigh])
	operation:NSCompositingOperationCopy
	fraction:1.0];
	[[NSGraphicsContext currentContext] flushGraphics];
	[NSGraphicsContext restoreGraphicsState];
	}
	self->inRendering = NO;
	}];
	}

and for both cases the bitmap is copied to the texture in each frame,

unity-webview/plugins/Mac/Sources/WebView.mm

Lines 686 to 700 in ac144eb

	if (s_useMetal) {
	id<MTLTexture> tex = (__bridge id<MTLTexture>)textureId;
	[tex replaceRegion:MTLRegionMake3D(0, 0, 0, w, h, 1) mipmapLevel:0 withBytes:d bytesPerRow:r];
	} else {
	int rowLength = 0;
	int unpackAlign = 0;
	glGetIntegerv(GL_UNPACK_ROW_LENGTH, &rowLength);
	glGetIntegerv(GL_UNPACK_ALIGNMENT, &unpackAlign);
	glPixelStorei(GL_UNPACK_ROW_LENGTH, (GLint)w);
	glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
	glBindTexture(GL_TEXTURE_2D, (GLuint)(long)textureId);
	glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, (GLsizei)w, (GLsizei)h, GL_RGBA, GL_UNSIGNED_BYTE, d);
	glPixelStorei(GL_UNPACK_ROW_LENGTH, rowLength);
	glPixelStorei(GL_UNPACK_ALIGNMENT, unpackAlign);
	}

so if separated: true is rendered correctly, there should be no issue in WebViewObject.cs at least.

Could you please also provide your Unity version and graphics mode (as shown in the title bar in the screenshot above)? The plugin can run for both OpenGL and Metal (on Catalina), but we should align the condition as much as possible for further investigation.

[Wave1]

I can confirm that running the sample app I see a white rectangle as shown in your screen grab. Same thing appears to be happening in my own app as well when running separated: true, just when running with separated: false things are going wrong. As I said previously, if I turn off my UI by disabling the game object containing the canvas whilst running in the Unity Editor the webview display can then be seen hiding underneath my UI, which is odd and seems to be against the way the OnGUI function works in my experience.

I'm using Metal and Unity 2019.3.9f1. Had to move over to Metal after upgrading Mac OS to 10.15.7 as using OpenGL was causing Unity to crash (I think OpenGL may finally have been deprecated in 10.15.7 but not 100% sure about that).

Not able to test right now but tomorrow I will try running on a newer Unity version just in case that makes a difference.

[Wave1]

Have just tried running my project in Unity 2019.4.12f1 and still seeing the same result in separated: false - the webview is being drawn underneath the rest of my UI.

However, I do get the following Warning message repeatedly output in the console log:

Metal: Fragment shader missing texture binding at index 0 (_MainTex / Hidden/Internal-GUITextureClip)

Adding a Debug.Log call in that outputs Time.frameCount shows that this message is not displayed every frame, but varies between every 5-30 frames. If I comment out the call to GUI.DrawTexture in WebViewObject.OnGUI this message is no longer displayed.

[KojiNakamaru]

Thank you for the detailed report. Based on your observation, I checked WebView.mm again found a flaw that could lead your issue. I've fixed it in #601 and updated binaries. Could you please try the latest?

[Wave1]

Sadly this seems to have made things worse. The webview is still not displayed and my project now runs for a second or so and then locks up the Unity Editor, eventually causing Unity to crash after about 10 seconds. This happens with separated set to true and false.

The webview sample code still works correctly however.

[Wave1]

I noticed you had pushed out another fix this morning, this fixes the crash I was seeing but unfortunately the webview is still being drawn behind all my UI and I still get the "Fragment shader missing texture binding" warning in the console log, although it doesn't seem to trigger quite as much as it did before.

One further thing I have just noticed, if I run with separated: true and then try to inspect a web page element by right clicking on it in the separated window, the debug inspection UI in that window rapidly flickers on and off. This only happens in my own project, not in the webview sample project. Hoping that might point you at something perhaps?

[KojiNakamaru]

Thanks for further testing. As you noticed, I've fixed one synchronization issue in #602. I also introduced another fix in #603 to keep some static buffer for Metal, as the original bitmap may be changed while GPU processes the texture. The binaries can be found at:

https://github.com/gree/unity-webview/tree/master

Moreover, I've made another variation with Metal support in #604 based on the former WebView (not WKWebView) version. Could you please try this variation if the latest on the master branch didn't work? The binaries can be found at:

https://github.com/gree/unity-webview/tree/experimental/metal_support_for_webview

[Wave1]

#603 causes my app to crash as soon as I make the webview visible. I was able to get this stack trace from Unity Editor.log by turning on the Full Log setting in Unity Console.

Native stacktrace:

	0   libmonobdwgc-2.0.dylib              0x000000013fdeb7a5 mono_handle_native_crash + 242
	1   libmonobdwgc-2.0.dylib              0x000000013fd4bbcd mono_sigsegv_signal_handler + 220
	2   libsystem_platform.dylib            0x00007fff6953e5fd _sigtramp + 29
	3   AppKit                              0x00007fff2d1908e5 .str.38 + 668019
	4   AppleIntelICLGraphicsMTLDriver      0x00007fff26798608 -[MTLIGAccelTexture replaceRegion:mipmapLevel:slice:withBytes:bytesPerRow:bytesPerImage:] + 1908
	5   AppleIntelICLGraphicsMTLDriver      0x00007fff26798923 -[MTLIGAccelTexture replaceRegion:mipmapLevel:withBytes:bytesPerRow:] + 64
	6   WebView                             0x00000001423e0730 -[CWebViewPlugin render] + 694
	7   WebView                             0x00000001423e0e8b UnityRenderEvent + 82
	8   Unity                               0x000000010a4a231f _ZN9GfxDevice26InsertCustomMarkerCallbackEPFviEi + 31
	9   Unity                               0x000000010b0333b8 _ZN15GfxDeviceWorker10RunCommandER20ThreadedStreamBuffer + 9576
	10  Unity                               0x000000010d0f966c GfxDeviceWorkerAutoreleasePoolProxy + 60
	11  Unity                               0x000000010b041690 _ZN15GfxDeviceWorker6RunExtER20ThreadedStreamBuffer + 672
	12  Unity                               0x000000010b030848 _ZN15GfxDeviceWorker18RunGfxDeviceWorkerEPv + 152
	13  Unity                               0x000000010c02b35c _ZN6Thread16RunThreadWrapperEPv + 1228
	14  libsystem_pthread.dylib             0x00007fff6954a109 _pthread_start + 148
	15  libsystem_pthread.dylib             0x00007fff69545b8b thread_start + 15

#604 does work initially, but eventually causes a crash when I navigate to a part of my app that hides the webview, though I'm not sure if that's the actual cause of the crash or not. Stack trace from Unity Editor.log file below, which appears very similar to the one above.

Native stacktrace:

	0   libmonobdwgc-2.0.dylib              0x00000001431eb7a5 mono_handle_native_crash + 242
	1   libmonobdwgc-2.0.dylib              0x000000014314bbcd mono_sigsegv_signal_handler + 220
	2   libsystem_platform.dylib            0x00007fff6953e5fd _sigtramp + 29
	3   ???                                 0x0000000000000001 0x0 + 1
	4   AppleIntelICLGraphicsMTLDriver      0x00007fff26798608 -[MTLIGAccelTexture replaceRegion:mipmapLevel:slice:withBytes:bytesPerRow:bytesPerImage:] + 1908
	5   AppleIntelICLGraphicsMTLDriver      0x00007fff26798923 -[MTLIGAccelTexture replaceRegion:mipmapLevel:withBytes:bytesPerRow:] + 64
	6   WebView                             0x000000013ac477eb -[CWebViewPlugin render] + 777
	7   WebView                             0x000000013ac47f05 UnityRenderEvent + 82
	8   Unity                               0x000000010d85b31f _ZN9GfxDevice26InsertCustomMarkerCallbackEPFviEi + 31
	9   Unity                               0x000000010e3ec3b8 _ZN15GfxDeviceWorker10RunCommandER20ThreadedStreamBuffer + 9576
	10  Unity                               0x00000001104b266c GfxDeviceWorkerAutoreleasePoolProxy + 60
	11  Unity                               0x000000010e3fa690 _ZN15GfxDeviceWorker6RunExtER20ThreadedStreamBuffer + 672
	12  Unity                               0x000000010e3e9848 _ZN15GfxDeviceWorker18RunGfxDeviceWorkerEPv + 152
	13  Unity                               0x000000010f3e435c _ZN6Thread16RunThreadWrapperEPv + 1228
	14  libsystem_pthread.dylib             0x00007fff6954a109 _pthread_start + 148
	15  libsystem_pthread.dylib             0x00007fff69545b8b thread_start + 15

[KojiNakamaru]

Thanks again for testing and logs. I further changed the texture updating mechanism, which now don't make any Metal/OpenGL call in the native part. This new approach should be free from the issue discussed here. I'm sorry for trouble, but could you please try again? The following both branches are updated.

https://github.com/gree/unity-webview/tree/master
https://github.com/gree/unity-webview/tree/experimental/metal_support_for_webview

[Wave1]

The good news is that the crashes have now been fixed in both versions, and the metal_support_for_webview version is working in my project.

The bad news is that the master version is still displaying the webview underneath the UI when running in separated: false mode, and the inspect element UI still flickers when running in separated: true.

I still get the Metal: Fragment shader missing texture binding warning every few frames, but when this happens it is now proceeded by a new error log "No texture data provided to LoadRawTextureData". The cause of this appears to be that _CWebViewPlugin_BitmapWidth and _CWebViewPlugin_BitmapHeight occasionally both return zero for the dimensions.

[Wave1]

The zero width/height being reported every so often got me thinking about how my project works and I realised I was making a call to WebViewObject.SetMargins every frame in my project. As a test I tried commenting out this call completely and the webview drew correctly over my UI!

Calling SetMargins every frame is definitely the cause of my problem. To confirm this I added an update method with a call to SetMargins in the webview sample code and sure enough the webview stopped drawing.

I have now changed my code so that I cache the current webview margin sizes and only call SetMargins when the margin sizes need to change and this has fixed the problem for me.

Many thanks for all your hard work in trying to track this down, it is very much appreciated and I hope it hasn't been a waste of your time.

[KojiNakamaru]

Thank you very much too. Your report and analysis helped me a lot and I also reached the current, more robust approach. I'll later incorporate your idea about SetMargin().