Recent engines now crash with CSC2.0 stress-test

[pkgw]

@DougBurke reports that the CSC 2.0 WWT viewer is having problems with the newer engine that it didn't use to have. To reproduce:

1. Navigate to https://cxc.cfa.harvard.edu/csc/nwwt.html
2. In the menu, click "Show Popular Places" then "Sgr A*"
3. Click "Load CSC 2.0 Sources"
4. Click again after the button changes to "Show CSC 2.0 Sources"

For me, the view locks up pretty much instantly and eventually crashes. It used to just work, and it apparently still works using the very old version of the SDK obtained through http://www.worldwidetelescope.org/scripts/wwtsdk.aspx. The issue seems to be cross-browser and cross-platform.

CC @imbasimba @Carifio24, although Henrik, we do need to prioritize your time for the tiled FITS stuff.

[DougBurke]

The source locations are added as annotations rather than any of the newer layer stuff, since I couldn't work out how to control the layer stuff last time I looked (e.g. change radius/color).

There is a little more detail below, but the basic idea is that the 'show 2.0 sources' option causes a bunch of circle annotations to get added to the display, chosen to lie within a radius of the current location (roughly the screen size), and for this area it can cause ~28000 circles to get added.

The JS is written in an "old-school" style (no bundles or modern packaging approaches) and the onload handler calls

wwt.initialize(outlines);

where outlines is data loaded from a JS file and the wwt code is defined in

https://github.com/cxcsds/cscwwt/blob/main/website/wwt.js

I cleverly decided to use the wwt "namespace" just to collide with the WWT code...

The "load csc 2.0 sources" will fill up the catalogProps.csc20 structure which the 'show csc 2.0' sources is processed, eventually with _showSources which identifies the sources "near" the center of the screen and then makes an annotation (circle) for all the nearby sources and then displays them (annotations created with the makeCircle routine which is just a wrapper around the WWT createCircle function). Unfortunately there's a lot of indirection in the code (as it's used byseveral catalogs) so it's a bit annoying to trace.

[pkgw]

Finally able to sit down and look at this. Some initial debugging notes:

1. Set a JS breakpoint at Annotation.prepBatch in renderOneFrame
2. Disable it so that the app can run normally
3. Set a breakpoint at _showSources in nwwt.js
4. Go through the triggering conditions as above
5. When the _showSources breakpoint fires, re-enable the renderOneFrame breakpoint
6. Unpause JS execution. The app will almost-instantly stop at the renderOneFrame breakpoint now.

After the call to Annotation.prepBatch, the annotation code will iterate over ~40,000 annotations, adding ~4.8 million entries to the Annotation.lineList buffer. On Firefox, I find that the JS seizes up before Annotation.drawBatch does anything with GL — looks like we might be running out memory?

• In "normal" operations, with the FOV overlays but not a bunch of sources to be drawn, the Annotation.lineList buffer is at about ~650,000 entries.
• In between prepBatch and drawBatch when the sources have been added, the JS memory consumption goes up from ~300 MB to ~3 GB in Chrome. It becomes big enough that the memory profiling tools start breaking ...
• Each circle annotation is rendered as 72 line segments
• Each linelist record adds two 3D points (three numbers each, XYZ), a color (four numbers. RGBA), and a date (two numbers, start and end). So if those are stored as 64-bit numbers, that's ~100 bytes per linelist record, which gets us to several gigs of memory.
• Oh hey, the Annotation.triangleList buffer hits 97 million points!! That will be a problem.
• Each filled circle adds 213 triangles

[pkgw]

@DougBurke OK! If I hack a local copy of the app to not fill in the source circles, the stress test works for me. As annotated in my notes above, each filled circle adds 213 triangles to the WebGL render list, and 213 * 42,000 ~= 9 million triangles seems to be too many.

If you really really want to render many filled circles we could investigate an optimization of that rendering path, but if there's a way to not fill the circles that makes you happy, that is gonna be the much simpler solution.

[DougBurke]

Sorry - currently on holiday. Will get back to this when I'm back.

[pkgw]

Hi @DougBurke — with #160 merged, this problem should be better, although performance is still pretty gnarly in the stress-test case. I can now at least navigate around and not actually crash the renderer.