Pure JPEG encoding

[dcommander]

Referring to this thread, there is some benefit to a pure JPEG RFB encoding, as opposed to the hybrid Tight + JPEG encoding methods that TurboVNC currently uses. This is particularly true for medium-quality and low-quality JPEG and with applications that use heavily image-based rendering techniques (including 3D applications running with VirtualGL) and/or render a large number of unique colors and/or render few areas of solid color. However, in order to realize that benefit in many cases, it is necessary to modify the TurboJPEG API library so that it can generate "abbreviated image datastreams", i.e. JPEG images without embedded tables (the equivalent of motion-JPEG frames.)

Tasks:

1. Add a new parameter (working title: TJPARAM_NOTABLES/TJ.PARAM_NOTABLES) to the TurboJPEG API that, when set, will cause the compression and transformation functions to call jpeg_suppress_tables(cinfo, TRUE);. Document the parameter. Since this is a non-disruptive API extension that doesn't affect ABI compatibility, it may be possible to introduce it in libjpeg-turbo 3.0.x rather than wait for libjpeg-turbo 3.1.
2. Add support for the pure JPEG RFB encoding (RFB encoding number 21) to the TurboVNC Server. The pure JPEG encoder would simply pass all framebuffer update rectangles directly to libjpeg-turbo, setting the aforementioned TurboJPEG parameter. It would respond to the JPEG Quality Level, JPEG Fine-Grained Quality Level, and JPEG Subsampling Level pseudo-encodings, as the TurboVNC encoder currently does. Document the pure JPEG encoding in the man pages. (It probably won't be documented in the User's Guide initially, since it is an advanced use case.) Bonus points for multithreading it.
3. Add support for the pure JPEG RFB encoding (RFB encoding number 21) to the TurboVNC Viewer. The pure JPEG decoder could reuse the existing TightDecoder native methods, which provide a Java interface for the TurboJPEG C decompress function. Document the new encoding in the TurboVNC Viewer usage screen and man page.

[dcommander]

My latest data is attached. The results are really mixed. Let's assume, for starters, that the primary goal of pure JPEG encoding is to minimize CPU usage (and thus maximize throughput) on very fast connections, such as gigabit or even local (e.g. using TurboVNC to access a VM.) That means that we're primarily interested in improving performance relative to the "Tight + Perceptually Lossless" (hereafter "T+PL") encoding method, even if doing so comes at the expense of compression ratio. Pure JPEG, even with no DHT/DQT tables, simply doesn't have a significant (> 5%) speedup relative to T+PL except for the Pro/E and Quake datasets, and it doesn't have a compelling (> 10%) speedup relative to CL 1 except for the Pro/E dataset on an AVX2-equipped CPU. It also incurs some pretty brutal slowdowns (in the range of 20-60%) for half of the 3D datasets. The other issue with the pure JPEG RFB encoding is that, as specified, it doesn't provide a way for the server to inform the viewer of the JPEG image size. It specifies that the viewer should continue reading until it encounters an "EOF marker". (I can only assume that they mean a JPEG EOI marker?) That is problematic for a VNC viewer, both from a performance and a security point of view. From a performance point of view, the VNC viewer would have to scan through each packet that it reads, looking for an "EOF marker." From a security point of view, a misbehaved server could lock up the viewer by sending a pure JPEG rectangle with no "EOF marker."

Using Tight encoding without zlib (i.e. foregoing the indexed color and mono subencodings and using only the solid and JPEG subencodings) is more compelling. We will call this "CL 0" for the sake of discussion, since if implemented, such an encoding method would be activated by selecting Compression Level 0 (currently unused) when the JPEG subencoding is enabled. Tight encoding requires DHT/DQT tables, so the compression ratios regressed a lot more with CL 0 vs. T+PL for the 2D datasets than with pure JPEG vs. T+PL. (The performance regressions were similar between CL 0 and pure JPEG for the 2D datasets.) However, CL 0 sped up all of the 3D datasets except for TeamCenter Vis and Google Earth. (The latter slowdown was not significant.) On the SSE2-equipped CPU, the 3D Studio MAX, Pro/E, and Quake datasets sped up significantly, whereas the speedup with the SolidWorks dataset was compelling. On the AVX2-equipped CPU, the Catia, Ensight, Lightscape, and Quake datasets sped up significantly, whereas the speedup with the 3D Studio MAX, Maya, Pro/E, and SolidWorks datasets was compelling.

Furthermore, the proposed solid+JPEG (CL 0) encoding method would require no changes to the TurboJPEG API. It could be implemented with only a few modified lines of code in the Tight encoder. I am waiting for a response from the original poster to see if this proposal would solve the problem from their point of view.

[dcommander]

The required additions to the TurboJPEG API (libjpeg-turbo/libjpeg-turbo#777) missed the libjpeg-turbo 3.1 release due to time and funding constraints, so this feature will miss TurboVNC 3.2 as a result [*]. At the moment, my long-term plan is to include it only on the server side in order to support noVNC, since that is the only compelling use case. (Since noVNC has a lot more computational overhead than the TurboVNC Viewer, pure JPEG encoding is more of a benefit for it. Its architecture is also more conducive to detecting a JPEG EOI marker, which would be difficult for the TurboVNC Viewer.)

[*] Technically, I could implement it using the libjpeg API if I really wanted to, but that would introduce additional build dependencies and implementation complexity.