Skip to content

OpenCV GSoC 2020 Proposal

Jump to bottom
Ningxin Hu edited this page Mar 14, 2020 · 8 revisions

  1. IDEA: OpenCV.js: WASM SIMD optimization 2.0

    • Description: Through GSoC 2019 project Improve the performance of JavaScript version of OpenCV (OpenCV.js), the performance of some kernels, e.g. threshold, are improved significantly by applying WebAssembly (WASM) SIMD intrinsics and WebWorker/SharedArrayBuffer threads. However as WASM SIMD spec was still under active development in 2019, there were some remaining issues that include low performance for some kernels, e.g. cvtColor and resize, and lack of 64bit intrinsics support. And due to limited GSoC duration, only 3 kernels were tested and optimized specifically. With WASM SIMD spec will be in stable in 2020, it would be useful to improve the WASM SIMD optimization of OpenCV.js by fixing the remaining issues and targeting more kernels.
    • Expected Outcomes:
      • Implement 64bit intrinsics
      • Improve the performance of cvtColor and resize kernels
      • Refactory the js perf test framework for better code reuse
      • Add more perf tests for kernels of imgproc, dnn and objectdetect modules
      • Fix any potential perf issues due to new perf tests
      • Create sample code of OpenCV.js loader that can detect browser features and load according OpenCV.js build, e.g. SIMD-enabled or SIMD-disabled.
    • Resources:
    • Skills Required: C++, JavaScript, HTML, WebAssembly, performance optimization, multi-threading and SIMD. Fluent in written English.
    • Mentors: Vitaly Tuzov, Ningxin Hu
    • Difficulty: Medium to Hard

  2. IDEA: Develop OpenCV.js DNN modules for promising web use cases together with their tutorials

    • Description: OpenCV.js has JavaScript API of dnn module that allows web apps to do deep learning model inference in browser. However, except a face identification tutorial, there are lack of comprehensive tutorials about the deployment and inference of models in different formats, for promising usages like image classification and object detection with interactive web content like HTML5 video and WebRTC video streams.
    • Expected Outcomes:
      • Develop comprehensive tutorials about the end-to-end deployment and inference of deep learning models on the Web with OpenCV.js. The model formats include ONNX, TensorFlow, Caffe, Darknet and Pytorch.
      • Develop reusable sample code and helper functions for processing visual contents on Web including image, video (HTML5 video) and camera capture (WebRTC video stream) with dnn JS API.
      • Develop interactive samples for the deep learning usages including image classification, object detection, segmentation, pose estimation and image processing.
    • Resources:
    • Skills Required: C++, JavaScript, HTML, image, canvas, video, WebRTC and experience with deep neural networks. Fluent in written English.
    • Possible Mentors: Ningxin Hu
    • Difficulty: Medium

  3. IDEA: GPU-enabled OpenCV.js

    • Description: Today's OpenCV.js compiles kernels into WebAssembly format, so Web browsers can execute them on CPU efficiently. With the recent development of WebGPU and Web Shading Language(WSL), Web browser is adding the capability to run GPU compute shader for general purpose computing. Some web-based frameworks, like TensorFlow.js, are adopting WebGPU support for better performance. OpenCV has sophisticated GPU backends include Cuda, OpenCL, Vulkan and Halide. It would be useful to start prototyping OpenCV.js GPU backend with WebGPU by investigating different solutions.
    • Expected Outcomes:
      • Create a prototype of OpenCV.js GPU backend with WebGPU. Explore the possible solutions include (but not limited):
        • Translating Vulkan compute shader (SPIR-V) of OpenCV.js Vulkan backend to WSL and execute by WebGPU API.
        • Supporting WebGPU in Halide, so the OpenCV.js Halide backend can generate WSL for WebGPU execution.
      • Create a prototype of OpenCV.js I/O module with WebGPU support for more efficient buffer sharing between video/image and OpenCV.js WebGPU kernels.
      • Develop a benchmark to compare OpenCV.js CPU (WASM) and GPU (WebGPU) backends.
    • Resources:
    • Skills Required: C++, JavaScript, HTML, WebGL, WebGPU, OpenCL, Vulkan, Halide.
    • Possible Mentors: (OpenCV GPU expert to be added), Ningxin Hu
    • Difficulty: Hard
Clone this wiki locally