v0.8.1.0 Release

v0.8.1.0 Release

CHANGELOG

@@ -1,6 +1,16 @@
-Changelog (v0.7.6.2)
- - 3D Body Pose Keypoint Tracking
-    - Quality improvement in accuracy and stability
-    - Property NvAR_Parameter_Config(NVAR_MODE) is now NvAR_Parameter_Config(Mode)
- - Migrated to TensorRT 8.0.1.6
- - Migrated to CUDA 11.3u1
+Changelog (v0.8.1.0)
+--------------------
+ - NEW! Eye Contact feature: an AI algorithm to help users keep their gaze engaged in video communication. The feature jointly estimates a user’s gaze direction and redirects it to frontal in video sequences.
+ - NEW! Face Expression Estimation (Beta) feature estimates facial expression coefficients From the video or the provided facial landmarks. ExpressionApp is added to demonstrate the new Face Expressions feature.
+ - NEW! Default face model for the Face 3D mesh and tracking feature, face_model2.nvf, now ships with the SDK. The old SFM based face_model0.nvf is no longer required.
+ - 3D Body Pose Estimation:
+    - NEW! Added the support for Multi Person Tracking. This feature is supported by the Windows SDK only.
+    - FocalLength is now a NvAR_Parameter_Input. Users can now change FocalLength at every NvAR_Run() without having to call NvAR_Load().
+    - The reference pose returned by the feature has been updated
+ - Facial landmark estimation
+    - NEW! There are now 2 modalities for facial landmark tracking: {0,1} -> {performance, quality}. Make sure to choose the preferred mode for your application. The default for face mesh fitting and expression estimation are 1, and the others are 0.
+    - Head Pose output from the NvAR_Feature_LandmarkDetection feature is now in the OpenGL convention. Changed from X-back(towards the camera), Y-right, Z-down to X-right, Y-up, Z-back(towards the camera).
+    - The sample apps now show the headpose in the OpenGL convention. The color coding of the axes is  Red - X , Green - Y, Blue - Z
+ - NvCVImage_Transfer() now sets alpha to 255 or 1.0f when doing RGB -> RGBA. NvCVImage_CompositeRect() has a premultiplied alpha mode added
+ - Migrated to TensorRT 8.4.2.2
+ - Migrated to CUDA 11.6u1

CMakeLists.txt

@@ -11,19 +11,95 @@ set(CMAKE_CONFIGURATION_TYPES "Release")
 
 set(CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake" ${CMAKE_MODULE_PATH})
 
-set(CMAKE_CXX_STANDARD 14)
-set(CMAKE_CXX_STANDARD_REQUIRED ON)
-set(CMAKE_CXX_EXTENSIONS OFF)
+if(MSVC)
+    set(CMAKE_CXX_STANDARD 14)
+    set(CMAKE_CXX_STANDARD_REQUIRED ON)
+    set(CMAKE_CXX_EXTENSIONS OFF)
 
-set(SDK_INCLUDES_PATH ${CMAKE_CURRENT_SOURCE_DIR}/nvar/include)
+    set(SDK_INCLUDES_PATH ${CMAKE_CURRENT_SOURCE_DIR}/nvar/include)
 
-# Add target for nvARPose
-add_library(nvARPose INTERFACE)
-target_include_directories(nvARPose INTERFACE ${SDK_INCLUDES_PATH})
+    # Add target for nvARPose
+    add_library(nvARPose INTERFACE)
+    target_include_directories(nvARPose INTERFACE ${SDK_INCLUDES_PATH})
 
-# Add target for NVCVImage
-add_library(NVCVImage INTERFACE)
-target_include_directories(NVCVImage INTERFACE ${SDK_INCLUDES_PATH})
+    # Add target for NVCVImage
+    add_library(NVCVImage INTERFACE)
+    target_include_directories(NVCVImage INTERFACE ${SDK_INCLUDES_PATH})
+
+else()
+    set(CMAKE_CXX_STANDARD 14)
+    set(CMAKE_CXX_STANDARD_REQUIRED ON)
+    set(CMAKE_CXX_EXTENSIONS OFF)
+
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread")
+
+    # Add target for ARSDK
+    add_library(nvARPose INTERFACE)
+
+    # found in different locations depending on type of package
+    find_path(ARSDK_INCLUDES
+        NAMES nvAR.h
+        PATHS
+        /usr/local/ARSDK/include
+        /usr/include/x86_64-linux-gnu
+        /usr/include
+        ${CMAKE_CURRENT_SOURCE_DIR}/nvar/include
+        REQUIRED
+        )
+
+    target_include_directories(nvARPose INTERFACE ${ARSDK_INCLUDES})
+
+    find_library(ARSDK_LIB
+        NAMES libnvARPose.so
+        PATHS
+        /usr/local/ARSDK/lib
+        /usr/lib/x86_64-linux-gnu
+        /usr/lib64
+        /usr/lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/bin
+        REQUIRED
+        NO_DEFAULT_PATH)
+
+    target_link_libraries(nvARPose INTERFACE "${ARSDK_LIB}")
+
+    message(STATUS "ARSDK_LIB: ${ARSDK_LIB}")
+
+
+    # Add target for NVCVImage
+    add_library(NVCVImage INTERFACE)
+
+    # found in different locations depending on type of package
+    find_path(NVCVImage_INCLUDES
+        NAMES nvCVImage.h
+        PATHS
+        /usr/local/ARSDK/include
+        /usr/include/x86_64-linux-gnu
+        /usr/include
+        ${CMAKE_CURRENT_SOURCE_DIR}/nvar/include
+        REQUIRED
+        )
+
+    target_include_directories(NVCVImage INTERFACE ${NVCVImage_INCLUDES})
+
+
+    find_library(NVCVImage_LIB
+        NAMES libNVCVImage.so
+        PATHS
+        /usr/local/ARSDK/lib
+        /usr/lib/x86_64-linux-gnu
+        /usr/lib64
+        /usr/lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/bin
+        REQUIRED
+        NO_DEFAULT_PATH)
+
+    target_link_libraries(NVCVImage INTERFACE "${NVCVImage_LIB}")
+
+    message(STATUS "NVCVImage_LIB: ${NVCVImage_LIB}")
+    message(STATUS "NVCVImage_INCLUDES_PATH: ${NVCVImage_INCLUDES}")
+
+
+endif()
 
 add_definitions(-DNOMINMAX -DWIN32_LEAN_AND_MEAN)
 

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2021 NVIDIA Corporation
+Copyright (c) 2020 NVIDIA Corporation
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in

README.MD

@@ -1,29 +1,37 @@
 # README
 ## NVIDIA MAXINE AR SDK: API Source Code and Sample Applications
 
-NVIDIA MAXINE AR SDK enables real-time modeling and tracking of human faces from video. The SDK is powered by NVIDIA graphics processing units (GPUs) with Tensor Cores, and as a result, the algorithm throughput is greatly accelerated, and latency is reduced.
+NVIDIA MAXINE AR SDK offers AI-based, real-time 3D face tracking and modeling, as well as body pose estimation based on a standard web camera feed. Developers can create unique AR effects such as overlaying 3D content on a face, driving 3D characters and virtual interactions in real time. The SDK is powered by NVIDIA graphics processing units (GPUs) with Tensor Cores, and as a result, the algorithm throughput is greatly accelerated, and latency is reduced.
 
 The SDK has the following features:
 
-- **Face detection and tracking**, which detects, localizes, and tracks human faces in images or videos by using bounding boxes.
-- **Facial landmark detection and tracking**, which predicts and tracks the pixel locations of human facial landmark points and head poses in images or videos. It can predict 68 and 126 landmark points. The 68 detected facial landmarks follow the _Multi-PIE 68 point mark-ups_ information in [facial point annotations](https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/). The 126 landmark points detector can predict more points on the cheeks, the eyes, and on laugh lines.
-- **Face 3D mesh and tracking**, which reconstructs and tracks a 3D human face and its head pose from the provided facial landmarks.
-- **3D Body Pose and tracking**, which predicts and tracks the 3D human pose from images or videos. It predicts 34 keypoints of body pose in 2D and 3D.
+- **Face tracking**, which detects, localizes, and tracks human faces in images or videos by using bounding boxes.
+- **Face landmark tracking**, which predicts and tracks the pixel locations of human facial landmark points using 68 or 126 landmark points. The 68 detected facial landmarks follow the Multi-PIE 68 point mark-ups information in [facial point annotations](https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/). The 126 landmark points detector can predict more points on the cheeks, the eyes, and on laugh lines. Additionally, it tracks head pose and facial deformation due to head movement and expression in three degrees of freedom in real time.
+- **Face mesh**, which reconstructs and tracks a human face via a 3D mesh, as well as its head pose, from the provided facial landmarks.
+- **Body Pose Estimation**, which predicts and tracks 34 key points of the human body, with joint angles, in 2D and 3D. It also supports multi-person tracking.
+- **Eye contact**, which simulates eye contact by estimating and aligning gaze with the camera to enhance engagement in video communication.
+- **Face Expression Estimation**, which estimates face expression (blendshape) coefficients from the provided facial landmarks.
 
 <p align="center">
-<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_001.png" alt="Face detection and tracking" width="320" height="180"/>
-<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_002.png" alt="Facial landmark detection and tracking - 68 pts" width="320" height="180" />
+<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_001.png" alt="Face tracking" width="320" height="180"/>
+<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_002.png" alt="Face landmark tracking - 68 pts" width="320" height="180" />
 </p><p align="center">
-<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_003.png" alt="Facial landmark detection and tracking - 126 pts" width="320" height="180"/>
-<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_004.png" alt="Face 3D mesh and tracking" width="320" height="180"/>
+<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_003.png" alt="Face landmark tracking - 126 pts" width="320" height="180"/>
+<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_004.png" alt="Face mesh" width="320" height="180"/>
 </p>
 </p><p align="center">
-<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_005.png" alt="Body 3D Pose and tracking" width="480" height="270"/>
+<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_005.png" alt="Body Pose estimation" width="480" height="270"/>
+</p><p align="center">
+<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_006.png" alt="Eye contact" width="640" height="237"/>
+</p><p align="center">
+<img src="https://github.com/NVIDIA/MAXINE-AR-SDK/blob/master/resources/ar_007.png" alt="Face Expression Estimation" width="640" height="175"/>
 </p>
 
-The SDK provides two sample applications that demonstrate the features listed above in real time by using a webcam or offline videos.
-- **FaceTrack App** which demonstrates the face tracking, landmark tracking and 3D mesh tracking features.
-- **BodyTrack App** which demonstrates the 3D Body Pose tracking feature.
+The SDK provides four sample applications that demonstrate the features listed above in real time by using a webcam or offline videos.
+- **FaceTrack App** which demonstrates the face tracking, landmark tracking and face mesh tracking features.
+- **BodyTrack App** which demonstrates the Body Pose estimation feature.
+- **GazeRedirect App** which demonstrates the Eye Contact feature.
+- **ExpressionApp** which demonstrates the Face Expression Estimation feature.
 
 NVIDIA MAXINE AR SDK is distributed in the following parts:
 
@@ -33,14 +41,12 @@ NVIDIA MAXINE AR SDK is distributed in the following parts:
 Please refer to [SDK System guide](https://docs.nvidia.com/deeplearning/maxine/ar-sdk-system-guide/index.html) for configuring and integrating the SDK, compiling and running the sample applications. Please visit the [NVIDIA MAXINE AR SDK](https://developer.nvidia.com/maxine-getting-started) webpage for more information about the SDK.
 
 ## System requirements
-The SDK is supported on NVIDIA GPUs that are based on the NVIDIA® Turing™ or Ampere™ architecture and have Tensor Cores.
+The SDK is supported on NVIDIA GPUs that are based on the NVIDIA® Turing™, Ampere™ or Ada™ architecture and have Tensor Cores.
 
 * Windows OS supported: 64-bit Windows 10 or later
 * Microsoft Visual Studio: 2017 (MSVC15.0) or later
 * CMake: v3.12 or later
-* NVIDIA Graphics Driver for Windows: 465.89 or later
-* NVIDIA CUDA Toolkit: 11.3.1
-* NVIDIA TensorRT: 8.0.1.6
+* NVIDIA Graphics Driver for Windows: 511.65 or later
 
 ## NVIDIA MAXINE Branding Guidelines
 If you integrate an NVIDIA MAXINE SDK within your product, please follow the required branding guidelines that are available [here](
@@ -64,7 +70,7 @@ The open source repository includes the source code to build the sample applicat
 *	To complete configuring the Visual Studio solution file, click Finish.
 *	To generate the Visual Studio Solution file, click Generate.
 *	Verify that the build folder contains the NvAR_SDK.sln file.
-3.  Use Visual Studio to generate the FaceTrack.exe or BodyTrack.exe file from the NvAR_SDK.sln file.
+3.  Use Visual Studio to generate the FaceTrack.exe, BodyTrack.exe, GazeRedirect.exe or ExpressionApp.exe file from the NvAR_SDK.sln file.
 *	In CMake, to open Visual Studio, click Open Project.
 *	In Visual Studio, select Build > Build Solution.