Skip to content
This repository has been archived by the owner on Jan 26, 2023. It is now read-only.

Commit

Permalink
RDKC-9030, RDKC-9366 : deliveryDetection in SmTN,2TN with same timestamp
Browse files Browse the repository at this point in the history 
Reason for change: Added delivery detection prebuilt binaries
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9030 : Implement delivery detection in SmartThumbnail

Reason for change: Added delivery detection thread. Added memory usage reduction.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9030 : Implement delivery detection in SmartThumbnail

Reason for change: Added delivery detection prebuilt binaries. Edited IMAGE
METADATA json string
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9030 : deliveryDetection in SmTN

Reason for change: Added new union bounding box for detection.
Added new logs.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9368, RDKC-9030 : Add delivery detection test support in TestHarness

Reason for change: Added methods to read frame from xvision and send
delivery information to xvision. Integrated mediapipe changes to fix
negative coordinates in person cropping.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9713, RDKC-9714 : Add debug symbols, Reduce Delivery Threshold to 0.3

Reason for change: Added debug symbols for crash analysis. Reduce the delivery
threshold to 0.3
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9713, RDKC-9727 : Add crash symbols, configurable detection constants

Reason for change: Added debug sym files for smartthumbnail, make constants
for delivery detection configurable.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9713, RDKC-9727 : Add crash symbols, configurable detection constants

Reason for change: Added debug sym files for smartthumbnail, make constants
for delivery detection configurable.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9707, RDKC-9746 : upload delivery notification anytime,mediapipe crash

Reason for change: Added debug symbols for crash analysis. Reduce the delivery
threshold to 0.3. safeguard crop region - to fix crash from mediapipe [change in file:
simple_image_cropping_calculator.cc]
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9707, RDKC-9746 : upload delivery notification anytime,mediapipe crash

Reason for change: Added debug symbols for crash analysis. Reduce the delivery
threshold to 0.3. safeguard crop region - to fix crash from mediapipe [change in file:
simple_image_cropping_calculator.cc]
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9752 : Crash during smartthumbnail upload with delivery metadata

Reason for change: Avoid creating smartthumbnail already one delivery detection
is in progress.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9758 : detection fails even if the score exceeds threshold

Reason for change: Corrected the default value macros
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9030, RDKC-9699 : Integrate mediapipe into RDKC

Reason for change: Added libdeliverydetector.so in LDFLAGS
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9701 : RFC feature control for delivery detection

Reason for change: Added RFC feature checking.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9992 : Smart thumbnails pending processing...discard

Reason for change: Corrected the chance for a deadlock that can
happen due to thread switch
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10066 : RFC check from service script

Reason for change: Added check for RFC feature change. Test Procedure: Build with no issues. Risks:LOW Signed-off-by: [email protected]

RDKC-9833 : Detection stops at thumbnail generation

Reason for change: Implemented as the new design. Detection stops at
thumbnail generation
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

Signed-off-by: Sajna Nazeer K <[email protected]>

RDKC-9758, RDKC-9858, RDKC-9888: DetectionStats,last frame,motion  issue

Reason for change: Added new parameters in Detection Stats log.
Changed mpipe_onThunbmailEvet api to mpipe_onLastFrameEvent.
Added a fix for do not uploading motion events issue.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-9888 : maximum of mean augmentation score in Detection Stats

Reason for change: Changed mean max augmentation score calculation.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10085 : Wrong value for the time upload thread waited

Reason for change: corrected the time calculation
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10093: continuous smartthumbnail process crash

Reason for change: Corrected the wrong memset
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10086 : Received motion ROI is continuously logged

Reason for change: Changed the logging
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10127: Delivery detection to run exactly on 1fps

Reason for change: Adjusted sleep time in
delivery detection thread to 1fps
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10159 : smart thumbnail cvr lite memory leak

Reason for change: rtMessage_Release() to relaese memory associated
with rtMessage in xcam2 instead of free().
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10175, RDKC-9700 : Revised Logic for Smart Thumbnail Cropping

Reason for change: cropping smartthumbnail from resized frame, if
union bounding box is larger than the thumbnail size
Also, added 400x300 cropping from buffer 1.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10167 : Excessive logs in smart thumbnail logs with delivery detection

Reason for change: modified mediapipe loglines to add timestamp.
Test Procedure: Build with no issues.
Risks:LOW
Signed-off-by: [email protected]

RDKC-10376 : Separate comcast proprietary code from thumbnail

Reason for change: Separating comcast edited mediapipe code to separate repo.
Test Procedure: Build with no issues.
Risks: LOW
Signed-off-by: [email protected]

Change-Id: Ia078c4973d850b0222248e031800594fe88142cd
  • Loading branch information
@Sajnakn @scthunderbolt
Sajnakn authored and scthunderbolt committed Nov 3, 2021
1 parent fda4b3b commit b42b40e
Show file tree
Hide file tree
Showing 13 changed files with 1,779 additions and 45 deletions.
26 changes: 26 additions & 0 deletions rdk_build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,11 @@ else
. ${RDK_PROJECT_ROOT_PATH}/build/components/sdk/setenv2
fi

if [ "$XCAM_MODEL" == "XHB1" ]; then
echo "Enabling delivery detection by default for DBC"
export ENABLE_DELIVERY_DETECTION=true
fi

export PLATFORM_SDK=${RDK_TOOLCHAIN_PATH}
export FSROOT=$RDK_FSROOT_PATH

Expand Down Expand Up @@ -133,9 +138,28 @@ function install()

make install

if [ $ENABLE_DELIVERY_DETECTION == 'true' ]; then
cp -r ./motion_notification/pre_built/mediapipe ${RDK_SDROOT}/etc
fi

echo "thumbnail Installation is done"
}

# This function enables delivery detection
function setDeliveryDetection()
{
echo "setDeliveryDetection - Enable delivery detection"
export ENABLE_DELIVERY_DETECTION=true
}

# This function enables TESTHARNESS
function setTH()
{
echo "setTH - Enable delivery detection and TestHarness"
export ENABLE_DELIVERY_DETECTION=true
export ENABLE_TEST_HARNESS=true
}

# This function disables XSTREAMER flag for Hydra
function setHydra()
{
Expand All @@ -149,6 +173,8 @@ HIT=false
for i in "$@"; do
case $i in
enableHydra) HIT=true; setHydra ;;
enableDeliveryDetection) HIT=true; setDeliveryDetection ;;
enableTH) HIT=true; setTH ;;
configure) HIT=true; configure ;;
clean) HIT=true; clean ;;
build) HIT=true; build ;;
Expand Down
16 changes: 16 additions & 0 deletions smart_thumbnail/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,14 @@ ifeq ($(XCAM_MODEL), SCHC2)
USE_DEWARP = yes
endif

ifeq ($(ENABLE_DELIVERY_DETECTION), true)
USE_MISC = yes
endif

ifeq ($(ENABLE_TEST_HARNESS), true)
CFLAGS += -DENABLE_TEST_HARNESS
endif

ENABLE_DIRECT_FRAME_READ := true

ifneq ($(ENABLE_XSTREAMER), true)
Expand Down Expand Up @@ -84,6 +92,14 @@ LIBS += -L$(RDK_PROJECT_ROOT_PATH)/opensource/lib -lopencv_core -lopencv_img
LIBS += -Wl,-rpath-link=$(PROJ_ROOT)/vendor/img/fs/shadow_root/usr/lib
CFLAGS += -DUSE_MFRLIB

ifeq ($(ENABLE_DELIVERY_DETECTION), true)
CFLAGS += -D_OBJ_DETECTION_
SRC += mpipe_port.cpp
CFLAGS += -I$(BUILD_ROOT_DIR)/opensource/include
LIBS += -L$(BUILD_ROOT_DIR)/opensource/lib -lbase64
LIBS += -ldeliverydetector
endif

SRC += smart_thumbnail.cpp main.cpp

OBJ = $(SRC:.cpp=.o)
Expand Down
46 changes: 46 additions & 0 deletions smart_thumbnail/include/mpipe_port.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#ifndef MPIPE_PORT_H__
#define MPIPE_PORT_H__

#include <vector>
#include <string>

typedef struct DetectionResult_ {
/* The bounding boxes of each (n) person detected in the thumbnail */
std::vector<std::vector<int>> personBBoxes;
/* The detection score of each (n) person detected in the thumbnail */
std::vector<float> personScores;
float deliveryScore;
int maxAugScore;
} DetectionResult;

void *__mpipe_thread_main__();

/* open the device in the given path. The 'width and height' is the resolution that
* mpipe desires. If the camera does not support, the camera's resolution is updated
* to widht and height */
int mpipe_port_initialize(const std::string &input_video_path, int &width, int &height);
int mpipe_port_initialize(int &width, int &height);

/* get next frame, in COLOR_RGB format */
cv::Mat mpipe_port_getNextFrame();
void mpipe_port_term();


/* ----------------------------------------------------------------------
* The following are mpipe APIs available for porting application to call.
* porting application should not implement these APIs.
* ----------------------------------------------------------------------*/
/* header only. Do not implement */
/* notify mpipe of last frame for person detection, in cv::Mat, RGB
* the lastFrame can be empty or non-empty. If non-empty, it will be
* process for person dection before continue onto delivery detection.
*/
void mpipe_port_onLastFrameEvent(const cv::Mat &thumbnail, int width, int height);
/* notify mpipe of motion deteced event */
/* header only. Do not implement */
void mpipe_port_onMotionEvent(bool motionDetected);
typedef void (* OnDetectionChanged_t) (const DetectionResult &detection_result);
/* mpipe will notify application of detection results */
/* header only. Do not implement */
void mpipe_port_setOnDetectionChanged(OnDetectionChanged_t cb);
#endif
98 changes: 96 additions & 2 deletions smart_thumbnail/include/smart_thumbnail.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@
#include <fcntl.h>
#include <vector>
#include <iterator>
#ifdef _OBJ_DETECTION_
#include <jansson.h>
#include <fileUtils.h>
#include "RFCCommon.h"
#endif
#ifdef ENABLE_TEST_HARNESS
#include "dev_config.h"
#endif

#ifdef LEGACY_CFG_MGR
#include "dev_config.h"
Expand Down Expand Up @@ -65,6 +73,11 @@ extern "C" {

#include "opencv2/opencv.hpp"

#ifdef _OBJ_DETECTION_
#include "mpipe_port.h"
#include "base64.h"
#endif

#define CACHE_SMARTTHUMBNAIL "/tmp/cache_smart_thumbnail.txt"

#define FW_NAME_MAX_LENGTH 512
Expand All @@ -77,6 +90,7 @@ extern "C" {
#define STN_HRES_BUFFER_ID 2
#elif defined(XHB1) || defined (XHC3)
#define STN_HRES_BUFFER_ID 2
#define STN_MRES_BUFFER_ID 1
#else
#define STN_HRES_BUFFER_ID 0
#endif
Expand All @@ -90,6 +104,12 @@ extern "C" {
//default width and height of smart thumbnail
#define STN_DEFAULT_WIDTH 640
#define STN_DEFAULT_HEIGHT 480
#ifdef XHB1
#define STN_HRES_CROP_WIDTH 640
#define STN_HRES_CROP_HEIGHT 480
#define STN_MRES_CROP_WIDTH 400
#define STN_MRES_CROP_HEIGHT 300
#endif

#define STN_TIMESTAMP_TAG "timestamp"
#define STN_UPLOAD_TIME_INTERVAL 30
Expand Down Expand Up @@ -117,6 +137,31 @@ extern "C" {
#define BLOB_BB_MAX_LEN 256
#define INVALID_BBOX_ORD (-1)

#ifdef ENABLE_TEST_HARNESS
#define TEST_HARNESS_ON_FILE_ENABLED "Test_Harness_on_File_Enabled"
#endif

#ifdef _OBJ_DETECTION_
#define DEFAULT_INPUT_DEV "/dev/video0"
#define DEFAULT_GRAPH_PATH "/etc/mediapipe/graphs/rdk/delivery_detection/g_delivery_detection_cpu.pbtxt"
#define DEFAULT_FRAME_READ_DELAY "1000"
#define DEFAULT_MAX_FRAMES_CACHED_FOR_DELIVERY_DETECTION "5"
#define DEFAULT_DELIVERY_DETECTION_MODEL_MIN_SCORE_THRESHOLD "0.3"
#define DEFAULT_DELIVERY_DETECTION_MIN_SCORE_THRESHOLD "1"
#define DETECTION_CONFIG_FILE "/opt/usr_config/detection_attr.conf"
#define DEFAULT_FRAME_COUNT_TO_PROCESS "5"

typedef struct detection_config_ {
std::string input_video_path;
std::string delivery_detection_graph_path;
std::string frame_read_delay;
std::string max_num_frames_cached_for_delivery_detection;
std::string delivery_detection_model_min_score_threshold;
std::string delivery_detection_min_score_threshold;
std::string frame_count_to_process;
}DetectionConfig;
#endif

typedef enum {
STH_ERROR = -1,
STH_SUCCESS,
Expand Down Expand Up @@ -146,6 +191,11 @@ typedef struct {
#ifdef _HAS_DING_
uint64_t dingtstamp;
#endif
#ifdef _OBJ_DETECTION_
json_t *detectionResult;
uint64_t motionTime;
bool deliveryDetected;
#endif
}STHPayload;

typedef struct {
Expand All @@ -170,11 +220,12 @@ class SmartThumbnail
public:
static SmartThumbnail* getInstance();
//Initialize the buffers and starts msg monitoring, upload thread.
STH_STATUS init(char* mac,bool isCVREnabled,int stnondelayType,int stnondelayTime);
STH_STATUS init(char* mac,bool isCVREnabled,int stnondelayType,int stnondelayTime,bool isDetectionEnabled);
//get upload status
bool getUploadStatus();
//set upload status
STH_STATUS setUploadStatus(bool status);

//Upload smart thumbnail data
void uploadPayload();
//notify start or end of smart thumbnail process
Expand All @@ -183,6 +234,17 @@ class SmartThumbnail
STH_STATUS destroy();
//Thread routine to receive data
STH_STATUS receiveRtmessage();
#ifdef _OBJ_DETECTION_
#ifdef ENABLE_TEST_HARNESS
void notifyXvision(const DetectionResult &result);
void waitForNextDetectionFrame();
#endif
STH_STATUS setDetectionStatus(bool status);
bool getDetectionStatus();
void onCompletedDeliveryDetection(const DetectionResult &result);
friend cv::Mat mpipe_port_getNextFrame();
friend unsigned char * mpipe_port_getNextFrame(int *h, int *w);
#endif

static void sigHandler(int signum);

Expand All @@ -195,6 +257,10 @@ class SmartThumbnail
STH_STATUS delSTN(char* uploadFname);
void printSTNList();
STH_STATUS createPayload(char* uploadFname);
#ifdef _OBJ_DETECTION_
STH_STATUS updateUploadPayload(char * fname, DetectionResult result);
json_t* createJSONFromDetectionResult(DetectionResult result);
#endif
STH_STATUS getTnUploadConf();
STH_STATUS getEventConf();
//sets the camera firmware version.
Expand All @@ -211,7 +277,7 @@ class SmartThumbnail
STH_STATUS resizeAspect(cv::Mat im, int w, int h, cv::Mat& im_resized);
cv::Point2f getActualCentroid(cv::Rect boundRect);
cv::Point2f alignCentroid(cv::Point2f orgCenter, cv::Mat origFrame, cv::Size cropSize);
cv::Size getCropSize(cv::Rect boundRect,double w,double h);
cv::Size getCropSize(cv::Rect boundRect,double w,double h, double *rescaleSize);
cv::Rect getRelativeBoundingBox(cv::Rect boundRect, cv::Size cropSize, cv::Point2f allignedCenter);
void stringifyEventDateTime(char* strEvtDateTime , size_t evtdatetimeSize, time_t evtDateTime);
void resetObjFrameData();
Expand Down Expand Up @@ -266,6 +332,30 @@ class SmartThumbnail
bool isPayloadAvailable;
std::mutex stnMutex;
std::mutex uploadMutex;
#ifdef _OBJ_DETECTION_
std::mutex detectionMutex;
cv::Mat nextFrameForDetection;
unsigned char* mpipe_hres_yuvData;
cv::Rect currentBbox;
bool detectionCompleted;
json_t *detectionResult;
struct timeval detectionStartTime, detectionEndTime, uploadTriggeredTime, clipStartTime;
int mpipeProcessedframes;
double motionTriggeredTime;
char currDetectionSTNFname[CONFIG_STRING_MAX];
bool detectionInProgress;
bool detectionEnabled;
#ifdef ENABLE_TEST_HARNESS
bool testHarnessOnFileFeed;
std::vector<cv::Mat> yuvPlanes, yuvChannels;
cv::Mat fileFrameYUV, planeUV, curr_frame;
uint64_t currTstamp, detectionTstamp;
std::condition_variable detectionCv;
std::mutex hres_data_lock;
int THFileNum, THFrameNum;
int FileNum = 0, FrameNum = 0;
#endif
#endif
std::condition_variable cv;
rtConnection connectionRecv;
rtConnection connectionSend;
Expand Down Expand Up @@ -295,6 +385,7 @@ class SmartThumbnail

int sTnHeight;
int sTnWidth;
uint16_t buf_id;
char uploadFname[CONFIG_STRING_MAX];
char currSTNFname[CONFIG_STRING_MAX];
char modelName[CONFIG_STRING_MAX];
Expand All @@ -319,6 +410,9 @@ struct SmarttnMetadata_thumb
char const *strFramePTS;
int32_t event_type;
double motionScore;
#ifdef _OBJ_DETECTION_
BoundingBox deliveryUnionBox;
#endif
BoundingBox unionBox;
BoundingBox objectBoxs [UPPER_LIMIT_BLOB_BB];
char const *s_curr_time;
Expand Down
23 changes: 22 additions & 1 deletion smart_thumbnail/main.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,23 @@
#include "breakpadwrap.h"
#endif

#ifdef _OBJ_DETECTION_
SmartThumbnail *smTnInstance = NULL;
#endif

int main(int argc, char** argv)
{
#ifndef _OBJ_DETECTION_
SmartThumbnail *smTnInstance = NULL;
#endif
STH_STATUS status = STH_SUCCESS;
time_t remainingTime = 0;
char *param = NULL;
const char* debugConfigFile = "/etc/debug.ini";
int cvrEnabled = 0;
int stnondelayType = 0;
int stnondelayTime = 60;
int isDetectionEnabled = 0;

struct timespec currTime;
struct timespec startTime;
Expand Down Expand Up @@ -121,6 +128,20 @@ int main(int argc, char** argv)
break;
}
}

if(strcmp(argv[itr],"--detectionEnabled")==0)
{
itr++;

if (itr < argc)
{
isDetectionEnabled = atoi(argv[itr]);
}
else
{
break;
}
}
itr++;
}

Expand All @@ -135,7 +156,7 @@ int main(int argc, char** argv)
}

//initialize smart thumbnail
status = smTnInstance-> init(param,cvrEnabled,stnondelayType,stnondelayTime);
status = smTnInstance-> init(param,cvrEnabled,stnondelayType,stnondelayTime,isDetectionEnabled);
if (STH_ERROR == status) {
RDK_LOG( RDK_LOG_ERROR,"LOG.RDK.SMARTTHUMBNAIL","%s(%d): Error creating Smart thumbnail instance.\n", __FILE__, __LINE__);
return STH_ERROR;
Expand Down
Loading

0 comments on commit b42b40e

Please sign in to comment.