From 1fa6da722715e4c5c6e0bdafbd576468b35b7aac Mon Sep 17 00:00:00 2001 From: Chris Choy Date: Sun, 3 Nov 2019 06:02:47 -0800 Subject: [PATCH] discussion thread --- README.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 49a9701..9f16a19 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,16 @@ Modify the `BATCH_SIZE` accordingly. The first argument is the GPU id and the second argument is the path postfix and the last argument is the miscellaneous arguments. -The official evaluation metric for ScanNet is mIoU. OA, Overal Accuracy is not the official metric similar to the 2D image semantic segmentation as it is a lot easier and doesn't reflect the quality of the semantic segmentation. This is due to the fact that most of the scenes consist of large structures such as walls, floors and these will dominate the statistics. Thus, overall accuracy should not be used solely to prevent reviewers from comparing the baselines fairly. + +### mIoU vs. Overall Accuracy + +The official evaluation metric for ScanNet is mIoU. +OA, Overal Accuracy is not the official metric since it is not discriminative. This is the convention from the 2D semantic segmentation as the pixelwise overall accuracy does not capture the fidelity of the semantic segmentation. +On 3D ScanNet semantic segmentation, OA: 89.087 -> mIOU 71.496 mAP 76.127 mAcc 79.660 on the ScanNet validation set v2. + +Then why is the overall accuracy least discriminative metric? This is due to the fact that most of the scenes consist of large structures +such as walls, floors, or background and scores on these will dominate the statistics if you use Overall Accuracy. + ## Synthia 4D Experiment