Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dim check while calculating mIoU&pixAcc for segmentation test #991

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

chunhanl
Copy link

  1. The test.py example mentioned at
    segmentation.rst
    does not match the arguments mentioned in test.py
    should probably be change to
    python test.py --dataset ade20k --pretrained --model fcn --backbone resnet50 --eval

  2. The dimension doesn't seem right when calculating the pixACC and mIoU
    The output is 3D and the target is 2D
    The safest way I could think of is to add a new axis if batch dim is missing.
    Might have a better way to deal with.

Result after modification (without training)
Screenshot from 2019-10-17 20-26-19

@chunhanl chunhanl changed the title Add dim check for segmentation Add dim check while calculating mIoU&pixAcc for segmentation test Oct 18, 2019
@chunhanl
Copy link
Author

Apologies for the new spaces, it was automatically modified by Pycharm IDE

@mli
Copy link
Member

mli commented Oct 18, 2019

Job PR-991-1 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-991/1/index.html
Code coverage of this PR: pr.svg vs. Master: master.svg

@@ -31,7 +31,7 @@ Table of pre-trained models for semantic segmentation and their performance.
The test script :download:`Download test.py<../../scripts/segmentation/test.py>` can be used for
evaluating the models (VOC results are evaluated using the official server). For example ``fcn_resnet50_ade``::

python test.py --dataset ade20k --model-zoo fcn_resnet50_ade --eval
python test.py --dataset ade20k --pretrained --model fcn --backbone resnet50 --eval
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why make this change?

Copy link
Author

@chunhanl chunhanl Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello,
According in test.py

def parse_args():

The args defined in parse_args() no longer has args 'model-zoo'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add that back instead. Not sure who removed that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether this may have conflicts with these settings.

model_prefix = args.model + '_' + args.backbone
if 'pascal' in args.dataset:
model_prefix += '_voc'
withQuantization = True if (args.backbone in ['resnet101']) else withQuantization
elif args.dataset == 'coco':
model_prefix += '_coco'
withQuantization = True if (args.backbone in ['resnet101']) else withQuantization
elif args.dataset == 'ade20k':
model_prefix += '_ade'
elif args.dataset == 'citys':
model_prefix += '_citys'
else:
raise ValueError('Unsupported dataset {} used'.format(args.dataset))
if args.ngpus > 0:
withQuantization = False
if withQuantization and args.quantized:
model_prefix += '_int8'
if not args.deploy:
if args.calibration:
args.pretrained = True
# create network
if args.pretrained:
model = get_model(model_prefix, pretrained=True)
model.collect_params().reset_ctx(ctx=args.ctx)

# the category -1 is ignored class, typically for background / boundary
if len(output.shape) == 3:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batch_intersection_union is supposed to work for a batch with 4D.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK I'll try to modify from the dataset

@chunhanl
Copy link
Author

python test.py --dataset ade20k --pretrained --model fcn --backbone resnet50 --eval
Screenshot from 2019-10-18 16-35-31

python test.py --dataset pascal_voc --pretrained --model fcn --backbone resnet101 --eval
Screenshot from 2019-10-18 16-31-43

@mli
Copy link
Member

mli commented Oct 18, 2019

Job PR-991-2 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-991/2/index.html
Code coverage of this PR: pr.svg vs. Master: master.svg

@@ -120,7 +120,7 @@ def test(model, args, input_transform):
tbar = tqdm(test_data)
for i, (data, dsts) in enumerate(tbar):
if args.eval:
predicts = [pred[0] for pred in evaluator.parallel_forward(data)]
predicts = [pred for pred in evaluator.parallel_forward(data)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incorrect. The output of gluoncv segmentation network is always a list

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants