add SIFT Flow FCNs

These nets are jointly trained for segmentation of semantic and
geometric classes since this dataset includes annotations for both.

- FCN-32s SIFT Flow
- FCN-16s SIFT Flow
- FCN-8s SIFT Flow

TODO: fix semantic class evaluation for this dataset, which requires
special care since there are missing classes in the test set.

README.md

@@ -29,17 +29,26 @@ Unlike the FCN-32/16/8s models, this network is trained with gradient accumulati
 
 To reproduce the validation scores, use the [seg11valid](https://github.com/shelhamer/fcn.berkeleyvision.org/blob/master/data/pascal/seg11valid.txt) split defined by the paper in footnote 7. Since SBD train and PASCAL VOC 2011 segval intersect, we only evaluate on the non-intersecting set for validation purposes.
 
-**NYUDv2 models**: trained online with high momentum on color, depth, and HHA features (from Gupta et al. https://github.com/s-gupta/rcnn-depth):
+**NYUDv2 models**: trained online with high momentum on color, depth, and HHA features (from Gupta et al. https://github.com/s-gupta/rcnn-depth).
+These models demonstrate FCNs for multi-modal input.
 
 * [FCN-32s NYUDv2 Color](nyud-fcn32s-color): single stream, 32 pixel prediction stride net on color/BGR input
 * [FCN-32s NYUDv2 HHA](nyud-fcn32s-hha): single stream, 32 pixel prediction stride net on HHA input
 * [FCN-32s NYUDv2 Early Color-Depth](nyud-fcn32s-color-d): single stream, 32 pixel prediction stride net on early fusion of color and (log) depth for 4-channel input
 * [FCN-32s NYUDv2 Late Color-HHA](nyud-fcn32s-color-hha): single stream, 32 pixel prediction stride net by late fusion of FCN-32s NYUDv2 Color and FCN-32s NYUDv2 HHA
 
-**The following models have not yet been ported to master and trained with the latest settings. Check back soon.**
+**SIFT Flow models**: trained online with high momentum for joint semantic class and geometric class segmentation.
+These models demonstrate FCNs for multi-task output.
+
+* [FCN-32s SIFT Flow](siftflow-fcn32s): single stream stream, 32 pixel prediction stride net
+* [FCN-16s SIFT Flow](siftflow-fcn16s): two stream, 16 pixel prediction stride net
+* [FCN-8s SIFT Flow](siftflow-fcn8s): three stream, 8 pixel prediction stride net
 
-SIFT Flow model (also fine-tuned from VGG-16):
-* [FCN-16s SIFT Flow](https://gist.github.com/longjon/f35e3a101e1478f721f5#file-readme-md): two stream, 16 pixel prediction stride version
+*Note*: in this release, the evaluation of the semantic classes is not quite right at the moment due to an issue with missing classes.
+This will be corrected soon.
+The evaluation of the geometric classes is fine.
+
+**The following models have not yet been ported to master and trained with the latest settings. Check back soon.**
 
 PASCAL-Context models including architecture definition, solver configuration, and bare-bones solving script (fine-tuned from the ILSVRC-trained VGG-16 model):
 * [FCN-32s PASCAL-Context](https://gist.github.com/shelhamer/80667189b218ad570e82#file-readme-md): single stream, 32 pixel prediction stride version

data/sift-flow/README.md

@@ -0,0 +1,16 @@
+# SIFT Flow
+
+SIFT Flow is a semantic segmentation dataset with two labelings:
+
+- semantic classes, such as "cat" or "dog"
+- geometric classes, consisting of "horizontal, vertical, and sky"
+
+Refer to `classes.txt` for the listing of classes in model output order.
+Refer to `../siftflow_layers.py` for the Python data layer for this dataset.
+
+Note that the dataset has a number of issues, including unannotated images and missing classes from the test set.
+The provided splits exclude the unannotated images.
+As noted in the paper, care must be taken for proper evalution by excluding the missing classes.
+
+Download the dataset:
+http://www.cs.unc.edu/~jtighe/Papers/ECCV10/siftflow/SiftFlowDataset.zip

data/sift-flow/classes.txt

@@ -0,0 +1,47 @@
+Semantic and geometric segmentation classes for scenes.
+
+Semantic: 0 is void and 1–33 are classes.
+
+01 awning
+02 balcony
+03 bird
+04 boat
+05 bridge
+06 building
+07 bus
+08 car
+09 cow
+10 crosswalk
+11 desert
+12 door
+13 fence
+14 field
+15 grass
+16 moon
+17 mountain
+18 person
+19 plant
+20 pole
+21 river
+22 road
+23 rock
+24 sand
+25 sea
+26 sidewalk
+27 sign
+28 sky
+29 staircase
+30 streetlight
+31 sun
+32 tree
+33 window
+
+Geometric: -1 is void and 1–3 are classes.
+
+01 sky
+02 horizontal
+03 vertical
+
+N.B. Three classes (cow, desert, and moon) are absent from the test set, so
+they are excluded from evaluation. The highway_bost181 and street_urb506 images
+are missing annotations so these are likewise excluded from evaluation.

data/sift-flow/test.txt

@@ -0,0 +1,200 @@
+coast_natu975
+insidecity_art947
+insidecity_urb781
+highway_bost374
+coast_n203085
+insidecity_a223049
+mountain_nat116
+street_art861
+mountain_land188
+street_par177
+opencountry_natu524
+forest_natu29
+highway_gre37
+street_bost77
+insidecity_art1125
+street_urb521
+highway_bost178
+street_art760
+street_urb885
+insidecity_art829
+coast_natu804
+mountain_sharp44
+coast_natu649
+opencountry_land691
+insidecity_hous35
+tallbuilding_art1719
+mountain_n736026
+mountain_moun41
+insidecity_urban992
+opencountry_land295
+tallbuilding_art527
+highway_art238
+forest_for114
+coast_land296
+tallbuilding_sky7
+mountain_n44009
+tallbuilding_art1316
+forest_nat717
+highway_bost164
+street_par29
+forest_natc52
+tallbuilding_art1004
+coast_sun14
+opencountry_land206
+opencountry_land364
+mountain_n219015
+highway_a836030
+forest_nat324
+opencountry_land493
+insidecity_art1598
+street_street27
+insidecity_a48009
+coast_cdmc889
+street_gre295
+tallbuilding_a538076
+street_boston378
+highway_urb759
+street_par151
+tallbuilding_urban1003
+tallbuilding_urban16
+highway_bost151
+opencountry_nat965
+highway_gre661
+forest_for42
+opencountry_n18002
+insidecity_art646
+highway_gre55
+coast_n295051
+forest_bost103
+highway_n480036
+mountain_land4
+forest_nat130
+coast_nat643
+insidecity_urb250
+street_gre11
+street_boston271
+opencountry_n490003
+mountain_nat762
+street_par86
+coast_arnat59
+mountain_land787
+highway_gre472
+opencountry_tell67
+mountain_sharp66
+opencountry_land534
+insidecity_gre290
+highway_bost307
+opencountry_n213059
+forest_nat220
+forest_cdmc348
+tallbuilding_art900
+insidecity_art569
+street_urb200
+coast_natu468
+coast_n672069
+insidecity_hous109
+forest_land862
+opencountry_natu65
+tallbuilding_a805096
+opencountry_n291058
+forest_natu439
+coast_nat799
+tallbuilding_urban991
+tallbuilding_sky17
+opencountry_land638
+opencountry_natu563
+tallbuilding_urb733
+forest_cdmc451
+mountain_n371066
+mountain_n213081
+mountain_nat57
+tallbuilding_a463068
+forest_natu848
+tallbuilding_art306
+insidecity_boston92
+insidecity_urb584
+tallbuilding_urban1126
+coast_n286045
+street_gre179
+coast_nat1091
+opencountry_nat615
+coast_nat901
+forest_cdmc291
+mountain_natu568
+mountain_n18070
+street_bost136
+tallbuilding_art425
+coast_bea3
+tallbuilding_art1616
+insidecity_art690
+highway_gre492
+highway_bost320
+forest_nat400
+highway_par23
+tallbuilding_a212033
+forest_natu994
+tallbuilding_archi296
+highway_gre413
+tallbuilding_a279033
+insidecity_art1277
+coast_cdmc948
+forest_for15
+street_par68
+mountain_natu786
+opencountry_open61
+opencountry_nat423
+mountain_land143
+tallbuilding_a487066
+tallbuilding_art1751
+insidecity_hous79
+street_par118
+highway_bost293
+mountain_n213021
+opencountry_nat802
+coast_n384099
+opencountry_natu998
+mountain_n344042
+coast_nat1265
+forest_text44
+forest_for84
+insidecity_a807066
+opencountry_nat1117
+coast_sun42
+insidecity_par180
+opencountry_land923
+highway_art580
+street_art1328
+coast_cdmc838
+opencountry_land660
+opencountry_cdmc354
+coast_natu825
+opencountry_natu38
+mountain_nat30
+coast_n199066
+forest_text124
+forest_land222
+tallbuilding_city56
+tallbuilding_city22
+opencountry_fie36
+mountain_ski24
+coast_cdmc997
+insidecity_boston232
+opencountry_land575
+opencountry_land797
+insidecity_urb362
+forest_nat1033
+mountain_nat891
+street_hexp3
+tallbuilding_art1474
+tallbuilding_urban73
+opencountry_natu852
+mountain_nat1008
+coast_nat294
+mountain_sharp20
+opencountry_fie14
+mountain_land275
+forest_land760
+coast_land374
+mountain_nat426
+highway_gre141