Issue with no detections but training showed model convergence

[Fred-Macdo]

I am having an issue running test using simrdwn on my own dataset. I was using the same dataset I trained on to test so I figured it would work fine. It looks like the model is not making any detections. Did I run training incorrectly or format the data incorrectly?
I have train data in png format and tried to use tiff or png for test and was unable to get results.

Args from training log file:

Args: Namespace(BGR2RGB=0, alpha_scaling=0, append_date_string=1, batch_size=64, boxes_per_grid=5, building_csv_file='', core_dir='/simrdwn/simrdwn/core', date_string='2019_08_14_15-31-26', dpi=300, edge_buffer_test=-1000, extension_list=['.png', '.tif', '.TIF', '.TIFF', '.tiff', '.JPG', '.jpg', '.JPEG', '.jpeg'], figsize=(12, 12), framework='yolt2', gpu=0, inference_graph_path2='/raid/local/src/simrdwn/outputs/ssd/output_inference_graph/frozen_inference_graph.pb', inference_graph_path_tot='/simrdwn/results/frozen_model/frozen_inference_graph.pb', keep_test_slices=False, label_map_dict={1: 'Destroyed_Building'}, label_map_dict_rev={'Destroyed_Building': 1}, label_map_dict_rev_tot={'Destroyed_Building': 1}, label_map_dict_tot={1: 'Destroyed_Building'}, label_map_path='/simrdwn/data/train_data/class_labels_car.pbtxt', label_map_path2='', labels_log_file='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/logs/labels_list.txt', log_dir='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/logs', log_file='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/logs/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26.log', max_batches=30000, max_edge_aspect_ratio=3, min_retain_prob=0.025, mode='train', multi_band_delim='#', n_test_output_plots=10, nbands=3, nms_overlap_thresh=0.5, now=datetime.datetime(2019, 8, 14, 15, 31, 26, 722083), outname='dense_DestBuilding', overwrite_inference_graph=0, plot_line_thickness=2, plot_thresh=array([0.3]), plot_thresh_str='0.3', plot_thresh_str2='0.3', res_name='train_yolt2_dense_DestBuilding_2019_08_14_15-31-26', results_dir='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26', results_topdir='/simrdwn/results', rotate_boxes=False, save_json=1, show_labels=0, show_test_plots=0, shuffle_val_output_plot_ims=0, simrdwn_dir='/simrdwn', single_gpu_machine=0, slice_overlap=0.35, slice_sizes=array([416]), slice_sizes2=[], slice_sizes_str='416', slice_sizes_str2='0', str_delim=',', subdivisions=16, test_add_geo_coords=True, test_box_rescale_frac=1.0, test_im_compression_level=6, test_ims_list=[], test_make_legend_and_title=True, test_prep_only=0, test_presliced_list='', test_presliced_list_tot='/simrdwn/results/', test_presliced_tfrecord_path='', test_presliced_tfrecord_tot='', test_slice_sep='__', test_splitims_locs_file='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/test_splitims_input_files.txt', test_splitims_locs_file_root='test_splitims_input_files.txt', test_splitims_locs_file_root2='test_splitims_input_files2.txt', test_tfrecord_out='', testims_dir='test_images', testims_dir_tot='/simrdwn/data/test_images/test_images', tf_cfg_dir='tf/cfg', tf_cfg_train_file='tf/cfg/', tf_cfg_train_file_out='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/logs/', tf_model_output_directory='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/frozen_model', tf_plot_file='/simrdwn/simrdwn/core/tf_plot_loss.py', this_file='/simrdwn/simrdwn/core/simrdwn.py', train_data_dir='/simrdwn/data/train_data', train_input_height=416, train_input_width=416, train_model_path='', train_model_path2='', train_tf_record='/simrdwn/data/train_data/', train_val_tf_record='', use_CUDNN=1, use_GPU=1, use_opencv='1', use_tfrecords=0, val_df_path_aug='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/test_predictions_aug.csv', val_df_path_init='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/test_predictions_init.csv', val_df_path_tot='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/test_predictions_tot.csv', val_df_root_aug='test_predictions_aug.csv', val_df_root_aug2='test_predictions_aug2.csv', val_df_root_init='test_predictions_init.csv', val_df_root_init2='test_predictions_init2.csv', val_df_root_tot='test_predictions_tot.csv', val_prediction_df_refine_tot='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/test_predictions_refine_thresh=0.3', val_prediction_df_refine_tot_root_part='test_predictions_refine', weight_file='/simrdwn/yolt2/input_weights/yolov2.weights', weight_file2='', weight_file_tot='/simrdwn/yolt2/input_weights/yolov2.weights', yolov3_filters=10, yolt_cfg_dir='/simrdwn/yolt2/cfg', yolt_cfg_file='yolo.cfg', yolt_cfg_file2='yolo.cfg', yolt_cfg_file_in='/simrdwn/yolt2/cfg/yolo.cfg', yolt_cfg_file_tot='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/logs/yolo.cfg', yolt_classnum=1, yolt_dir='/simrdwn/yolt2', yolt_final_output=30, yolt_loss_file='/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/logs/yolt_loss.txt', yolt_nms_thresh=0.0, yolt_object_labels=['Destroyed_Building'], yolt_object_labels_str='Destroyed_Building', yolt_plot_file='/simrdwn/simrdwn/core/yolt_plot_loss.py', yolt_test_classes_files=['/simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26/Destroyed_Building.txt'], yolt_train_images_list_file='SR_yolt_train_list.txt', yolt_train_images_list_file_tot='/simrdwn/data/train_data/SR_yolt_train_list.txt', yolt_weight_dir='/simrdwn/yolt2/input_weights', zero_frac_thresh=0.5)

The test command is here:
python simrdwn.py --framework yolt2 --mode test --outname santarosa_test4 --label_map_path /simrdwn/data/train_data/class_labels_car.pbtxt --train_model_path /simrdwn/results/train_yolt2_dense_DestBuilding_2019_08_14_15-31-26 --testims_dir /simrdwn/results/test__SR_test_2019_08_16_18-03-24/SantaRosa_split --test_presliced_list /simrdwn/results/test__SR_test_2019_08_16_18-03-24/test_splitims_input_files.txt --weight_file yolo_final.weights --yolt_cfg_file yolo.cfg --n_test_output_plots 4 --edge_buffer_test 1 --plot_thresh_str 0.15 --batch_size 4 --min_retain_prob 0.03 --show_labels 1 --alpha_scaling 1

Traceback:

test_file: /simrdwn/results/test_yolt2_santarosa_test4_2019_08_16_20-06-27/Destroyed_Building.txt
Augmenting dataframe of initial length: 0 ...
set image path, make sure the image exists...
Time to augment dataframe of length: 0 = 0.015589714050292969 seconds
Adding geo coords...
Traceback (most recent call last):
File "simrdwn.py", line 1941, in
main()
File "simrdwn.py", line 1933, in main
execute(args, train_cmd1, test_cmd_tot, test_cmd_tot2)
File "simrdwn.py", line 1545, in execute
test_add_geo_coords=args.test_add_geo_coords)
File "simrdwn.py", line 1134, in run_test
outProj_str='epsg:3857', verbose=verbose)
File "/simrdwn/simrdwn/core/add_geo_coords.py", line 229, in add_geo_coords_to_df
df_['lon0'] = out_arr[:, 0]
IndexError: too many indices for array

[avanetten]

Without knowing your training command and output it's impossible to tell if training ran correctly. I would check the loss file in the training directory to ensure that training converged.

Also, the issue here is not with add_geo_coords. Notice that you have no detections, so the failure is not in add_geo_coords, but in the lack of detections.

[Fred-Macdo]

This is my training command. The loss file showed that loss started around >25 and ended around .025

python /simrdwn/simrdwn/core/simrdwn.py
--framework yolt2
--mode train
--outname dense_DestBuilding
--yolt_object_labels_str Destroyed_Building
--yolt_cfg_file yolo.cfg
--weight_file /simrdwn/yolt2/input_weights/yolov2.weights
--yolt_train_images_list_file SR_yolt_train_list.txt
--label_map_path /simrdwn/data/train_data/class_labels_car.pbtxt
--max_batches 30000
--batch_size 64
--subdivisions 16
--gpu 0

[sdrll]

Hi @Fred-Macdo, did you resolved this issue? I have exactly the same issue

[1246886181]

Hi @Fred-Macdo, did you resolved this issue? I have exactly the same issue

did you resolved this issue? I have exactly the same issue