Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OutOfRangeError exception running objectTrackingSingleFrame.py. Intentional? #14

Open
petrock99 opened this issue Feb 23, 2021 · 2 comments
Open

OutOfRangeError exception running objectTrackingSingleFrame.py. Intentional? #14

petrock99 opened this issue Feb 23, 2021 · 2 comments

Comments

@petrock99
Copy link

petrock99 commented Feb 23, 2021 •
edited
Loading

Hello, when I run

python objectTrackingSingleFrame.py --seq 'test'

for '1.1. Object pose initialization' from the ReadMe I get an OutOfRangeError exception at what appears to be the very end of the run:.

maskPC for Image test/0/02294 is 0.036168
[Loading New frame ][0/02294]
'runOptimization'  63.31 ms
0.0034784062
0.0529846
0.026492303
'runOptimization'  561.64 ms
maskPC for Image test/0/02297 is 0.035820
[Loading New frame ][0/02297]
'runOptimization'  78.96 ms
0.003642647
0.060919605
0.03045981
'runOptimization'  1479.31 ms
Traceback (most recent call last):
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
	 [[{{node cond/IteratorGetNext}}]]
	 [[cond/IteratorGetNext/_15]]
  (1) Out of range: End of sequence
	 [[{{node cond/IteratorGetNext}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "objectTrackingSingleFrame.py", line 329, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 142, in objectTracker
    opti1.runOptimization(session, 1, {loadData:True})
  File ".../HOnnotate/HOnnotate/optimization/ghope/utils.py", line 159, in timed
    result = method(*args, **kw)
  File ".../HOnnotate/HOnnotate/optimization/ghope/optimization.py", line 69, in runOptimization
    session.run(self.optOp, feed_dict=feedDict)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
	 [[node cond/IteratorGetNext (defined at .../HOnnotate/HOnnotate/optimization/ghope/loss.py:227) ]]
	 [[cond/IteratorGetNext/_15]]
  (1) Out of range: End of sequence
	 [[node cond/IteratorGetNext (defined at .../HOnnotate/HOnnotate/optimization/ghope/loss.py:227) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'cond/IteratorGetNext':
  File "objectTrackingSingleFrame.py", line 329, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 62, in objectTracker
    frameCntInt, loadData, realObservs = LossObservs.getRealObservables(ds, numFrames, w, h)
  File ".../HOnnotate/HOnnotate/optimization/ghope/loss.py", line 242, in getRealObservables
    lambda: dummyFunc(fidV, segV, depthV, colV, maskV, frameCntIntV))
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1977, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1814, in BuildCondBranch
    original_result = fn()
  File ".../HOnnotate/HOnnotate/optimization/ghope/loss.py", line 241, in <lambda>
    frameID, seg, depth, col, mask, frameCntInt = tf.cond(loadRealObservs, lambda: loadVars(fidV, segV, depthV, colV, maskV, frameCntIntV),
  File ".../HOnnotate/HOnnotate/optimization/ghope/loss.py", line 227, in loadVars
    frameID, seg, depth, col, mask = dataset.make_one_shot_iterator().get_next()
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 426, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1947, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

I'm wondering if this is expected/intended as the way for the script to bail out of the while(True) loop in objectTracker(...) in objectTrackingSingleFrame.py? If I add a try/except around session.run(self.optOp, feed_dict=feedDict) in runOptimization(...) the script never ends (due to the while(True) in objectTracker(...).

                try:
                    session.run(self.optOp, feed_dict=feedDict)
                except tf.errors.OutOfRangeError:
                    break # break the while(True)

Should I instead, put the try/except around the call site for objectTracker so that the while(True) will bail out? Is there a better method? Thanks.

                try:
                    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
                except tf.errors.OutOfRangeError:
                    pass	# We're done

Thanks a lot.

Python: 3.6.12
HOnnotate git hash: d94f6b7
dirt git hash: 571addc359201b668d9dc450086c6dce6c18d0b6
CUDA: 11
tensorflow: 1.14
GCC: 8.3.1
@petrock99
Copy link
Author

btw, the same problem happens in handObjectTrack in handObjectTrackingSingleFrame.py with the runOptimization on line 190. Frustrating to say the least since it took almost an hour to get to that point.

[Loading New frame ][02298]
'runOptimization'  82.18 ms
'runOptimization'  1429.59 ms
Traceback (most recent call last):
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
	 [[{{node cond/IteratorGetNext}}]]
  (1) Out of range: End of sequence
	 [[{{node cond/IteratorGetNext}}]]
	 [[cond/IteratorGetNext/_19]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "handObjectTrackingSingleFrame.py", line 488, in <module>
    handObjectTrack(w, h, objParamInitList[0], handParamInitList[0], mesh, camProp, out_dir)
  File "handObjectTrackingSingleFrame.py", line 190, in handObjectTrack
    opti1.runOptimization(session, 1, {loadData: True})
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/utils.py", line 159, in timed
    result = method(*args, **kw)
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization
    session.run(self.optOp, feed_dict=feedDict)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
	 [[node cond/IteratorGetNext (defined at /s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py:225) ]]
  (1) Out of range: End of sequence
	 [[node cond/IteratorGetNext (defined at /s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py:225) ]]
	 [[cond/IteratorGetNext/_19]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'cond/IteratorGetNext':
  File "handObjectTrackingSingleFrame.py", line 488, in <module>
    handObjectTrack(w, h, objParamInitList[0], handParamInitList[0], mesh, camProp, out_dir)
  File "handObjectTrackingSingleFrame.py", line 65, in handObjectTrack
    frameCntInt, loadData, realObservs = LossObservs.getRealObservables(ds, numFrames, w, h)
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py", line 240, in getRealObservables
    lambda: dummyFunc(fidV, segV, depthV, colV, maskV, frameCntIntV))
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1977, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1814, in BuildCondBranch
    original_result = fn()
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py", line 239, in <lambda>
    frameID, seg, depth, col, mask, frameCntInt = tf.cond(loadRealObservs, lambda: loadVars(fidV, segV, depthV, colV, maskV, frameCntIntV),
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py", line 225, in loadVars
    frameID, seg, depth, col, mask = dataset.make_one_shot_iterator().get_next()
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 426, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1947, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

@shreyashampali
Copy link
Owner

Hi,
Sorry for the late reply here.
You are right, the tracking scripts do not have a graceful exit at the sequence and the problem is mainly with the while(true) loop. Irrespective of whether the script crashes or not at the end, you should have the poses dumped in the output folders which are required for subsequent stages. One way to fix the crash at the end of the sequence is to replace the while loop with a for loop if you know the number of frames you are tracking for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shreyashampali @petrock99