Skip to content

Correction Mode with Smoothing Interpolation

Jump to bottom
Allen Lee edited this page Jul 19, 2016 · 6 revisions

Questions/Open Issues

  • Currently manual unlocking can only occur frame-by-frame (using the button). We could add bulk lock/unlocking, eg movie-wide, or in selected intervals.

Usage

  1. In pref.yaml, set Track: Type: to Interpolator (no quotes necessary). See Interpolator.m.
  2. Open Labeler
  3. Open a project (.lbl file)
  4. Let's assume that each movie in the project has a .trk file located in the same dir as the movie, with name .trk. File > Import ... to read the .trk files into the project. After this call, the trk positions should appear as labels.
  5. Select the "Error Correction" labeling mode under the 'Setup' menu. (Other modes can also be used, but are not tailored for manual label correction.)
  6. Use the Track > Set Parameters File to set the tracking yaml. See /interpolator.yaml for an example.
  7. Adjust labels using any/all of the following
  • Click-drag any point
  • Select a point using number key; then use arrows, or click
  1. Frames that have been "touched" will become Locked. They can be manually unlocked using the button. At the moment, locking is set at the frame-level (all pts in a frame are either locked or not).
  2. If desired, select a time interval by toggling the "Select" button below the timeline; then click-dragging the timeline.
  3. Select frames-to-be-tracked using pulldown menu below Track button. Eg: Current Movie, Selected Frames.
  4. Click Track to run the Interpolator.
  • I put in a toy algo for the current Interpolator algorithm so you should update accordingly.
  • When tracking is complete, the new interpolated labels are set back on the Labeler.
  1. Repeat 7-11.
  2. To export labels, use File > Export Labels to Trk Files.

Description

A set of labels is given, eg in .trk files. Maybe they were generated by a tracker (based on a previous, "original" set of manual labels). The user would like to review the labels and adjust/correct them.

During manual adjustment, we would like to know which points have been corrected, and display them differently.

After adjusting some labels, the user would like an automated smoothing/interpolation process to correct labels in neighboring frames, eg to enforce a smoothness constraint. This process should not alter labels manually adjusted by the user, ie manually-adjusted points should be "locked". The user may want to specify that this interpolation only occur within a certain interval containing the manually-adjusted points, as opposed to the entire movie.

When the user is satisfied with the final labels, he/she exports them to .trk files.

Solution/Implementation

  • The given set of labels (.trk files) is converted into a .lbl project with a utility function. Alternatively, Labeler may import .trk files for its current movies under the File> menu. If the project-to-be will contain many movies, the utility function may be easier as interactively creating the project could be cumbersome.
  • An "Error Correction" labeling mode (similar to Template mode) will be added to facilitate manual adjustment of labels. This mode will display "touched" labels differently. Users will be able to manually "untouch" labels. See below.
  • A simple concrete Tracker ("Interpolator") can be created to do the interpolation. It does not need training so Interpolator.train() can do nothing. On track(), it just interpolates/smoothes etc as appropriate.
    • In this use case, tracked labels will become the labels themselves, rather than being displayed as separate "tracked results". After tracking, Interpolator will use Labeler's labeledpos API to set the tracking results into the labels.
    • Note, any adjustments to points performed by Interpolator will not be considered "touched". "Touched" will mean manually-touched. Any manually-touched points will remain manually-touched after tracking/interpolation.
  • A timeline with window-drag will be added that will be used to specify frames to Track. An option "Current movie, selected frames" will be added to the Tracking pulldown menu.
    • The user-selected frames-to-be-tracked are passed to Interpolator.track(). It will be up to Interpolator not to set points outside the selected window. Labeler could enforce this, but for now it is the honor system.
  • At this point, the user is browsing the generated labels; adjusting as appropriate; creating windows where he/she would like to see re-interpolated results; and pushing Track to perform the interpolation. Repeat as necessary.
  • When the user is satisfied, Labeler exports the corrected/interpolated labels to .trk files using the File> menu.

Error Correcting label mode

  • Regular/untouched labels are displayed one way
  • Touched labels are displayed with emphasis (bolder colors etc)
  • Labeling action is same as Template mode. "Accept" will be shown as "Lock"
  • New .labeledposTouched flag stored (logicals) for touched-ness
  • Points can be "marked as untouched" either en masse or individually
  • Generally, all frames are expected to have labels already. Frames with no labels will just not have anything shown. But Template-mode actions will work, eg hitting a number key and mouse-clicking will bring the point on-screen (as touched of course).
Clone this wiki locally