Kwiver Release 1.8.0

KWIVER v1.8.0 Release Notes

This is a minor release of KWIVER that provides both new functionality and fixes
over the previous v1.7.0 release.

Updates

KWIVER

• Adopted C++17.

• Added setup_KWIVER.ps1.

• Added the MSVC 2022 compiler as a case in our CMake logic.

Vital

Vital Algo

• Added API for algorithms to find nearest neighbor to a set of point in 3D.

• Expanded the pointcloud_io API to include the ability to load point cloud data

• Added option to open file given to metadata_map_io in binary mode.

• Added buffered_metadata_filter.

Vital Types

• Added new pointcloud type to hold point cloud data

• Added VITAL_META_UNIX_TIMESTAMP_SOURCE, VITAL_META_VIDEO_DATA_STREAM_SYNCHRONOUS.

• Removed VITAL_META_ICING_DETECTED.

• Added equality operators for metadata_item.

• Added metadata_[io]stream.

Vital Util

• Added support for different text codecs (ASCII, UTF-8, etc).

• Added threeway_compare().

Arrows

• Passed a constant seed to the random point generator used by several tests to
  prevent occasional random failure.

Arrows: Core

• Implemented a general-purpose CSV reader and writer.

• Prevent UV coordinates from going to NaN for degenerate triangles in uv_unwrap_mesh.

• Fixed race condition in close_loops_exhaustive that could result in a crash
  when matching and merging feature tracks.

• Fixed undefined behavior leading to a crash in track_features_core when the
  track set remained empty after the first frame.

• Fixed an incorrect cast of a geo_point to double in derive_metadata.

Arrows: FFmpeg

• Added support for FFmpeg 5.1.2.

• Added basic configuration options to ffmpeg_video_output.

• Added CUVID video decoding/encoding support.

• Added support in ffmpeg_video_output for non-RGB24 input images.

• Fixed many bugs with timestamps.

• Implemented direct copying of video stream packets from input to output.

• Improved robustness against invalid input data.

• Added config options to ffmpeg_video_input to ignore the video or KLV streams.

• Properly handle MISP nanosecond timestamps.

• Added padding to end of image memory to avoid segfaults.

• Improved the search paths to find an FFmpeg installation.

• Fixed alignment between KLV packets and frames when reading and writing.

• Limit total amount of past KLV maintained in memory.

• Added new synthetic test videos.

• Added derivation of KLV synchronicity from stream and packet characteristics.

• Made initial probing logic more persistent.

Arrows: KLV

• Implemented ST1107.

• Updated ST0903 implementation to version 6.

• Implemented the BOOLEAN, UINT, and RLE APAs in ST1303.

• Adjusted tag number requirements of ST0806.

• Changed data type of EG0104 Episode Number to string.

• Implemented configuration of KLV muxing update rates.

• Modified interface to use pointers when reading/writing instead of
  std::vector iterators.

• Used std::nullopt to indicate no ST0601 control command timestamp instead of 0.

• Used std::chrono to better express MISP timestamps.

• Changed type of ST0601 Waypoint Record bitfield.

• Added detailed length constraint information.

• Fixed bug when calculating length of ST0601 Airbase Locations.

• Fixed bug which prevented detection of some too-long KLV packet lengths.

• Added apply_child_klv and update_klv metadata filters.

• Implemented CRC-8-CCITT checksum.

• Fixed bug when ST0903 local set packs fail to parse.

• Made headers public which were included by other public headers.

• Improved descriptions of formats.

• Implemented handling of different string encodings.

• Fixed possible out-of-bounds memory read leading to crash when KLV parsing
  fails.

• Fixed bug in KLV packet reader which failed to check if the packet's length
  was larger than the available number of input bytes.

• Fixed bug in ST0601 view domain writer which would not write the length of
  the final field.

• Added a missing type to the ST0601 KLV JSON exporter.

Arrows: PDAL

• Added implementation of the load API to pointcloud_io to load point data via the PDAL library

Arrows: Serialize JSON

• Fixed bug when loading a KLV local set item with a null value.

• Fixed bug preventing the proper loading of strings with embedded null ('\0')
  characters.

• Increased the precision of exported floating point values.

• Added a configuration option to compress JSON on output.

Arrows: Zlib

• Added this arrow with bytestream_compressor functionality.

KwiverSys

• Made compiling regular expression thread-safe.

Applets

Applets: bundle adjustment tool

• Added an implementation bundle adjustment (BA) applet that optimizes a set of
  cameras given multiple 3D-2D point correspondences.

Applets: transcode

• Fixed a bug in which metadata was erroneously written after each frame,
  instead of before.

Applets: dump-klv

• Added an option to use multithreading when dumping frame images.

• Removed early exit when no KLV stream is found.

Applets: compare-klv

• Added this applet.

Python:

• Removed the custom pybind11.h header file since the current version of
  pybind11 now handles cases where the GIL is acquired via the
  PyGILState_* API.

Kwiver Release 1.7.0

KWIVER v1.7.0 Release Notes

This is a minor release of KWIVER that provides both new functionality and fixes
over the previous v1.6.1 release.

Updates

Vital

Vital Types

• Modified rotation_ constructor from yaw-pitch-roll to keep the given
  North-East-Down coordinate system as-is instead of automatically converting to
  East-North-Up. Added utility functions which perform conversion between NED
  and ENU coordinates.

Build System

• Removed some CMake statements and support for old CMake version predating the
  current minimum version specification.

Vital KLV

• Moved KLV to a new arrow.

Arrows

Arrows: Core

• Added transcode applet.

• Added frame dumping capability to dump-klv.

• Added the option to choose the metadata export format in dump-klv.

• Added the derive_metadata filter.

• Enabled exporting components of composite data types separately in
  metadata_map_io_csv.

• Added frame decimation to metadata_map_io_csv.

• Allowed the overriding of column names in metadata_map_io_csv.

Arrows: FFmpeg

• Enabled reading MISP timestamps from stream.

• Added additional video metadata reporting.

• Added exporting of encoding settings from input video.

• Added basic video encoder.

Arrows: KLV

• Moved KLV arrow from Vital.

• Ensured proper handling of ST0601 Report-on-Change behavior.

• Rewrote EG0104 and ST0601 to be able to read and write all tags.

• Newly implemented reading and writing for the following standards: ST0102,
  ST0806, ST0903, ST1102, ST1010, ST1108, ST1202, ST1204, ST1206, ST1303,
  ST1601, ST1602, and ST1607.

Arrows: Serialize

• Added partial KLV-JSON reader/writer.

Bug Fixes

Arrows: Core

• Fixed metadata_map_io_csv printing the wrong number of commas for some empty
  fields.

Kwiver Release 1.6.1

KWIVER v1.6.1 Release Notes

This is a patch release of KWIVER that provides fixes over the previous v1.6.0 release.

Bug Fixes since v1.6.0

Arrows: Ceres

• Fixed a bug in which the FORCE_COMMON_INTRINSICS option in bundle
  adjustment was not actually forcing intrinsics to be common.

Arrows: FFMPEG

• Fixed a bug in which slight variations in presentation timestamps would
  sometimes cause frames to be skipped while decoding.

• Fixed a bug in which the video reader would not request any frames at end of
  video that FFmpeg still had buffered internally.

Arrows: MVG

• Fixed a typo in the command line argument name for camera path in the
  init_cameras_landmarks applet. This was preventing changing the camera
  directory in the applet.

Arrows: VTK

• Fixed an error in the calculation of depth used in mesh coloring. This
  caused coloring to fail to apply color in some cases. It was primarily
  an issue when the near plane was much closer to the camera than to the
  far plane.

• Fix an error that always caused a segmentation fault in the
  color_mesh applet for VTK version greater than 9.0.1. The issue was
  that rendering was not properly initialized at build time.

Build System

• Fix missing ZeroMQ_INCLUDE_DIR set when building statically.

• Fix missing-closing-bracket typo in CMake/utils/kwiver-utils-python.cmake.

• Use xvfb in Docker image to support running VTK applets in a headless
  environment.

• Include CUDA include directories when building CUDA arrow. This is needed
  if CUDA is not installed in standard system locations.

• Enable the CUDA arrow in the Docker container.

Vital Algorithms

• Fix potential leak of stream pointers in the base functionality of
  write_object_track_set.

Vital: Logging

• Fixed an issue in the default logger where leading zeros of fractional
  seconds were being stripped leading to the printing of the incorrect time.

Kwiver Release 1.6.0

KWIVER v1.6.0 Release Notes

This is a minor release of KWIVER that provides both new functionality and fixes
over the previous v1.5.0 release.

Updates since v1.5.0

• Renamed symbols or other approaches to reduce "shadowing" warnings.

Vital

Vital Algo

• Extended the API for compute_depth to also return an uncertainty image.

• Added API for camera resectioning.

Vital Applets

• Added utility functions to help validate config file entries for input
  and output files and directories.

Vital Config:

• Added use of environment variable KWIVER_CONFIG_PREFIX in conjunction with previous
  search strategy for config location.

• Introduced function using config search pattern to locate files, returning valid path(s).

Vital Types

• Added new method to metadata class to add a metadata item being
  passed by shared_ptr.

• Added methods to track to facilitate removing states by frame number or iterator.

• Use specific typing frame_id_t for track_features::track.

Vital Plugin-loader

• Added support for filtering plugins when they are loaded. Filter
  objects can be added to the plugin loader at run time to select or
  exclude specific plugins.

• Added optional bit mask to load_all_plugins() which allows selected
  groups of plugins to be loaded. In addition the build/install
  directory structure for the plugins was reorganized.

• Internal change to consolidate determining and configuring the
  build-time plugin search paths.

Vital Util

Vital Logger

Vital Tools

Vital Types

Vital Bindings

• Remove ctypes python bindings

• Moved python bindings associated with config from sprokit.pipeline to
  vital.config.

• Force logging level of TRACE to be DEBUG in bindings because Python logging
  does not support the TRACE level.

Arrows

• Cleaned up arrows library links. Many arrows linked to the core arrow
  when really only needed to link directly to vital libraries.

Arrows: Core

• Updated dump-klv applet to load default configuration from a file.

• Updated dump-klv to support option including metadata in write out.

• Added a utility function to depth_utils.h to find similar cameras by view
  angle.

• Added a utility function to depth_utils.h to gather corresponding cameras,
  images, and masks from video sources for use in depth map estimation.

• Added serialization of video metadata to CSV output for file export.

Arrows: Ceres

• Improve logging in bundle_adjust to allow logging per-iteration summaries
  and independently control logging fully summaries after optimization
  completes.

• Refactored camera estimation configuration options and moved those utilities
  to the new MVG arrow for easier reuse without depending on Ceres Solver.

Arrows: CUDA

Arrows: FFmpeg

• Now checks the timestamps of the metadata packets to catch packets
  that are arriving early. Packets that arrive early are cached until the
  timestamp of the current frame is larger than the timestamp of the
  metadata packet. This synchronization can be disabled if desired.

• Fixed an issue where only metadata packets from the first data stream
  are saved.

• Added a libavfilter pipeline into the ffmpeg_video_input reader. This
  allows users to inject filters in the video reader, much the same as
  is possible in the ffmpeg command line with the -vf option. The default
  filter applies deinterlacing to any interlaced frames.

• Fixed the memory leak that occurred when a video was opened and closed.

Arrows: GDAL

Arrows: KPF

Arrows: MVG

• Created new MVG arrow to make a new home for Multi-View Geometry and
  other 3D vision code. Much of this code will move from the "core" arrow
  which had grown too large.

• Moved track_features applet from core into mvg.

• Moved the following algorithms from core into mvg:

  • hierarchical_bundle_adjust
  • initialize_cameras_landmarks_keyframe (renamed to initialize_cameras_landmarks)
  • initialize_cameras_landmarks (renamed to initialize_cameras_landmarks_basic)
  • triangulate_landmarks

• Moved the following utility headers from core into mvg:

  • epipolar_geometry.h
  • interpolate_camera.h
  • necker_reverse.h
  • metrics.h
  • projected_track_set.h
  • sfm_utils.h
  • transform.h
  • triangulate.h

• Added a new applet, init_cameras_landmarks, to run the estimate cameras
  and landmarks from feature tracks produced by the track_features applet.

• Added an implementation of integrate_depth_maps. This works exactly the same
  as the implementation in the CUDA arrow, but does not require a GPU.

Arrows: OpenCV

• Added a new camera resectioning algorithm implementation using
  cv::calibrateCamera().

• Added Telea inpainting method.

• Added Navier-Stokes inpainting method.

• Added Black-Mask inpainting method.

• Added OpenCV implementation of resection camera algorithm.

Arrows: PROJ

• Accepted use of PROJ 5 and 6.

Arrows: Serialization

• Added capacity to read and write video metadata as json types.

Arrows: Super3d

• Changed the compute_depth parameters to take a sampling rate, in pixels,
  instead of an absolute number of depth slices. This allows the algorithm
  the scale the number of slices appropriately ensuring at least a minimum
  sampling rate when projecting into support images.

• Updated the Super3D compute_depth algorithm to compute uncertainty.

Arrows: VTK

• Created new VTK arrow.

• Added utility function to convert depth images to vtkImageData.
  This function was ported from TeleSculptor.

• Added an applet to estimate depth maps from multiple views.

• Added a new applet for fusing depth maps in volume and extracting a
  surface mesh.

Arrows: VXL

• Added pixel extractor type converter.

Arrows: Python

Sprokit

Sprokit: Pipeline

• Added signal indicating pipeline endcap input termination.

• Embedded pipelines search config paths on construction.

Sprokit: Processes

• Added four new PyTorch-using Python Sprokit processes for tracking
  (srnn_tracker), descriptor computation (alexnet_descriptors and
  resnet_descriptors), and augmentation (resnet_augmentation).

• Added method to request data from Adapter Dataset with support for a
  fallback option if data is unavailable.

• Replaced API call get_port_data with more conventional value.

Sprokit: Bindings

• Removed redundant logger support sprokit.sprokit_logging module.

Tools

Build System

• Added a new build flag, KWIVER_ENABLE_PYTORCH, to include
  PyTorch-dependent code in the build.

• Enabled the use of folders for CMake targets to provided better
  organization in IDEs that support folders, like Visual Studio.

Unit Tests

General Documentation

Python

• Added argument to build_pipeline command to include additonal directories in which
  to search for pipe files to load.

• Python virtual env creation and dependency install removed from build process.
  Added Kwiver Python dependency install to setup script instead.

• Added Python Algo for Camera resection Vital Algo.

• Added metadata_map_io Python Algo.

Bug Fixes

Arrows: Core

• Fixed several uses of wrong values in the core implementation of
  compute_ref_homography.

• Fixed failure to report end of video status.

Arrows: Ceres

Arrows: CUDA

• Fixed an issue in which depth imagery was loaded transposed and flipped
  and then later flipped and transposed back during CUDA processing.

Arrows: FFmpeg

• Prevented looping seeks in some video files.

• Fixed imporper setting of capability metadata.

Arrows: Super3D

Sprokit: Processes

• Fixed the compute_track_descriptors process's handling of null
  output from its nested algorithm.

• Fixed configuration check for read_object_track, enabling configuration to
  succeed.

Sprokit: Pipelines

Python:

• Fixed kwiver.arrows.core mesh exposure to module importation.

Kwiver Release 1.5.1

KWIVER v1.5.1 Release Notes

This is a patch release of KWIVER that provides fixes over the previous
v1.5.0 release.

Bug Fixes since v1.5.0

Config

• Add support for Visual Studio 2019 in the setup_KWIVER.bat

Vital

• Added a missing include file in track.cxx required to build on MSVC 2019.

• Changed incorrect metadata tag in convert_0601_metadata.cxx from
  KLV_0601_TARGET_LOCATION_LAT to KLV_0601_TARGET_LOCATION_ELEV.

• Fixed a typo in the TARGET_LOCATION metadata field name.

• Updated metadata field names for kv::geo_point types to be more
  descriptive. Made the distinction between elevation and altitude.

Arrows: Ceres

• Fixed a bug in which the update_state_every_iteration config option
  for bundle_adjust was not respected and was always manually overridden with
  "true" if a callback was registered.

Arrows: Core

• Remove some unnecessary computation in initialize_camera_landmarks_keyframe
  by adding checks to detect earlier when computations that cannot achieve
  their desired result due to insufficient data.

• Fixed a bug in initialize_cameras_landmarks_keyframe in which the
  principal point was not initialized from the image size unless there
  was also metadata useful for determining focal length. Distortion
  parameters were also not properly preserved in this initialization
  and this has also been resolved.

• Fixed a bug in initialize_cameras_landmarks_keyframe in which the
  camera estimation loop would sometimes exit early without processing
  the remaining cameras. This is further improved to try the lists of
  remaining cameras multiple times until no further cameras are added.

• Fixed a bug in initialize_cameras_landmarks_keyframe in which the
  updated landmarks are not returned from a helper function and therefore
  intermediate results are returned as the final output.

• Fixed a bug in track features that was causing a crash on Linux after
  detecting features on the first frame of video.

Arrows: CUDA

• Fixed a linking error on Linux with some versions of CUDA requiring
  to "recompile with -fPIC".

• Fixed a bug in launching the integrate_depth_maps CUDA kernel.
  Previously, the kernel would fail to launch if the size of the volume
  along the X direction was greater than 1024.

• Added a max_voxels_per_launch configuration option to the CUDA
  integrate_depth_maps algorithm. This works around a CUDA limitation
  in which long-running CUDA kernels cause the GPU to freeze the system.
  On Windows this leads to a time out after 2 seconds and the application
  is terminated. To avoid this we now launch the kernel multiple times
  covering a subset of voxels in each chunk.

Arrows: OpenCV

• Fixed improper use of RANSAC in the OpenCV estimate_pnp algorithm.
  This algorithm was producing a lower then expected success rate when
  resectioning cameras.

• Fixed a long standing bug with memory management when converting between
  OpenCV and Vital images. Reference counting was not handled correctly.
  A workaround was previously added that forced a deep copy, but this created
  another bug resulting in images with invalid first_pixel() values.
  Both of these issues have been addressed.

Arrows: Serialize

• Fixed a bug in the JSON serializer for metadata where bool fields caused a
  runtime error.

• Fixed a bug in JSON serialization of metadata maps which caused name
  collisions for multiple metadata packets associated the same frame. This
  made the output unusable by applications other than KWIVER.

KWIVER Release 1.5.0

KWIVER v1.5.0 Release Notes

This is a minor release of KWIVER that provides both new functionality and fixes over the previous v1.4.0 release.

Updates since v1.4.0

Config

• Default configuration files have been added to KWIVER for several
  algorithms. These configuration files provide recommended ways to combine
  algorithms across arrows, assuming all arrows are available. The
  configuration files are installed and may be loaded automatically by
  applets as their default configuration.

Vital

Vital Algo

• Changed the callback API on compute_depth to accept a string status
  message. Integer progress is now interpreted as percent complete
  rather than number of iterations.

• Added a variant of the integrate_depth_maps API that takes weight images
  along with depth images. This feature was added to include binary image
  masks in the fusion, but it also extends to continuous weight values
  between zero and one.

Vital Types

• Added kwiver::vital::set mixin class to define standard set methods as
  well as iterability.

• Added iterator, const_iterator and iterable classes to support container-
  independent iteration.

• Added a parameter to the clone() methods of the track_state, track and
  track_set types controlling whether to make an actual deep copy (default) or
  to make a shallow copy (previous behavior) where elements of the derived
  feature_track_state or object_track_state would still be shared. (This
  change also affects track_set::merge_in_other_track_set(). However, the copy
  constructors of feature_track_state and object_track_state still make only
  shallow copies.)

• Added a variant of config_block::get_enum_value that accepts a default
  value to fall back on.

• Added support for registering callbacks to the KWIVER logger. Callbacks
  receive any message that is logged with the KWIVER logging framework.

Vital Plugin-loader

• Added support for filtering plugins when they are loaded. Filter
  objects can be added to the plugin loader at run time to select or
  exclude specific plugins.

Vital Logger

• Log4cplus now searches in more locations to find the log4cplus.properties
  file during start up. If not found with the environment variable or in the
  current working directory it will now search in the location of the
  executable and also in "../lib/kwiver" relative to the executable.

Vital Types

• Added functions to camera_intrinsics to test if a point is in the valid
  domain of the mapping function. In the radial distortion case of
  simple_camera_intrinsics, this computes a maximum radius for which
  distortion is valid. This feature is to help avoid invalid projections
  of points into images due to the distortion being non-injective.

• Added get_vector() member function to access the vector of all metadata on
  a frame in metadata_map.

• Added kwiver::vital::set mixin class to define standard set methods as
  well as iterability.

• Added iterator, const_iterator and iterable classes to support container-
  independent iteration.

• Refactored detected_object_set to convert some methods to external
  functions since they were not properties of the class.

• Vital types protobuf support now requires protobuf version 3 or greater.

Vital Bindings

• Remove ctypes python bindings

• Moved python bindings associated with config from sprokit.pipeline to
  vital.config.

Vital Config

• Extended the application_config_file_paths() function to list configuration
  file paths for both the application name provided and for KWIVER-provided
  configuration files. This allows KWIVER-provided configuration files to be
  found if the file is not found in an application configuration directory
  first.

Arrows

Arrows: Core

• Added an algorithm implementation (transform_detected_object_set) for
  transforming detected object sets from one camera perspective to another.
  Implements vital::algo::detected_object_filter.

• Added an algorithm implementation (transfer_bbox_with_depth_map) for
  transforming detected object set bounding boxes from one camera perspective
  to another with respect to a depth map. Implements
  vital::algo::detected_object_filter.

• Added a detect_features adaptor that applies a filter_features
  algorithm to filter features returned by the detector. This allows
  a filter to be injected anywhere a detector is normally used.

• Added an adaptive non-max suppression filter for features points.
  This algorithm provides more uniformity to detected features by
  removing all features within a radius of the strongest points.
  The radius is dynamically adapted to achieve a target point count.

• Added an option to the fundamental matrix matcher that does a second
  round of filtering to remove outliers with large motion relative to
  most other matches.

• When resectioning cameras in the keyframe initializer we now copy the
  intrinsics from the nearest existing camera rather than from the base
  camera in the config or metadata. This way the focal length better matches
  existing cameras and the initial pose will better match as well.

• During keyframe camera initialization the algorithm detects if the camera
  graph has split into multiple components. Previously only the largest
  component was kept resulting in significant data loss. Instead the
  algorithm now tries to retriangulate tracks that previously held the
  camera graph together.

• Refactored initialize_cameras_landmarks_keyframe code for better
  reuseability and readability.

• Updated the initialize_cameras_landmarks_keyframe algorithm to better
  handle early termination through callbacks.

• Added a new SfM utility function to compute a set of well-distributed
  but well-connected keyframes for use in initializing SfM.

• Use a canonical transform estimator in initialize_cameras_landmarks_keyframe
  to reorient the solution when no metadata constraints are available.
  This also helps detect upside down cameras and trigger a Necker reversal to
  correct for that case.

• Added utility functions to estimate the ground center of landmarks and
  to apply a translational offset to landmarks and cameras.

• Added a utility to compute the subset of cameras located on the positive
  side of a specified plane.

• Improved Necker reversal correction in the keyframe camera and landmark
  initializer by allowing reversal of individual cameras that are upside down
  provided that all cameras are above the estimated ground plane.

• In the keyframe camera and landmark initializer, only use windowed bundle
  adjustment for filling in cameras after the entire initial sequence is
  complete.

• In the keyframe camera and landmark initializer, we no longer update
  the base_camera intrinsics configuration value with estimated values.
  These should remain fixed to their configured settings. Instead we now
  use nearby established cameras to initialize similar or shared intrinsics.

• Added a utility to compute the subset of cameras located on the positive
  side of a specified plane.

• Improved Necker reversal correction in the keyframe camera and landmark
  initializer by allowing reversal of individual cameras that are upside down
  provided that all cameras are above the estimated ground plane.

Arrows: Ceres

• Fixed long-broken bundle adjustment regularization parameters for
  camera smoothness and forward motion damping. The fixed versions now
  support non-sequential camera frame numbers and the residual costs are
  normalized relative to both the number of existing residuals and the
  spatial scale of the camera positions. These normalization allow the
  regularization to work similarly across different problem sizes and scales.

• Added a configuration option to the bundle adjuster that allow for
  callbacks to not provided updates on every iteration for efficiency.

• Add a new option for minimum horizontal field of view (hfov) in bundle
  adjustment and camera optimization. This is mostly for telescopic lenses
  when the hfov is very small (e.g. less than 5 degrees). There is a
  tendency for small FOVs to drift toward zero due to lack of perspective.
  This new parameter introduces a large cost if the HFOV drops below the
  set threshold.

Arrows: CUDA

• Added a new parameter for the truncated signed distance function in depth
  map integration. The new "Epsilon" value is used in the occluded portion
  of the function that was previously zero. Setting Epsilon > 0 can remove
  a double surface artifact. Setting Epsilon = 0 restores the previous
  behavior.

• CUDA depth map integration now uses optional weight maps to down-weight
  the contribution of some rays.

Arrows: FFMPEG

• Optimized the num_frames() call to produce a count of the number of frames
  in constant time without scanning the entire video and counting all frames.

Arrows: Serialization

• Updated serialization algorithms to use PLUGIN_INFO macro and added
  serializer specific plugin registrar.

Arrows: Super3d

• Replaced cout and cerr usage in Super3D with vital logger.

• Improved callback handling in compute_depth. Callbacks are now more
  frequent providing better progress updates and quicker early termination.
  However, depth maps updates still occur at the slower rate as before.

Sprokit: Processes

• Added a core sprokit process to shift a stream of detected object
  sets by a specified offset. When a negative offset is specified,
  detected object sets are initially consumed from the stream until
  the offset is satisfied. When a positive offset is specified,
  ...

KWIVER Release 1.4.0

This is a minor release of KWIVER that provides both new functionality and fixes over the previous v1.3.0 release.

Updates since v1.3.0

Vital

• Added constructors taking initializer lists to mesh types for easier
  initialization. * Added a new abstract algorithm, uv_unwrap_mesh, which unwraps
  a mesh and generates normalized texture coordinates.

• Added a foreach_pixel helper function to apply an operation at every pixel of
  a const image.

• Added the data_serializer base algorithm to support serialization of Vital
  types for distributed processing and other uses.

• Applied a new approach to adding metadata to plugins. The plugin name and
  description are now specified in the plugin header. Added new plugin registrar
  class framework to simplify the registration process.

• Promoted version.h from Sprokit to vital for use across all of KWIVER.

• Change the return type of the as_bytes() function on vital::descriptor to
  return a raw pointer to the bytes instead of a vector of bytes. This API change
  avoids many memory copies when comparing descriptors at the byte level.

• Added a sfm_constraints class and pass this to the structure-from-motion
  algorithms (initialize_cameras_landmark, bundle_adjust, optimize_cameras)
  instead of passing the raw metadata.

• Added additional range helpers indirect and valid. The former provides access
  to iterators while still allowing a range-based for to be used. The latter
  filters iteration so that only "valid" items are visited.

• Changes to track_set_impl to facilitate passing track changes between Sprokit
  processes rather than full track sets for efficiency.

• Moved python module loader and registration from Sprokit to vital

• Added metadata tag, VITAL_META_VIDEO_KEY_FRAME, which is a bool that marks if
  a video frame is a key frame.

Vital Util

• Added simple statistic class for accumulating values. Computes average and
  std-deviation

Vital Tools

• Converted tools to use applet plugin approach.

• Converted stand-alone tools to reside under the main "kwiver" top level tool.
  Individual tools are specified as a parameter to the "kwiver" tool runner.

Vital Types

• Added crop method to vital::image to get a cropped view of the image.

• Added version of the get_image method to vital::image_container to get a
  cropped view of the underlying image.

Vital Bindings

• Added python bindings for image_object_detector

Arrows

• Converted arrows to use new plugin registrar implementation.

Arrows: Core

• Created a new plugin for core applets

• Added video_input_image_list function which enables all images contained
  inside a directory to be read in.

• Converted Arrows core to use new PLUGIN_INFO and registrar approach.

• Added an applet to render meshes into depth or height maps

• Added mesh_operations.h, initially with a function to triangulate a mesh.
  That is, subdivide mesh faces into triangles.

• Added an implementation for triangular mesh unwrapping. The method transforms
  each face independently to 2D and packs them in an almost square texture.

• Improved triangulate_landmarks implementation with an option to use RANSAC
  for robustness to outliers and additional checks to reject poorly constrained
  landmarks with minimal view angle variation.

• Switched track_set_impl from using a vector to store track_sptr to using an
  unordered map. This allows for more efficient search for tracks.

• Set inlier flag for track states during triangulation. Stored max
  intersection angle of rays in landmark, a quality metric.

• Extracts ORB descriptors at KLT feature locations. No longer runs ORB
  detector.

• When triangulating switch to using squared re-projection error for
  efficiency. Optionally pass initial guess for landmark position to RANSAC
  triangulation. If location is already known or a good guess is available this
  can cut down the RANSAC loops considerably.

• Optionally estimates a fundamental matrix before loop closure. Accepts a loop
  only if the number of inlier matches is above a threshold.

• Improved the efficiency of loop closing and descriptor comparison.

• Added triangulate_landmarks method that takes a map of tracks rather than a
  track_set_sptr as input. This is done for efficiency. Using the track map we can
  directly access the track corresponding to a landmark that needs to be
  triangulated, rather than looping over all tracks.

• Added an alternative triangulation method based on "Triangulation Made Easy"
  Lindstrom CVPR 2010.

• Added a new video_input implementation, video_input_splice, that splices
  multiple video sources together into a single video source.

• Added depth utilities to compute depth ranges based on landmarks

• Added a convenience class for storing maps of arbitrary cameras and adapted
  SfM utils to use it.

• Added metrics for rms error by camera and a check that the rays meeting at a
  point have some minimum angle.

• Added functions to clip meshes with a plane or a camera frustum.

• Added a temporal down sampling option to video_input_filter where only every
  nth frame of the video source will be provided.

• Added a utility function to compute the average world distance per pixel in a
  provided world region of interest.

• Updated the keyframe loop closure algorithm to store keyframes in the track
  set data structure rather than internally. This change exposes the list of
  selected keyframes outside of this algorithm.

Arrows: Ceres

• Added ability to fix either cameras or landmarks in bundle adjustment.

• Automatically fixes the gauge if there are not enough fixed cameras or
  landmarks.

• Added option to necker reverse only the cameras but not the landmarks.

• Improved speed of klt/ORB feature tracking by choosing which features to
  track and match with visual vocabulary node ID. Made klt feature tracker in
  Sprokit return just the most recent tracked features, not the entire history at
  each step. Made keyframe selector remove non keyframes temporally between
  keyframes to improve speed. That way matcher process needs only get keyframes.
  After Sprokit klt tracks and keyframe orb feature tracks are merged into a
  single final track set.

• Added feature detection that adaptively changes thresholds to detect close to
  a target number of features in each frame. Also improves the distribution of
  features across the image with a gridded approach.

• Added functions that detect issues in the structure from motion solution and
  correct them. Issues could be disconnected components of cameras, cameras with
  too little coverage, landmarks without enough measurements to constrain them
  etc. The corrections typically remove offending items. Only the largest
  connected map component is kept. Under-constrained landmarks are removed etc.

Arrows: CUDA

• Added CUDA arrow

• Added depth map integration using CUDA. The algorithm takes a set of depth
  maps and associated cameras to produce a volume that a surface can be extracted
  from using an algorithm like marching cubes.

• Switch from std::map to std::unordered_map for internal bookkeeping for
  efficiency.

• Updated depth map integration to adjust the voxel size relative to the
  average world distance per pixel. This allows fusion to work on scenes of
  various scales without manually tuning the configuration parameters.

• Added utilities for checking CUDA functions for error codes and then logging
  the error and throwing an exception.

• Added utilities for CUDA memory management with unique_ptr. This is required
  to prevent memory leak when exceptions are thrown.

Arrows: FFmpeg

• Translated log messages from FFMPEG into KWIVER log messages so we can
  control the verbosity.

• Added metadata handling to FFMPEG implementation of video_input

• Improved the seek_frame call to seek to a nearby keyframe first before
  advancing to the precise frame requested. This is much faster than running
  through the entire video until hitting the requested frame.

• Using the new metadata tag, VITAL_META_VIDEO_KEY_FRAME, mark each frame of
  the video whether it is or isn't a key frame.

Arrows: KPF

• Updated the KPF YAML parser to allow for embedded YAML in metadata.

Arrows: Super3d

• Added region of interest specification for depth maps

• Used OpenMP to accelerate various operations with parallel loops.

Arrows: Python

• Added python arrow

• Added simple_image_object_detector.py that serves as a use-case for python
  bindings of image_object_detector and replicates the functionality of
  example_detector in python

Sprokit

• Changed KLT tracking pipeline to pass track change information between
  processes rather than full track sets for efficiency.

Sprokit: Processes

• Added an extension plugin capability to embedded_pipeline that provides a
  hook just before the pipeline is set up and a hook after pipeline has reached
  end of data.

• Converted processes to use the new process registrar approach.

• Improved process timing instrumentation to provide simple stats for step
  entry.

General Documentation

• Re-factored and improved documentation for video_input algo in vital.

• Greatly improved documentation of 'image' and 'image_of' classes in vital.

Bug Fixes

Arrows: Core

• Added gradual reprojection error threshold reduction in metadata based SfM to
  keep solutions from falling apart.

Arrows: CUDA

• In integrate_depth_maps.cu depthMapKernel a test was included for depth
  values equal to -1 which should be excluded. Testing for equality in a float is
  generally bad practice because of machine precision issues. Changed to testing
  if the depth value is less than or equal to zero. If it is, then the depth value
  is skipped. This can be used to mask out bad measurements in the depth ...

KWIVER Release v1.3.1

This is a patch release of KWIVER that provides fixes over the previous v1.3.0 release.

Bug Fixes since v1.3.0

• Fixed issue where ffmpeg_video_input does not set the depth step for
  frame images correctly.

• Fixed an issue in the MSVC environment in which and extra ";" was
  appended to each variable

• Fixed an issue in the MSVC environment in which redefining variables
  from setup_KWIVER.bat would append to the old value instead of replacing.

• Install two CMake find-package modules (FFMPEG and Sphinx) that were
  missing from the installation scripts.

KWIVER Release v1.3.0

KWIVER v1.3.0 Release Notes

This is a minor release of KWIVER that provides both new functionality and fixes over the previous v1.2.0 release.

There has been a batch of updates surrounding serialization. Vital, Arrows, and Sprokit have all received updates to enhance the support of serialization. Vital has new algorithms to support the serialization of different vital types. In Arrows, there are new JSON and protocol buffer serializers for bounding boxes and detected objects. In Sprokit, there are new processes that convert serialized byte strings to Sprokit data ports.

This update also includes a series of organizational changes that reduce conflicts and increase usability.

There are many other changes in this release. These are detailed in the change log below.

Updates since v1.2.0

• Loadable modules (algorithms and processes) have been moved under
  the .../lib/kwiver subdirectory so that they will not conflict with
  other packages.

• Replaces sprokit_private_header_group with kwiver equivalent.

• Remove unused Sprokit CMake support.

Vital

• Added new class to format config block to a stream and deprecated
  print method in config_block class. Config block formatter provides
  methods to produce different format outputs. Config block
  formatting can be extended with plugins.

• Added the data_serializer base algorithm to support serialization
  of Vital types for distributed processing and other uses.

Vital Bindings

• Added python bindings for timestamp class.

Arrows: Core

• Added support for RPC cameras to the triangulate_landmark implementation.
• Added a function to render a depth map of a mesh from a perspective camera.

Arrows: FFmpeg

• Created a new FFmpeg arrow with a video_input algorithm implementation.
  This provides an alternative to the vidl_ffmpeg_video_input in the VXL
  arrow. The goal is to provide direct access to FFMPEG without requiring
  VXL and to improve efficiency, especially with video seeking. Currently
  this implementation is not full featured and does not yet provide a
  complete replacement for the VXL implementation.

Arrows: GDAL

• Created a new GDAL arrow with an image_io algorithm implementation.
  This allows opening images with GDAL, which is especially useful for
  Geospatial image formats like GeoTIFF and NITF.

Arrows: Serialization

• Added JSON Serializers for:

  • bounding_box
  • detected_object (partial)
  • detected_object_set (partial)
  • detected_object_type

• Added Protocol Buffer Serializers for:

  • bounding_box
  • detected_object (partial)
  • detected_object_set (partial)
  • detected_object_type

Sprokit: Processes

• Added serializer_process and deserializer_process that convert from/to
  serialized byte strings and Sprokit data ports.

• Added zmq_transport_send_process and zmq_transport_recieve process to support
  Publish/Subscribe distributed processing with ZeroMQ.

Bug Fixes

• Fixed a problem where ports that were not supplied data in the
  input_adapter_process were incorrectly reported.

• Fix and issue in the feature_detect_extract_SURF arrow. The arrow was using a dynamicCast
  which causes a segmentation fault on the Mac. On the Mac, symbols exported from an image loaded
  with RTLD_GLOBAL mode are not available to other images unless built with -flat_namespace.

• Fixed issue with loading logger plugins on Windows due to a semicolon
  that gets appended to environment variables

KWIVER Release v1.2.0

Release Summary

This is a minor release of KWIVER that provides both new functionality and
fixes over the previous v1.1.1 release. There are also a significant number
of quality of life updates, as well as minor restructuring updates that are
included in this release. The primary changes in this release are to Vital,
Arrows: Core, and Sprokit.

Vital, the core framework of KWIVER, has been updated in several ways in
this release. There is an assortment of new algorithms, functions, and types
added in this release. Most are in support of a basic object tracker, including
readers/writers, matrix-based associate detections to track process, a generic
track initializer, and others. However, there is also a new abstract algorithm
called integrate_depth_maps for the integration of depth map images to implicit
surface volume. This release also includes expansions to the video_input API to
include a seek_frame() and frame_time() methods.

Arrows: Core has received a batch of updates in this release. New features include
a special feature tracker that augments keyframes with additional detections, new
keyframe selection classes, new loop closure algorithm, and more. In addition, there
is general restructuring of track merging and descriptions() from some
implementations had been removed.

Sprokit has also received a batch of updates in this release. Most of the updates
to Sprokit are in the area of managing processes. Process instrumentation calls
have been added to all processes, this allows the ability to monitor the activity
of processes on a per-process basis. There are new Sprokit processes corresponding
to the feature tracker and keyframe selection vital algorithms. As well as an
added detect_feature_if_keyframe process that calls the new augment features tracker and
also merges tracks.

KWIVER can now be built by utilizing Docker. There is a Dockerfile included in
/docker/ for this purpose. Optionally, it can be built by utilizing the dockersetup.sh
script in /docker/. This script automatically builds and configures the provided KWIVER
Docker container.

There are many other changes in this release. These are detailed in the change log below.

Updates since v1.1.1

Vital

• C++11 feature test macros have been removed. The current version of KWIVER
  assumes a fully C++11 compliant compiler; C++11 features should be used
  directly. VITAL_FOREACH (and the header vital_foreach.h which provided
  this), has also been removed. Remaining compatibility shims are deprecated
  and should not be used in new code and will eventually be removed also.
  Note also that [[noreturn]] must appear before _EXPORT; this was not
  the case with the old definition of VITAL_NO_RETURN.

• Added a small range utilities library. This implements a small subset of
  range adapters inspired by the upcoming Ranges TS.

• Added new data types geo_point and geo_polygon for representing points
  and polygons in geographic coordinates. These new types use the new
  geodetic conversion function and know how to convert themselves into
  different coordinate systems. These replace the old geo_UTM, geo_lat_lon
  and geo_corner_points types.

• The geo_map algorithm has been replaced with a new non-algorithm mechanism
  for geodetic conversion. The new approach uses member functions on the new
  geodetic types to request conversions in combination with a conversion
  algorithm that must be provided and registered by a suitable plugin (e.g.
  the PROJ.4 arrow). Registration is automatic when the plugin is loaded,
  with a mechanism in place for users to register their own algorithm if
  desired. The new API is also more general and supports arbitrary datums,
  not just WGS84.

• Generalized the track and track_state types to be useful for both feature
  and object tracks. Data that is specific to features or objects is now
  stored in subclasses of track_state.

• Generalized the track_set to be useful for both feature and object tracks.
  Added derived classes for feature_track_set and object_track_set to
  indicate the type of tracks in the set and provide specialized helper
  functions. Algorithm APIs previously using track_set now use either
  feature_track_set or object_track_set.

• The track_set class now has a separate implementation class named
  track_set_implementation. Using a bridge design pattern this allows
  any combination of implementations and derived track_set types. Custom
  track_set implementations allow alternative storage and query mechanisms
  for better scalability and speed under different use cases.

• The track_set is now a mutable class and has new member functions including
  'insert', 'remove', and 'merge_tracks' to allow in place modification.
  The intent is that most algorithms will now modify a track_set in place
  and return it rather than creating a new track_set from the tracks. This
  is important for a track_set because it is often a very large structure
  and may now have backend implementations with bookkeeping that we would
  like to persist.

• Track states now have a pointer (std::weak_ptr) back to the track object
  which contains them. This means that each track state can belong to at
  most one track, and tracks and track states must now be created and
  referenced only via shared_ptr. Constructors for track are now protected
  and "make()" static member is used for construction. A "clone()" function
  is now used for track copy construction.

• Array descriptors are now iterable (have begin/end methods), allowing them
  to be used in standard algorithms, range-based for, etc.

• Detected object classifications (detected_object_type) are now iterable
  (have begin/end methods), allowing them to be used in standard algorithms,
  range-based for, etc. This provides a much more efficient mechanism for
  visiting all classifications.

• Improved plugin explorer so it is easier to get information about
  processes and algorithms. Typical usage for locating processes would
  be to enter the following command to find which processes have to
  do with reading.

  $ plugin_explorer --proc read

  Then enter the following command when the specific process has been
  identified to get a detailed output about the process.

  $ plugin_explorer --proc image_file_reader -d

• Added macros to access low level datum objects using port traits.

• Select operations on detected_object_set now return the selected
  subset as a new detected_object_set.

• Detected object set is iterable. begin() and end() methods have
  been added in place of returning the vector of detections. This
  allows iteration over arbitrarily large detection sets. Also
  simplifies the select methods that now return a detected_object_set
  rather than a vector of detections.

• Added method to return a vector of values from a single config
  entry. The method is templated on the vector element type.

• Added at() method to detected_object_set to provide bounds checked
  direct indexing.

• Added process properties that are bound to the process
  object. Process properties were created based on the process class
  hierarchy, where the derived process would create the set of
  properties. There are cases where the derived process does not know
  all the properties, such as when process instrumentation is
  enabled. The solution is to add a set of properties that is part of
  the process object that can accumulate these condition based
  properties. Properties from the derived class are merged with the
  fixed set of properties when the set of properties is requested.

• The video_metadata library has been removed by refactoring its contents
  into other parts of Vital. The video_metadata type has moved into
  vital/types and was renamed to "metadata". The file I/O functions have
  moved into vital/io, and the KLV conversion functions have moved into
  vital/klv.

• Refactor timestamp types for time and frame number. Standardized on
  the base kwiver::vital types for frame_id and time. Renamed time_t
  to time_us_t to provide some distance from std::time_t and make the
  units apparent.

• Added frame_time() method to video_input to access the current time stamp.

• Expanded video_input API to include a seek_frame() method which will seek
  the video source to specific frame and a seekable() method which returns
  true if the video source supports frame seeking. This allows
  implementations to support seeking to a specific frame if possible.

• Added a new abstract algorithm, integrate_depth_maps, for the integration
  of depth map images into an implicit surface volume using truncated signed
  distance functions (TSDF) or similar approaches.

• Added clone() functions to the track_set and the class that it contains
  to allow deep copies of track sets.

• Added a track_set_frame_data class to store frame-level data along side
  the tracks. The feature_track_set uses this to mark frames as key frames.

• Added an assortment of types and algorithms in support of a basic object
  tracker, including readers/writers, a matrix-based associate detections to
  tracks process, a generic track initializer, and others.

Vital Bindings

• Vital C and Python bindings have been updated with respect the refactoring
  of tracks and track_sets to account for both feature and object tracks.

Arrows: Core

• Added a new frame-indexed track_set_implementation. When used in place of
  the default track_set_implementation this implementation will cache a mapping
  from frame number to the set of track states on that frame. This makes it
  substantially faster to query tracks by frame ID...