Skip to content

Latest commit

 

History

History
356 lines (305 loc) · 15.7 KB

README.md

File metadata and controls

356 lines (305 loc) · 15.7 KB

All About Camera Calibration

This repository is based on the implementation of the multi-camera calibration framework Multical with modification in the parameter initialization part to inhance the system's robustness, particularly in situations where camera views do not overlap. It also presents a complete guide for robot motion designed to facilitate automated image capture.

Table of Contents

Polygon design

Board pattern design

Continuous Image acquisition

Robot motion

  • box_attacher_3.py is adopted from box_attacher_2.py. The following functions are added that are helpful for image acquisition:

    • move_ef_position(self, xPose, yPose, zPose)
    • plan_from_pose_json(self, pose_json_path)
    • plan_box_param(self, library = 'arv', path = "")
    • write_json(self, json_dict, pose, file_name)
  • main.py contains script for using box_attacher_3.py

  • sample gripper_poses.json that could be used in plan_from_pose_json() function

Image acquisition libraries

Aravis

Expand

Camera Aravis API

  • For installation follow this link: https://aravisproject.github.io/aravis/building.html
  • Steps:
    sudo apt install libxml2-dev libglib2.0-dev cmake libusb-1.0-0-dev gobject-introspection \
                 libgtk-3-dev gtk-doc-tools  xsltproc libgstreamer1.0-dev \
                 libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev \
                 libgirepository1.0-dev gettext
    
    pip3 install --user meson==0.56.0
    
    • Install
    meson setup build
    cd build
    ninja
    ninja install
    sudo ldconfig

Alternative way to install API

Camera Aravis python

Camera Aravis ROS Package

ROS camera aravis application

  • Run roscore
  • To check which camera models are connected, run from another terminal arv-tool-0.4. Output:
    The Imaging Source Europe GmbH-11120229
    The Imaging Source Europe GmbH-42120643
    
  • To run it in a given namespace ROS_NAMESPACE=cam1 rosrun camera_aravis cam_aravis _guid:=GmbH-11120229
  • From another terminal, run ROS_NAMESPACE=cam2 rosrun camera_aravis cam_aravis _guid:=GmbH-42120643

Custom ROS Package (Implemented)

How to create ROS wrapper?
cd tx60_moveit/src
catkin_create_pkg camera_automation std_msgs rospy roscpp
catkin_make
<node pkg="xsens_driver" name="mtnode" type="mtnode.py" output="screen"> 
</node>
Issues
  • /usr/bin/env: ‘python3\r’: No such file or directory :
    • sudo apt install dos2unix
    • dos2unix python_file.py
Implemented Camera_automation ROS wrapper
roscore
rosrun camera_automation cam_node.py camera_serial_number exposure_time path_to_save_image

Example :

rosrun camera_automation cam_node.py 42120643 2000000 /home/raptor/tx60_moveit/src/tx60l_moveit_config/python_program/image/arv_im.png

Necessary Python Script (Implemented)

TIS

Expand

20 MP TIS camera original software

CVB

Overview of good dataset

Camera Calibration Framework

Packages

  • python 3.10
  • pip install opencv-contrib-python-headless==4.6.0.66
  • pip install PyQt5
  • pip install pandas
  • pip install multical
  • pip install plotly
  • pip install scikit-learn
  • pip install openpyxl
  • pip install numpy==1.23.4
  • pip install matplotlib==3.1.0
  • pip install opencv-contrib-python==4.6.0.66

Or, one can directly use the virtual environment venv

Usage

Steps:

  • git clone this_repository
  • Open this project from any IDE
  • One can use this program in two ways:
    • Manually
    • Automatically using GUI (This is still underdevelopment)
  • For Manual operation:
    • Prepare dataset folders as following:
      - dataset
        - cam1
        - cam2
        .
        .
        .
        - boards.yaml
      
    • Run main_handEye.py. It will create following files:
      - intrinsic_dataset.json : Views that are used for intrinsic calibration (needed for visualization GUI)
      - handEyeCamera.json : Information of all HandEye-pair groups (needed for visualization GUI)
      - initial_calibration_cam1.json : Initial calibration result, keeping cam1 as master camera
      - initial_calibration_cam2.json : Initial calibration result, keeping cam2 as master camera
      - meanCameras.json : Information of all extrinsic parameters keeping each camera as master camera (needed for visualization GUI)
      - workspace.pkl : workspace information before Bundle Adjustment
      
    • Prepare test dataset folder taking a subset of 20 poses for each camera. The folder should look like as following:
      - dataset_test
        - cam1
        - cam2
        .
        .
        .
        - boards.yaml
        - initial_calibration_cam1.json
      
      Here, cam1 is master camera
    • Run main_calibrate.py. It will generate results and some other files for visualization:
      - calibration_cam1.txt : Calibration result
      - calibration_08320217.json : Final intrinsic and Extrinsic Parameters
      - calibration_08320217.pkl : Final calibrated workspace (needed for visualization GUI)
      

User Interface

It has two parts:

  1. Automated camera calibration
  2. Vision guided robot movement

Automated Camera Calibration

This GUI automates the whole calibration process also shows results of the calibration. One can also investigate the dataset quality using this. The whole process of operating this is explained here.

Vision Guided Robot Movement

This feature allows the user to move the viewed feature with respect to the camera. Explanation is here

Multical++

For this work, I edited some parts of the original implemetation of Multical

HandEye Initialization

Bugs

  • GUI "Show Final Camera Poses" visualization issue

Paraview guide

  • https://www.youtube.com/watch?v=FTUBpqkC3Ss
  • To add python shell in paraview application: view > Python shell
  • Python program can be run from command prompt:
    • To enable this, add the path of pvpython.exe 'Program Files/ParaView 5.11.0/bin' in environment variable
    • From cmd, run pvpython.exe
    • for loading imagefile reader = OpenDataFile("D:/path_to_the_folder/group01_img1.png")
    • Or write a python script and run from cmd pvpython.exe D:/path_to_python_script/parav3.py

Issues

OpenCV Issues

  • For aruco, PyQT related issues

    • pip3 uninstall opencv-python
    • python3 -m pip install opencv-contrib-python
    • pip install opencv-contrib-python-headless==4.6.0.66
  • cv2.solvePnPGeneric sometimes show irrelevent results

    • take 3/5 pictures of the same poses. Calculate pose. Take average of the poses
    • cv2.solvePnP and cv2.solvePnPGeneric both are taking solvePnP_Iterative method as default
  • openCV >= 4.6 (Python>= 3.6):

    • solvePnPGeneric/ solvePnP working
    • PyQt5 not working
  • openCV = 4.2:

    • solvePnPGeneric not working
    • solvePnP working
    • PyQt5 working

Hand-Eye Calibration Issues

cv2.calibrateRobotWorldHandEye(cam_world_R, cam_world_t, base_gripper_R, base_gripper_t, method=cv2.CALIB_ROBOT_WORLD_HAND_EYE_SHAH)
cv2.error: OpenCV(4.6.0) /work/ci_py311/opencv-suite_1676837327081/work/modules/calib3d/src/calibration_handeye.cpp:524: error: (-7:Iterations do not converge) Rotation normalization issue: determinant(R) is null in function 'normalizeRotation'

Solution: Remove the outlier image

scipy issues

  • Error from stats.gaussian_kde(values)
    numpy.linalg.LinAlgError: 3-th leading minor of the array is not positive definite
    
    Solution: values should have more than 3 groups

numpy issues

AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:

Solution: numpy.int was deprecated in NumPy 1.20 and was removed in NumPy 1.24. Downgrade numpy. pip install numpy==1.23.4

PyQt5 Issues

Backend QtAgg is interactive backend. Turning interactive mode on. qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.  Available platform plugins are: eglfs, minimal, vkkhrdisplay, xcb, vnc, wayland, minimalegl, wayland-egl, linuxfb, offscreen.

Solution (might not work):

  • Problem might come from PyQt5 of Python>=3.6. or matplotlib. Downgrade matplotlib pip install matplotlib==3.1.0
  • try (NVlabs/instant-ngp#300)
  • Uninstall and install opencv couple of times. Sometimes it works.
  • pip install opencv-contrib-python-headless==4.6.0.66
    pip uninstall PyQt5
    sudo apt install python3-qtpy
    

Dockerfile commands:

docker build -t cam-calib-docker .
Docker run cam-calib-docker

Credits

  • This work is conducted at the Institute for Factory Automation and Production Systems(FAPS), FAU-Erlangen, Germany.
  • The Robot Motion part is based on the work from Oguz Kedilioglu
  • This work is based on the work of Multical
  • Much Inspiration is taken from CALICO and MC-Calib
  • Some visualization inspiration is taken from rc-visard