All About Camera Calibration

This repository is based on the implementation of the multi-camera calibration framework Multical with modification in the parameter initialization part to inhance the system's robustness, particularly in situations where camera views do not overlap. It also presents a complete guide for robot motion designed to facilitate automated image capture.

Polygon design

Board pattern design

Useful link (https://docs.opencv.org/4.x/da/d0d/tutorial_camera_calibration_pattern.html)
Package
Run write_board2.py
arguments --rows 9 --columns 12 -T charuco_board --square_size 13 --marker_size 8 -f DICT_6X6_1000.json
In order to change board offet: Inside write_board2.py change marker_id = 100
Create boards.yaml file accordingly

Continuous Image acquisition

Robot motion

box_attacher_3.py is adopted from box_attacher_2.py. The following functions are added that are helpful for image acquisition:
- move_ef_position(self, xPose, yPose, zPose)
- plan_from_pose_json(self, pose_json_path)
- plan_box_param(self, library = 'arv', path = "")
- write_json(self, json_dict, pose, file_name)
main.py contains script for using box_attacher_3.py
sample gripper_poses.json that could be used in plan_from_pose_json() function

Image acquisition libraries

Aravis

Expand

Camera Aravis API

For installation follow this link: https://aravisproject.github.io/aravis/building.html

Steps:

At first, download source code from https://github.com/AravisProject/aravis/releases
Unzip and from terminal navigate to the folder
Download dependencies for Ubuntu 20.04

sudo apt install libxml2-dev libglib2.0-dev cmake libusb-1.0-0-dev gobject-introspection \
             libgtk-3-dev gtk-doc-tools  xsltproc libgstreamer1.0-dev \
             libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev \
             libgirepository1.0-dev gettext

pip3 install --user meson==0.56.0

Install

    meson setup build
    cd build
    ninja
    ninja install
    sudo ldconfig

Alternative way to install API

Follow this method (https://github.com/roboticslab-uc3m/installation-guides/blob/master/install-aravis.md)
Steps:
- Download a newer version from (https://download.gnome.org/sources/aravis/0.4/)
- Uncompress the downloaded Aravis package and open a terminal in that folder.
- Run ./configure
- Run make
- Run sudo make install

Camera Aravis python

Camera Aravis ROS Package

Add this repo (https://github.com/FraunhoferIOSB/camera_aravis) in your catkin workspace
catkin_make

ROS camera aravis application

Run roscore
To check which camera models are connected, run from another terminal arv-tool-0.4. Output:
```
The Imaging Source Europe GmbH-11120229
The Imaging Source Europe GmbH-42120643
```
To run it in a given namespace ROS_NAMESPACE=cam1 rosrun camera_aravis cam_aravis _guid:=GmbH-11120229
From another terminal, run ROS_NAMESPACE=cam2 rosrun camera_aravis cam_aravis _guid:=GmbH-42120643

Custom ROS Package (Implemented)

How to create ROS wrapper?

Create ROS package (http://wiki.ros.org/ROS/Tutorials/CreatingPackage)

cd tx60_moveit/src
catkin_create_pkg camera_automation std_msgs rospy roscpp
catkin_make

Create python node (http://wiki.ros.org/ROS/Tutorials/WritingPublisherSubscriber%28python%29)
catkin_make
chmod +x ros_wrapper.py
Inside launch file

<node pkg="xsens_driver" name="mtnode" type="mtnode.py" output="screen"> 
</node>

Issues

/usr/bin/env: ‘python3\r’: No such file or directory :
- sudo apt install dos2unix
- dos2unix python_file.py

Implemented Camera_automation ROS wrapper

package
Command from console

roscore
rosrun camera_automation cam_node.py camera_serial_number exposure_time path_to_save_image

Example :

rosrun camera_automation cam_node.py 42120643 2000000 /home/raptor/tx60_moveit/src/tx60l_moveit_config/python_program/image/arv_im.png

Necessary Python Script (Implemented)

aravis_image_acquisition.py contains all functions for continuous image acquisition from all the cameras
aravis_show_image.py contains all functions for the PyQT GUI

TIS

Expand

20 MP TIS camera original software

https://www.theimagingsource.de/support/software-f%C3%BCr-linux/
It is installed with "sudo apt install ./tiscamera_0.n.n_amd64_ubuntu_1804.deb". "n" is to be replaced by the corresponding version number.
After installation, start the "tcam-capture" program in the terminal to use a camera.
Code example in https://github.com/TheImagingSource/tiscamera/tree/master/examples

CVB

Overview of good dataset

Camera Calibration Framework

Packages

python 3.10
pip install opencv-contrib-python-headless==4.6.0.66
pip install PyQt5
pip install pandas
pip install multical
pip install plotly
pip install scikit-learn
pip install openpyxl
pip install numpy==1.23.4
pip install matplotlib==3.1.0
pip install opencv-contrib-python==4.6.0.66

Or, one can directly use the virtual environment venv

Usage

Steps:

git clone this_repository
Open this project from any IDE
One can use this program in two ways:
- Manually
- Automatically using GUI (This is still underdevelopment)

For Manual operation:

Prepare dataset folders as following:

- dataset
  - cam1
  - cam2
  .
  .
  .
  - boards.yaml

Run main_handEye.py. It will create following files:

- intrinsic_dataset.json : Views that are used for intrinsic calibration (needed for visualization GUI)
- handEyeCamera.json : Information of all HandEye-pair groups (needed for visualization GUI)
- initial_calibration_cam1.json : Initial calibration result, keeping cam1 as master camera
- initial_calibration_cam2.json : Initial calibration result, keeping cam2 as master camera
- meanCameras.json : Information of all extrinsic parameters keeping each camera as master camera (needed for visualization GUI)
- workspace.pkl : workspace information before Bundle Adjustment

Prepare test dataset folder taking a subset of 20 poses for each camera. The folder should look like as following:
```
- dataset_test
  - cam1
  - cam2
  .
  .
  .
  - boards.yaml
  - initial_calibration_cam1.json
```
Here, cam1 is master camera

Run main_calibrate.py. It will generate results and some other files for visualization:

- calibration_cam1.txt : Calibration result
- calibration_08320217.json : Final intrinsic and Extrinsic Parameters
- calibration_08320217.pkl : Final calibrated workspace (needed for visualization GUI)

User Interface

It has two parts:

Automated camera calibration
Vision guided robot movement

Automated Camera Calibration

This GUI automates the whole calibration process also shows results of the calibration. One can also investigate the dataset quality using this. The whole process of operating this is explained here.

Vision Guided Robot Movement

This feature allows the user to move the viewed feature with respect to the camera. Explanation is here

Multical++

For this work, I edited some parts of the original implemetation of Multical

Edited package
Following are some important scripts

HandEye Initialization

Bugs

GUI "Show Final Camera Poses" visualization issue

Paraview guide

https://www.youtube.com/watch?v=FTUBpqkC3Ss
To add python shell in paraview application: view > Python shell
Python program can be run from command prompt:
- To enable this, add the path of pvpython.exe 'Program Files/ParaView 5.11.0/bin' in environment variable
- From cmd, run pvpython.exe
- for loading imagefile reader = OpenDataFile("D:/path_to_the_folder/group01_img1.png")
- Or write a python script and run from cmd pvpython.exe D:/path_to_python_script/parav3.py

Issues

OpenCV Issues

For aruco, PyQT related issues
- pip3 uninstall opencv-python
- python3 -m pip install opencv-contrib-python
- pip install opencv-contrib-python-headless==4.6.0.66
cv2.solvePnPGeneric sometimes show irrelevent results
- take 3/5 pictures of the same poses. Calculate pose. Take average of the poses
- cv2.solvePnP and cv2.solvePnPGeneric both are taking solvePnP_Iterative method as default
openCV >= 4.6 (Python>= 3.6):
- solvePnPGeneric/ solvePnP working
- PyQt5 not working
openCV = 4.2:
- solvePnPGeneric not working
- solvePnP working
- PyQt5 working

Hand-Eye Calibration Issues

cv2.calibrateRobotWorldHandEye(cam_world_R, cam_world_t, base_gripper_R, base_gripper_t, method=cv2.CALIB_ROBOT_WORLD_HAND_EYE_SHAH)
cv2.error: OpenCV(4.6.0) /work/ci_py311/opencv-suite_1676837327081/work/modules/calib3d/src/calibration_handeye.cpp:524: error: (-7:Iterations do not converge) Rotation normalization issue: determinant(R) is null in function 'normalizeRotation'

Solution: Remove the outlier image

scipy issues

Error from stats.gaussian_kde(values)

numpy.linalg.LinAlgError: 3-th leading minor of the array is not positive definite

Solution: values should have more than 3 groups

numpy issues

AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:

Solution: numpy.int was deprecated in NumPy 1.20 and was removed in NumPy 1.24. Downgrade numpy. pip install numpy==1.23.4

PyQt5 Issues

Backend QtAgg is interactive backend. Turning interactive mode on. qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.  Available platform plugins are: eglfs, minimal, vkkhrdisplay, xcb, vnc, wayland, minimalegl, wayland-egl, linuxfb, offscreen.

Same issue for Dockerfile

Solution (might not work):

Problem might come from PyQt5 of Python>=3.6. or matplotlib. Downgrade matplotlib pip install matplotlib==3.1.0
try (NVlabs/instant-ngp#300)
Uninstall and install opencv couple of times. Sometimes it works.

pip install opencv-contrib-python-headless==4.6.0.66
pip uninstall PyQt5
sudo apt install python3-qtpy

Dockerfile commands:

docker build -t cam-calib-docker .
Docker run cam-calib-docker

Credits

This work is conducted at the Institute for Factory Automation and Production Systems(FAPS), FAU-Erlangen, Germany.
The Robot Motion part is based on the work from Oguz Kedilioglu
This work is based on the work of Multical
Much Inspiration is taken from CALICO and MC-Calib
Some visualization inspiration is taken from rc-visard

Name		Name	Last commit message	Last commit date
Latest commit History 431 Commits
.idea		.idea
Camera_Board_Workspace		Camera_Board_Workspace
GUI		GUI
Images		Images
Mc_Calib		Mc_Calib
Polygon drawings		Polygon drawings
Sample_files		Sample_files
Tutorials		Tutorials
camera_automation		camera_automation
ceres_solver_windows		ceres_solver_windows
multical-master		multical-master
test_files		test_files
tx60l_moveit_config		tx60l_moveit_config
venv		venv
LICENSE		LICENSE
README.md		README.md

License

TabassumNova/Multi_Camera_Calibration

Folders and files

Latest commit

History

Repository files navigation

All About Camera Calibration

Table of Contents

Polygon design

Board pattern design

Continuous Image acquisition

Robot motion

Image acquisition libraries

Aravis

Camera Aravis API

Alternative way to install API

Camera Aravis python

Camera Aravis ROS Package

ROS camera aravis application

Custom ROS Package (Implemented)

How to create ROS wrapper?

Issues

Implemented Camera_automation ROS wrapper

Necessary Python Script (Implemented)

TIS

20 MP TIS camera original software

CVB

Overview of good dataset

Camera Calibration Framework

Packages

Usage

User Interface

Automated Camera Calibration

Vision Guided Robot Movement

Multical++

HandEye Initialization

Bugs

Paraview guide

Issues

OpenCV Issues

Hand-Eye Calibration Issues

scipy issues

numpy issues

PyQt5 Issues

Credits

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Languages

Packages