Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Egu #58

Merged
merged 66 commits into from
Apr 13, 2024
Merged

Egu #58

Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
8b12f70
fixed bug with MatcherBase with no tile preselection active
franioli Mar 25, 2024
63e847d
updated reccomended roma config
franioli Mar 25, 2024
a179da6
updated config.py
franioli Mar 25, 2024
9c12be1
updated roma to last version
franioli Mar 26, 2024
89e75f4
testing new tile preselection based on roma
franioli Mar 26, 2024
662fe92
remove unused reconstruction.main function
franioli Mar 28, 2024
6cb3533
updated for roma and lightglue
franioli Mar 28, 2024
aea581d
added geom verification test
franioli Mar 28, 2024
133fabf
replaced _config with config in all classes
franioli Mar 28, 2024
04799fb
updated config managment in base matcher classes
franioli Mar 28, 2024
0d01b62
updated default parameters in roma and exposed to yaml config
franioli Mar 28, 2024
ad32941
cleaned reconstruction.py
franioli Mar 28, 2024
143e28c
updated reconstruction.py
franioli Mar 28, 2024
dd9a87c
cleaned up reconstruction
franioli Mar 28, 2024
31d7d16
removed triangulation from DIM
franioli Mar 28, 2024
1b51e6e
updated roma
franioli Mar 28, 2024
c676a44
updated database
franioli Mar 29, 2024
ffcf92c
cleaning dependancies
franioli Mar 29, 2024
10bed40
updated import in __init__
franioli Mar 29, 2024
be29080
update pyproject.toml
franioli Mar 29, 2024
b20a990
removed pandas dependency
franioli Mar 29, 2024
2e277b7
restructured all imports to be all inside the package
franioli Mar 29, 2024
5dffba2
updated ImageMatcher, ExtractorBase and MatcherBase, classes. Some te…
franioli Mar 29, 2024
618366f
fixed bug in aliked
franioli Mar 29, 2024
8e996f1
updated _default_config in all extractors and matchers
franioli Mar 29, 2024
0f5f65d
added context manager to COLMAPDatabase
franioli Mar 30, 2024
396ccaf
added new triangulation module
franioli Mar 30, 2024
f5b68ed
added utils.py
franioli Mar 30, 2024
63b4310
updated constants and imports
franioli Mar 30, 2024
fc1123a
added notebook for dense triangulation with known camera poses
franioli Mar 30, 2024
1567ddc
updated triangulation notebook
franioli Mar 30, 2024
7433e77
updated triangulation module
franioli Mar 30, 2024
6cea00d
updated imports to avoid crashes with no pycolmap
franioli Mar 30, 2024
265cbc8
updated github actions for installing pytest
franioli Mar 30, 2024
597a942
moved graph template dir to utils
franioli Mar 30, 2024
7d8af52
moved hloc to thirdparty
franioli Mar 30, 2024
d8e2e78
updated scripts
franioli Apr 8, 2024
d9e6fd4
bump version 1.0.0 -> 1.1.0
franioli Apr 8, 2024
55511a1
updated pyproject.toml
franioli Apr 8, 2024
de8da33
bump version 1.1.0 -> 1.1.1
franioli Apr 8, 2024
54b297a
updated readme
franioli Apr 9, 2024
d15d2e4
belv notebook
franioli Apr 9, 2024
e60142d
updated __init__
franioli Apr 9, 2024
f4473c8
Merge branch 'imports' into egu
franioli Apr 9, 2024
00bb947
imporved roma lowres preselection
franioli Apr 10, 2024
9e9211e
various small updates
franioli Apr 10, 2024
eb25063
Merge branch 'dev' of github.com:3DOM-FBK/deep-image-matching into dev
franioli Apr 10, 2024
468d677
Merge branch 'dev' into egu
franioli Apr 10, 2024
cae0c07
fixed geoemtric verification with NONE parameter
franioli Apr 10, 2024
bb94131
updated triangulation from known poses
franioli Apr 10, 2024
3adec4e
added notebooks
franioli Apr 10, 2024
c227edd
added option to select lightglue or roma preselection
franioli Apr 11, 2024
c603719
updated notebook
franioli Apr 11, 2024
4c78a5f
small updates in roma
franioli Apr 11, 2024
4d6f50d
updated notebooks
franioli Apr 11, 2024
9c44ce4
updates export_to_colmap to read cameras.yaml
franioli Apr 12, 2024
b8970ea
updated notebooks
franioli Apr 13, 2024
3ef8dae
updated export to openmvg and export to db to read camera option file
franioli Apr 13, 2024
0718db0
updated all tests and fixed bug in config reader
franioli Apr 13, 2024
c2c260e
fixed tests
franioli Apr 13, 2024
d62db1c
removed xformers from depandancy due to installation problem on macos
franioli Apr 13, 2024
afb91df
bump version 1.1.1 -> 1.2.0
franioli Apr 13, 2024
ea1c37b
added action to publish on pypi
franioli Apr 13, 2024
d43e50b
updated action to publish to pypi
franioli Apr 13, 2024
e6a98f4
bump version 1.2.0 -> 1.2.1
franioli Apr 13, 2024
d6d0e71
triangulation and reconstruction module imported only if pycolmap is …
franioli Apr 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added new triangulation module
franioli committed Mar 30, 2024
commit 396ccaf455b70b2d286c896cc1c93f32f2210a56
1 change: 1 addition & 0 deletions src/deep_image_matching/__init__.py
Original file line number Diff line number Diff line change
@@ -4,6 +4,7 @@
from . import io
from . import utils
from . import reconstruction
from . import triangulation
from . import extractors
from . import matchers

71 changes: 15 additions & 56 deletions src/deep_image_matching/hloc/triangulation.py
Original file line number Diff line number Diff line change
@@ -5,16 +5,13 @@
from pathlib import Path
from typing import Any, Dict, List, Optional

import h5py
import numpy as np
import pycolmap
from tqdm import tqdm

from . import logger
from .utils.database import COLMAPDatabase, image_ids_to_pair_id
from .utils.geometry import compute_epipolar_errors
from .utils.io import get_keypoints, get_matches
from .utils.parsers import parse_retrieval
from .utils import compute_epipolar_errors, get_keypoints, get_matches, parse_retrieval
from .utils.database import COLMAPDatabase


class OutputCapture:
@@ -44,7 +41,7 @@ def create_db_from_model(reconstruction: pycolmap.Reconstruction, database_path:

for i, camera in reconstruction.cameras.items():
db.add_camera(
camera.model_id,
camera.model.value,
camera.width,
camera.height,
camera.params,
@@ -76,8 +73,8 @@ def import_features(image_ids: Dict[str, int], database_path: Path, features_pat
def import_matches(
image_ids: Dict[str, int],
database_path: Path,
matches_path: Path,
pairs_path: Path,
matches_path: Path,
min_match_score: Optional[float] = None,
skip_geometric_verification: bool = False,
):
@@ -106,53 +103,15 @@ def import_matches(
db.close()


def import_matches2(
image_ids: Dict[str, int],
database_path: Path,
matches_path: Path,
skip_geometric_verification: bool = False,
):
logger.info("Importing matches into the database...")

db = COLMAPDatabase.connect(database_path)
match_file = h5py.File(str(matches_path), "r")

added = set()
n_keys = len(match_file.keys())
n_total = (n_keys * (n_keys - 1)) // 2

with tqdm(total=n_total) as pbar:
for key_1 in match_file.keys():
group = match_file[key_1]
for key_2 in group.keys():
id_1 = image_ids[key_1]
id_2 = image_ids[key_2]

pair_id = image_ids_to_pair_id(id_1, id_2)
if pair_id in added:
logger.warning(f"Pair {pair_id} ({id_1}, {id_2}) already added!")
continue

matches = group[key_2][()]
db.add_matches(id_1, id_2, matches)

if skip_geometric_verification:
db.add_two_view_geometry(id_1, id_2, matches)

added.add(pair_id)

pbar.update(1)

db.commit()
db.close()
match_file.close()


def estimation_and_geometric_verification(database_path: Path, pairs_path: Path, verbose: bool = False):
logger.info("Performing geometric verification of the matches...")
with OutputCapture(verbose):
with pycolmap.ostream():
pycolmap.verify_matches(database_path, pairs_path, max_num_trials=20000, min_inlier_ratio=0.1)
pycolmap.verify_matches(
database_path,
pairs_path,
options=dict(ransac=dict(max_num_trials=20000, min_inlier_ratio=0.1)),
)


def geometric_verification(
@@ -178,7 +137,7 @@ def geometric_verification(
kps0, noise0 = get_keypoints(features_path, name0, return_uncertainty=True)
noise0 = 1.0 if noise0 is None else noise0
if len(kps0) > 0:
kps0 = np.stack(cam0.image_to_world(kps0))
kps0 = np.stack(cam0.cam_from_img(kps0))
else:
kps0 = np.zeros((0, 2))

@@ -189,7 +148,7 @@ def geometric_verification(
kps1, noise1 = get_keypoints(features_path, name1, return_uncertainty=True)
noise1 = 1.0 if noise1 is None else noise1
if len(kps1) > 0:
kps1 = np.stack(cam1.image_to_world(kps1))
kps1 = np.stack(cam1.cam_from_img(kps1))
else:
kps1 = np.zeros((0, 2))

@@ -203,11 +162,11 @@ def geometric_verification(
db.add_two_view_geometry(id0, id1, matches)
continue

qvec_01, tvec_01 = pycolmap.relative_pose(image0.qvec, image0.tvec, image1.qvec, image1.tvec)
_, errors0, errors1 = compute_epipolar_errors(qvec_01, tvec_01, kps0[matches[:, 0]], kps1[matches[:, 1]])
cam1_from_cam0 = image1.cam_from_world * image0.cam_from_world.inverse()
errors0, errors1 = compute_epipolar_errors(cam1_from_cam0, kps0[matches[:, 0]], kps1[matches[:, 1]])
valid_matches = np.logical_and(
errors0 <= max_error * noise0 / cam0.mean_focal_length(),
errors1 <= max_error * noise1 / cam1.mean_focal_length(),
errors0 <= cam0.cam_from_img_threshold(noise0 * max_error),
errors1 <= cam1.cam_from_img_threshold(noise1 * max_error),
)
# TODO: We could also add E to the database, but we need
# to reverse the transformations if id0 > id1 in utils/database.py.
170 changes: 170 additions & 0 deletions src/deep_image_matching/triangulation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
import logging
from collections import defaultdict
from pathlib import Path
from typing import Dict

import h5py
import numpy as np
import pycolmap
from tqdm import tqdm

from deep_image_matching.utils import (
COLMAPDatabase,
compute_epipolar_errors,
get_pairs_from_file,
)

logger = logging.getLogger("dim")


# Utils functions
def parse_retrieval(path):
retrieval = defaultdict(list)
with open(path, "r") as f:
for p in f.read().rstrip("\n").split("\n"):
if len(p) == 0:
continue
q, r = p.split()
retrieval[q].append(r)
return dict(retrieval)


def create_db_from_model(reconstruction: pycolmap.Reconstruction, database_path: Path) -> Dict[str, int]:
if database_path.exists():
logger.warning("The database already exists, deleting it.")
database_path.unlink()

with COLMAPDatabase.connect(database_path) as db:
db.create_tables()

for i, camera in reconstruction.cameras.items():
db.add_camera(
camera.model.value,
camera.width,
camera.height,
camera.params,
camera_id=i,
prior_focal_length=True,
)

for i, image in reconstruction.images.items():
db.add_image(image.name, image.camera_id, image_id=i)

db.commit()

return {image.name: i for i, image in reconstruction.images.items()}


def get_keypoints(features_h5: Path, name: str) -> np.ndarray:
with h5py.File(str(features_h5), "r", libver="latest") as f:
if name not in f:
raise KeyError(f"Key '{name}' not found in '{features_h5}'")
return f[name]["keypoints"][:]


def get_matches(matches_h5: Path, name0: str, name1) -> np.ndarray:
with h5py.File(str(matches_h5), "r", libver="latest") as f:
if name0 not in f:
if name1 in f:
name0, name1 = name1, name0
else:
raise KeyError(f"Key '{name0}' and '{name1}' not found in '{matches_h5}'")
return f[name0][name1][:]


def import_keypoints(features_h5: Path, image_ids: Dict[str, int], database_path: Path) -> None:
with COLMAPDatabase.connect(database_path) as db:
for name, image_id in tqdm(image_ids.items(), desc="Importing keypoints"):
keypoints = get_keypoints(features_h5, name)
keypoints += 0.5 # COLMAP origin
db.add_keypoints(image_id, keypoints)
db.commit()


def import_matches(
matches_h5: Path,
image_ids: Dict[str, int],
database_path: Path,
pair_file: Path,
add_two_view_geometry: bool = False,
):
pairs = get_pairs_from_file(pair_file)
with COLMAPDatabase.connect(database_path) as db:
for name0, name1 in tqdm(pairs, desc="Importing matches"):
matches = get_matches(matches_h5, name0=name0, name1=name1)
id0, id1 = image_ids[name0], image_ids[name1]
db.add_matches(id0, id1, matches)
if add_two_view_geometry:
db.add_two_view_geometry(id0, id1, matches)
db.commit()


def import_verifed_matches(
image_ids: Dict[str, int],
reference: pycolmap.Reconstruction,
database_path: Path,
features_path: Path,
matches_path: Path,
pairs_path: Path,
max_error: float = 4.0,
):
logger.info("Performing geometric verification of the matches...")

pairs = parse_retrieval(pairs_path)

db = COLMAPDatabase.connect(database_path)

inlier_ratios = []
matched = set()
for name0 in tqdm(pairs, desc="Importing verified matches"):
id0 = image_ids[name0]
image0 = reference.images[id0]
cam0 = reference.cameras[image0.camera_id]
kps0 = get_keypoints(features_path, name0)
noise0 = 1.0
if len(kps0) > 0:
kps0 = np.stack(cam0.cam_from_img(kps0))
else:
kps0 = np.zeros((0, 2))

for name1 in pairs[name0]:
id1 = image_ids[name1]
image1 = reference.images[id1]
cam1 = reference.cameras[image1.camera_id]
kps1 = get_keypoints(features_path, name1)
noise1 = 1.0
if len(kps1) > 0:
kps1 = np.stack(cam1.cam_from_img(kps1))
else:
kps1 = np.zeros((0, 2))

matches = get_matches(matches_path, name0, name1)

if len({(id0, id1), (id1, id0)} & matched) > 0:
continue
matched |= {(id0, id1), (id1, id0)}

if matches.shape[0] == 0:
db.add_two_view_geometry(id0, id1, matches)
continue

cam1_from_cam0 = image1.cam_from_world * image0.cam_from_world.inverse()
errors0, errors1 = compute_epipolar_errors(cam1_from_cam0, kps0[matches[:, 0]], kps1[matches[:, 1]])
valid_matches = np.logical_and(
errors0 <= cam0.cam_from_img_threshold(noise0 * max_error),
errors1 <= cam1.cam_from_img_threshold(noise1 * max_error),
)
# TODO: We could also add E to the database, but we need
# to reverse the transformations if id0 > id1 in utils/database.py.
db.add_two_view_geometry(id0, id1, matches[valid_matches, :])
inlier_ratios.append(np.mean(valid_matches))
logger.info(
"mean/med/min/max valid matches %.2f/%.2f/%.2f/%.2f%%.",
np.mean(inlier_ratios) * 100,
np.median(inlier_ratios) * 100,
np.min(inlier_ratios) * 100,
np.max(inlier_ratios) * 100,
)

db.commit()
db.close()