+
PRO TIP:
-If you're unsure whether you've moved your .conda directory from home to the project, run ls -lha in your home directory to see the actual locations of all files, including the (eventually) soft-linked .conda. +If you're unsure whether you've moved your .conda directory from home to the project, run ls -lha in your home directory to see the actual locations of all files, including the (eventually) soft-linked .conda.
+
@@ -317,6 +324,7 @@ You can establish a SOFTWARE or TOOLS directory within your `/project/-If you're unsure whether you've moved your .conda directory from home to the project, run ls -lha in your home directory to see the actual locations of all files, including the (eventually) soft-linked .conda. +If you're unsure whether you've moved your .conda directory from home to the project, run ls -lha in your home directory to see the actual locations of all files, including the (eventually) soft-linked .conda.
+
*We will use this location later in this tutorial to* `git clone` *a few GitHub repositories with python utilities useful in land surveying tasks. You can also add your customized software here.*
B. On your local machine (alternatively):
@@ -354,11 +362,12 @@ Once you've set up the `geospatial` environment, theoretically all the necessary cd /project/
+  3. check available **conda** modules and load selected one *(if not loaded yet in this session)*: ``` @@ -371,8 +380,9 @@ Once you've set up the `geospatial` environment, theoretically all the necessary source activate geospatial ``` - 5. install required libraries *(optionally after* `git pull`*)*: + 5. \*install required libraries *(optionally after* `git pull`*)*: ``` + cd geo_utils pip install -r requirements.txt ``` @@ -383,6 +393,7 @@ Once you've set up the `geospatial` environment, theoretically all the necessary # GCP to ArUco mapper python gcp_to_aruco_mapper.py [-h] -g GCP_FILE -i IMAGERY_PATH -z ZONE [-o OUTPUT] [-d MAX_DIST] ``` + 
PRO TIP:
@@ -409,10 +420,11 @@ Once you've set up the `geospatial` environment, theoretically all the necessary
cd /project//SOFTWARE
```
- 2. clone the geo_utils repo from GitHub:
+ 2. clone the `Find-GCP` repo from GitHub:
```
git clone https://github.com/zsiki/Find-GCP.git
```
+ 
3. check available **conda** modules and load selected one *(if not loaded yet in this session)*:
```
@@ -425,7 +437,7 @@ Once you've set up the `geospatial` environment, theoretically all the necessary
source activate geospatial
```
- 5. install required libraries *(optionally after* `git pull`*)*:
+ 5. \*install required libraries *(optionally after* `git pull`*)*:
```
pip install opencv-python opencv-contrib-python PIL pil.imagetk numpy matplotlib
```
@@ -441,8 +453,10 @@ Once you navigate into the newly created Find-GCP directory, you should see 6 fi
```
cd Find-GCP
ls
+python gcp_find.py -h
```
-
+
+ ## **Automatic generation of ArUco codes** @@ -476,14 +490,15 @@ In the naming convention like `DICT_4X4_100` or `DICT_6X6_250`:
To produce ArUco markers for your land surveying project, start by installing the Find-GCP Python utility ([refer to the section above](#find-gcp-python-utility-installation)). Within the cloned Find-GCP repository directory, you'll locate the `aruco_make.py`. This tool assists you in generating markers from standard dictionaries, as well as more compact 3x3 square markers. * While in the **Find-GCP** directory, use `pwd` command to print the path on the screen. *You will need this path to run python scripts from another location in the file system.* - + * Navigate to the selected location in the file system and create the **markers** directory: ``` +cd /project///geospatial
mkdir markers
cd markers
```
-
+
* Then use the `aruco_make.py` script like this:
`python aruco_make.py -d -s -e -v`, for example:
@@ -615,15 +630,42 @@ This approach is for those possessing a GCP file with recognized ArUco IDs:
* ***This output file is ready for immediate use with OpenDroneMap (ODM) software.***
1. Login to the Atlas cluster using SSH protocol (command line) or OOD access (web-based).
-2. Navigate to your project working directory. *Use the command below:*
+2. Navigate to your ODM working directory. *Use the command below:*
+```
+cd /project//ODM
+```
+
+PRO TIP: If you haven't set up the ODM directory structure yet, please follow the guide provided in section [Create File Structure](https://geospatial.101workbook.org/IntroPhotogrammetry/OpenDroneMap/02-ODM-modules#create-file-structure) in the tutorial [Command-line ODM modules](https://geospatial.101workbook.org/IntroPhotogrammetry/OpenDroneMap/02-ODM-modules).
+Create a subdirectory for your new project in the IMAGES directory and create soft links for your imagery and (eventually) the `GCP_reference.txt` file: +``` +cd IMAGES +mkdir project-X +cd project-X +ln -s/* ./
+ls | head -10
+pwd # copy this path in the next step as the INPUTS_PATH variable
```
-cd /project/
+
+If your GCP reference file (here: gcp_epsg32611_2021_wbs1_coresite.csv) has format different than space-separated 4 columns: aruco_ID X Y Z, then you should adjust it accordingly to get something like this: ``` +0 523287.368 4779588.335 1397.823 +1 523305.976 4779572.588 1397.817 +2 523347.074 4779571.424 1397.653 +3 523364.648 4779587.932 1395.735 +4 523394.376 4779529.525 1398.728 +5 523363.938 4779530.027 1400.244 +``` +You can use `awk` *command to easily extract the columns you need.* Note that GCP_reference.txt file should not have a header. + + 3. Set paths as temporary variables or use them directly: ``` FIND_GCP_PATH=/path/to/Find-GCP_repo INPUTS_PATH=/path/to/input_imagery ``` +In my case the path variables look like this: + + 4. Activate the Conda environment (if not activated yet). *You should activate a specific conda environment related to this project (e.g., the geospatial env created in section
[Find-GCP python utility: installation](#find-gcp-python-utility-installation)):* ``` source activate geospatial @@ -636,7 +678,10 @@ python $FIND_GCP_PATH/gcp_find.py -v -t ODM -i $INPUTS_PATH/GCP_reference.txt -- ``` python $FIND_GCP_PATH/gcp_find.py -v -t ODM -i $INPUTS_PATH/GCP_reference.txt --epsg 32611 -d 0 -o gcp_list.txt $INPUTS_PATH/*.JPG ``` -*This will search the ArUco markers from DICT_4x4_50 in your imagery and match them with corresponding IDs provided in your GCP_reference.txt file. Providing the exact EPSG code will ensure the returned coordinates of the GCPs detected in the imagery are in the correct coordinate system. The list of images with detected GCPs is saved to the gcp_list.txt file, which looks like this:* + + +This will search the ArUco markers from DICT_4x4_50 in your imagery and match them with corresponding IDs provided in your `GCP_reference.txt` file. Providing the exact EPSG code will ensure the returned coordinates of the GCPs detected in the imagery are in the correct coordinate system. The list of images with detected GCPs is saved to the `gcp_list.txt` file, which looks like this: + ``` EPSG:32611 523287.368 4779588.335 1397.823 5041 91 R0036021.JPG 0 @@ -683,13 +728,13 @@ A practical strategy is to first **employ an automated filter to narrow down to 1. Make sure you have your local copy of the **[geo_utils](https://github.com/ISUgenomics/geo_utils)** GitHub repository, placed in your SOFTWARE or TOOLS directory on Atlas (e.g., `project///SOFTWARE`). *(You can follow the instructions in section [geo_utils Python utility: installation](#geo_utils-python-utility-installation) to download this repository.)* We will use the Python script `gcp_images_picker.py` located in the TOOLS subdir of this repo:
```
- ls project///SOFTWARE
+ ls /project///SOFTWARE
```
2. Make sure you navigate back to the IMAGES directory in your photogrammetry project. You can softlink the `gcp_images_picker.py` script for easy use:
```
- cd project///ODM/IMAGES/
- ln -s project///SOFTWARE/geo_utils/TOOLS/gcp_images_picker.py ./
+ cd project///ODM/IMAGES/
+ ln -s /project///SOFTWARE/geo_utils/TOOLS/gcp_images_picker.py ./
```
3. Run the `gcp_images_picker.py` script to automate selection of representative GCP images, minimizing manual inspection:
@@ -699,7 +744,7 @@ A practical strategy is to first **employ an automated filter to narrow down to
```
*for example:*
```
- python3 gcp_images_picker.py -i gcp_list.txt -w 6000 -l 4000 -n 10
+ python gcp_images_picker.py -i gcp_list.txt -w 6000 -l 4000 -n 10
```
*The script will write the selected data to the file specified by the -o option. If the option isn't provided, it defaults to* `gcp_list_selected.txt`.
```
@@ -721,6 +766,86 @@ A practical strategy is to first **employ an automated filter to narrow down to
Automation aids efficiency, but a human touch ensures precision!
+ ## **Automatic generation of ArUco codes** @@ -476,14 +490,15 @@ In the naming convention like `DICT_4X4_100` or `DICT_6X6_250`:
To produce ArUco markers for your land surveying project, start by installing the Find-GCP Python utility ([refer to the section above](#find-gcp-python-utility-installation)). Within the cloned Find-GCP repository directory, you'll locate the `aruco_make.py`. This tool assists you in generating markers from standard dictionaries, as well as more compact 3x3 square markers. * While in the **Find-GCP** directory, use `pwd` command to print the path on the screen. *You will need this path to run python scripts from another location in the file system.* - + * Navigate to the selected location in the file system and create the **markers** directory: ``` +cd /project/
`python aruco_make.py -d
+Create a subdirectory for your new project in the IMAGES directory and create soft links for your imagery and (eventually) the `GCP_reference.txt` file: +``` +cd IMAGES +mkdir project-X +cd project-X +ln -s
+If your GCP reference file (here: gcp_epsg32611_2021_wbs1_coresite.csv) has format different than space-separated 4 columns: aruco_ID X Y Z, then you should adjust it accordingly to get something like this: ``` +0 523287.368 4779588.335 1397.823 +1 523305.976 4779572.588 1397.817 +2 523347.074 4779571.424 1397.653 +3 523364.648 4779587.932 1395.735 +4 523394.376 4779529.525 1398.728 +5 523363.938 4779530.027 1400.244 +``` +You can use `awk` *command to easily extract the columns you need.* Note that GCP_reference.txt file should not have a header. + + 3. Set paths as temporary variables or use them directly: ``` FIND_GCP_PATH=/path/to/Find-GCP_repo INPUTS_PATH=/path/to/input_imagery ``` +In my case the path variables look like this: + + 4. Activate the Conda environment (if not activated yet). *You should activate a specific conda environment related to this project (e.g., the geospatial env created in section
[Find-GCP python utility: installation](#find-gcp-python-utility-installation)):* ``` source activate geospatial @@ -636,7 +678,10 @@ python $FIND_GCP_PATH/gcp_find.py -v -t ODM -i $INPUTS_PATH/GCP_reference.txt -- ``` python $FIND_GCP_PATH/gcp_find.py -v -t ODM -i $INPUTS_PATH/GCP_reference.txt --epsg 32611 -d 0 -o gcp_list.txt $INPUTS_PATH/*.JPG ``` -*This will search the ArUco markers from DICT_4x4_50 in your imagery and match them with corresponding IDs provided in your GCP_reference.txt file. Providing the exact EPSG code will ensure the returned coordinates of the GCPs detected in the imagery are in the correct coordinate system. The list of images with detected GCPs is saved to the gcp_list.txt file, which looks like this:* + + +This will search the ArUco markers from DICT_4x4_50 in your imagery and match them with corresponding IDs provided in your `GCP_reference.txt` file. Providing the exact EPSG code will ensure the returned coordinates of the GCPs detected in the imagery are in the correct coordinate system. The list of images with detected GCPs is saved to the `gcp_list.txt` file, which looks like this: + ``` EPSG:32611 523287.368 4779588.335 1397.823 5041 91 R0036021.JPG 0 @@ -683,13 +728,13 @@ A practical strategy is to first **employ an automated filter to narrow down to 1. Make sure you have your local copy of the **[geo_utils](https://github.com/ISUgenomics/geo_utils)** GitHub repository, placed in your SOFTWARE or TOOLS directory on Atlas (e.g., `project/
+ +### ***Visual check of representative images for a marker*** + +
+WARNING:
+
+Proceeding with this section requires the use of Secure SHell (SSH) paired with X11 forwarding. X11 forwarding enables graphical applications run on the cluster to manifest visually on your local machine.
+When connecting via SSH, use the -X (or -Y, which is less secure but more permissive) option to enable X11 forwarding.
+(For me, the -Y variant worked. Note it may be slow!)
+ssh -Y user.name@atlas-login.hpc.msstate.edu
+Using Atlas Open OnDemand (OOD) ⤴ (in-browser access) can offer a more responsive experience compared to traditional SSH with X11 forwarding, especially over slower connections.
+If the cluster doesn't permit the use of X11 forwarding and OOD also didn't work for you, you'll need to take a detour:
+1. Download the relevant images (only those from gcp_list.txt) to your local machine.
+2. Clone the Find-GCP repository again but this time on your local setup.
+3. Then continue with the subsequent steps of this section.
+*Be sure to modify the paths to align with your local filesystem during this process. + +
+Proceeding with this section requires the use of Secure SHell (SSH) paired with X11 forwarding. X11 forwarding enables graphical applications run on the cluster to manifest visually on your local machine.
+When connecting via SSH, use the -X (or -Y, which is less secure but more permissive) option to enable X11 forwarding.
+(For me, the -Y variant worked. Note it may be slow!)
+ssh -Y user.name@atlas-login.hpc.msstate.edu
+Using Atlas Open OnDemand (OOD) ⤴ (in-browser access) can offer a more responsive experience compared to traditional SSH with X11 forwarding, especially over slower connections.
+If the cluster doesn't permit the use of X11 forwarding and OOD also didn't work for you, you'll need to take a detour:
+1. Download the relevant images (only those from gcp_list.txt) to your local machine.
+2. Clone the Find-GCP repository again but this time on your local setup.
+3. Then continue with the subsequent steps of this section.
+*Be sure to modify the paths to align with your local filesystem during this process. + +
+ +Ensuring that selected images are truly representative for each Ground Control Point (GCP) is a crucial step for accurate georeferencing. The `gcp_check.py` tool (from [Find-GCP repo](https://github.com/zsiki/Find-GCP)) offers a **user-friendly graphical interface to facilitate the visual check** of GCPs detected by `gcp_find.py`. + +If you've followed this tutorial, you should have already cloned the Find-GCP repository (see section [Find-GCP Python utility: installation](https://geospatial.101workbook.org/IntroPhotogrammetry/OpenDroneMap/03-ODM-georeferencing#find-gcp-python-utility-installation)). As a result, the `gcp_check.py` utility would be included within your cloned repo, ready for use. + +Check your SOFTWARE path on the Atlas cluster (for reference, see step 7 "Create the storage directory for custom software and GitHub repos" in section [Create env for geospatial analysis](#create-env-for-geospatial-analysis)): +``` +ls /project/
+WARNING:
+
+Please be cautious: the script gcp_check.py can NOT be executed within the directory containing the input images. Instead, navigate one level up in the directory structure before running it. + +
+Please be cautious: the script gcp_check.py can NOT be executed within the directory containing the input images. Instead, navigate one level up in the directory structure before running it. + +
+ +Let's assume you store your `gcp_list.txt` file along with your imagery at the `IMAGES/
+The tool is designed to automatically identify markers within images. Once detected, it highlights these markers with a circle and displays the corresponding ArUco ID. This setup aids in easy visual verification, ensuring that markers are correctly recognized and labeled. + --- ### **SCENARIO 2:** *GCP file with custom IDs* @@ -753,13 +878,28 @@ For cases where you have a GCP file with custom IDs, your inputs will be **the i 1. Login to the Atlas cluster using SSH protocol (command line) or OOD access (web-based). 2. Navigate to your project working directory. *Use the command below:* ``` -cd /project/
+Create a subdirectory for your new project in the IMAGES directory and create soft links for your imagery and (eventually) the `GCP_reference.txt` file: ``` +cd IMAGES +mkdir project-X +cd project-X +ln -s
-The output `representatives` should contain the representative image for each marker ID: +The output `representatives` should contain the representative image for each marker ID:
*(with marker's XY coordinates in the image provided in the first 2 columns)* ``` 3546 1937 R0036737.JPG 0 3631 2017 R0036909.JPG 1 @@ -820,7 +963,8 @@ mkdir representative awk '{print $3}' < representatives > list for i in `cat list`; do k=`echo $i | awk -F"." '{print $1}'`; n=`cat representatives | awk -v A=$i '{if ($3==A) print $4}'` ; cp $INPUTS_PATH/$i representative/$k"_"$n.JPG; done ``` -Now, the images should be copied into the representative subdirectory and their names should change from R0036737.JPG to R0036737_0.JPG denoting the ID of detected ArUco marker (which is required in the next step). +Now, the images should be copied into the representative subdirectory and their names should change from R0036737.JPG to R0036737_0.JPG denoting the ID of detected ArUco marker (which is required in the next step).
+ **STEP 2.** In the second step, a Python script `gcp_to_aruco_mapper.py` automatically matches the GCP coordinates with the representative images (by calculating the distance between GCPs from reference file and GPS coordinates of each picture). @@ -828,19 +972,20 @@ for i in `cat list`; do k=`echo $i | awk -F"." '{print $1}'`; n=`cat representat 1. Make sure you have your local copy of the **[geo_utils](https://github.com/ISUgenomics/geo_utils)** GitHub repository, placed in your SOFTWARE or TOOLS directory on Atlas (e.g., `project/