Skip to content

Commit

Permalink
Addressing review comments
Browse files Browse the repository at this point in the history
  • Loading branch information
@omshinde
omshinde committed Mar 27, 2024
1 parent 23c00d3 commit 422a500
Showing 1 changed file with 165 additions and 35 deletions.
200 changes: 165 additions & 35 deletions copc/lidar-las-to-copc.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
"\n",
"## About the Dataset\n",
"\n",
"We will be using the [G-LiHT Lidar Point Cloud V001](http://doi.org/10.5067/Community/GLIHT/GLLIDARPC.001) from the NASA EarthData. To access NASA EarthData into Jupyter Notebook, you can create an account by visiting [NASA's Earthdata Login page](https://urs.earthdata.nasa.gov/users/new). This will enable you to register for an account and retrieve the datasets used in the notebook.\n",
"We will be using the [G-LiHT Lidar Point Cloud V001](http://doi.org/10.5067/Community/GLIHT/GLLIDARPC.001) from the NASA EarthData. To access NASA EarthData in Jupyter you need to register for an [Earthdata account](https://urs.earthdata.nasa.gov/users/new).\n",
"\n",
"We will use [earthaccess](https://github.com/nsidc/earthaccess) library to set up credentials to fetch data from NASA's EarthData catalog."
]
Expand Down Expand Up @@ -140,12 +140,10 @@
],
"source": [
"# Search Granules\n",
"short_name = 'GLLIDARPC'\n",
"version = '001'\n",
"\n",
"las_item_results = earthaccess.search_data(\n",
" short_name=short_name,\n",
" version=version,\n",
" short_name=\"GLLIDARPC\",\n",
" version=\"001\",\n",
" temporal = (\"2020\"), \n",
" count=3\n",
")"
Expand Down Expand Up @@ -194,7 +192,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 39,
"metadata": {},
"outputs": [
{
Expand All @@ -208,7 +206,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"QUEUEING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 2236.96it/s]\n"
"QUEUEING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 1869.12it/s]\n"
]
},
{
Expand All @@ -222,8 +220,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"PROCESSING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 3968.12it/s]\n",
"COLLECTING RESULTS | : 100%|██████████| 1/1 [00:00<00:00, 12787.51it/s]"
"PROCESSING TASKS | : 100%|██████████| 1/1 [00:00<00:00, 16131.94it/s]\n",
"COLLECTING RESULTS | : 100%|██████████| 1/1 [00:00<00:00, 33554.43it/s]"
]
},
{
Expand All @@ -244,7 +242,7 @@
"source": [
"# Download Data - Selecting the 3rd file from the `las_item_results` list\n",
"gliht_las_file = earthaccess.download(las_item_results[2], data_dir)\n",
"las_filename = f\"{gliht_las_file[0]}\"\n",
"las_filename = gliht_las_file[0]\n",
"print(las_filename)"
]
},
Expand All @@ -254,11 +252,11 @@
"source": [
"## A Brief Introduction to PDAL \n",
"\n",
"[PDAL](https://pdal.io/) (Point Data Abstraction Library) is a C/C++ based open-source library for processing point cloud data. It allows chained operations for processing point clouds. Additionally, it also has a PDAL-Python wrapper to work in a Pythonic environment. \n",
"[PDAL](https://pdal.io/) (Point Data Abstraction Library) is a C/C++ based open-source library for processing point cloud data. Additionally, it also has a PDAL-Python wrapper to work in a Pythonic environment. \n",
"\n",
"#### Accessing and Getting Metadata Information\n",
"\n",
"PDAL CLI provides multiple [applications](https://pdal.io/en/2.7.0/apps/index.html) for processing point clouds. Similar to `gdal info` for TIFFs, we can run `pdal info <filename>` on the command line for getting metadata from a point cloud file without reading it in memory. "
"PDAL CLI provides multiple [applications](https://pdal.io/en/2.7.0/apps/index.html) for processing point clouds. Also, it allows chaining of these applications for processing point clouds. Similar to `gdal info` for TIFFs, we can run `pdal info <filename>` on the command line for getting metadata from a point cloud file without reading it in memory. "
]
},
{
Expand Down Expand Up @@ -497,7 +495,7 @@
"\n",
"#### PDAL Pipelines\n",
"\n",
"For converting the LiDAR LAS file to COPC format, we will define a [pdal pipeline](https://pdal.io/en/2.6.0/pipeline.html). A pipeline defines data processing within pdal for reading (using [pdal readers](https://pdal.io/en/2.6.0/stages/readers.html)), processing (using [pdal filters](https://pdal.io/en/2.6.0/stages/filters.html)) and writing operations (using [pdal writers](https://pdal.io/en/2.6.0/stages/writers.html)). The pipelines can also represent sequential operations and can be executed as [_stages_](https://pdal.io/en/2.6.0/pipeline.html#stage-object).\n",
"For converting the LiDAR LAS file to COPC format, we will define a [pdal pipeline](https://pdal.io/en/latest/pipeline.html). A pipeline defines data processing within pdal for reading (using [pdal readers](https://pdal.io/en/latest/stages/readers.html)), processing (using [pdal filters](https://pdal.io/en/latest/stages/filters.html)) and writing operations (using [pdal writers](https://pdal.io/en/latest/stages/writers.html)). The pipelines can also represent sequential operations and can be executed as [_stages_](https://pdal.io/en/latest/pipeline.html#stage-object).\n",
"\n",
"A pdal pipeline is defined in a JSON format either as a JSON object or a JSON array. Below is an example of a pdal pipeline taking a `.las` file as input, generating `stats` and writing it to a COPC format. \n",
"\n",
Expand Down Expand Up @@ -544,12 +542,12 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"# Defining output filename. Usually, COPC files are saved as .copc.laz\n",
"copc_filename = las_filename.split('.')[0]+'.copc.laz'"
"copc_filename = las_filename.replace('.laz', '.copc.laz')"
]
},
{
Expand Down Expand Up @@ -588,56 +586,188 @@
},
{
"cell_type": "code",
"execution_count": 73,
"execution_count": 48,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 253960\n",
"-rw-r--r-- 1 26M Feb 29 14:05 GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\n",
"total 239888\n",
"-rw-r--r-- 1 26M Mar 20 11:55 GLLIDARPC_FL_20200311_FIA8_l0s22.copc.laz\n",
"-rw-r--r-- 1 91M Feb 29 11:27 GLLIDARPC_FL_20200311_FIA8_l0s22.las\n"
]
}
],
"source": [
"# using -go for removing user details and h for getting memory size in MBs\n",
"!ls -goh {os.path.dirname(copc_filename)}"
"!ls -goh ./data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's read the created COPC file again and explore it's stats."
"Let's read the created COPC file again and explore the point cloud statistics."
]
},
{
"cell_type": "code",
"execution_count": 29,
"execution_count": 51,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"metadata\n",
"dict_keys(['filters.stats', 'readers.las'])\n"
]
"data": {
"text/plain": [
"[{'average': 506237.8598,\n",
" 'count': 3409439,\n",
" 'maximum': 506487.7363,\n",
" 'minimum': 505941.2263,\n",
" 'name': 'X',\n",
" 'position': 0,\n",
" 'stddev': 101.3857552,\n",
" 'variance': 10279.07135},\n",
" {'average': 2795977.637,\n",
" 'count': 3409439,\n",
" 'maximum': 2796533.993,\n",
" 'minimum': 2795533.003,\n",
" 'name': 'Y',\n",
" 'position': 1,\n",
" 'stddev': 274.313888,\n",
" 'variance': 75248.10912},\n",
" {'average': 2.192797205,\n",
" 'count': 3409439,\n",
" 'maximum': 69.99,\n",
" 'minimum': -12.54,\n",
" 'name': 'Z',\n",
" 'position': 2,\n",
" 'stddev': 1.788122887,\n",
" 'variance': 3.197383461},\n",
" {'average': 30205.96606,\n",
" 'count': 3409439,\n",
" 'maximum': 65535,\n",
" 'minimum': 14789,\n",
" 'name': 'Intensity',\n",
" 'position': 3,\n",
" 'stddev': 5497.346879,\n",
" 'variance': 30220822.71},\n",
" {'average': 1.200336478,\n",
" 'count': 3409439,\n",
" 'maximum': 5,\n",
" 'minimum': 1,\n",
" 'name': 'ReturnNumber',\n",
" 'position': 4,\n",
" 'stddev': 0.4267302243,\n",
" 'variance': 0.1820986843},\n",
" {'average': 1.400683808,\n",
" 'count': 3409439,\n",
" 'maximum': 5,\n",
" 'minimum': 1,\n",
" 'name': 'NumberOfReturns',\n",
" 'position': 5,\n",
" 'stddev': 0.5529822902,\n",
" 'variance': 0.3057894133},\n",
" {'average': 0.5248757933,\n",
" 'count': 3409439,\n",
" 'maximum': 1,\n",
" 'minimum': 0,\n",
" 'name': 'ScanDirectionFlag',\n",
" 'position': 6,\n",
" 'stddev': 0.4993808847,\n",
" 'variance': 0.249381268},\n",
" {'average': 0.002420339534,\n",
" 'count': 3409439,\n",
" 'maximum': 1,\n",
" 'minimum': 0,\n",
" 'name': 'EdgeOfFlightLine',\n",
" 'position': 7,\n",
" 'stddev': 0.04913738087,\n",
" 'variance': 0.002414482199},\n",
" {'average': 1.239596602,\n",
" 'count': 3409439,\n",
" 'maximum': 2,\n",
" 'minimum': 1,\n",
" 'name': 'Classification',\n",
" 'position': 8,\n",
" 'stddev': 0.4268373506,\n",
" 'variance': 0.1821901239},\n",
" {'average': 2.569738893,\n",
" 'count': 3409439,\n",
" 'maximum': 33,\n",
" 'minimum': -31,\n",
" 'name': 'ScanAngleRank',\n",
" 'position': 9,\n",
" 'stddev': 16.33805559,\n",
" 'variance': 266.9320603},\n",
" {'average': 1.475124207,\n",
" 'count': 3409439,\n",
" 'maximum': 2,\n",
" 'minimum': 1,\n",
" 'name': 'UserData',\n",
" 'position': 10,\n",
" 'stddev': 0.4993808847,\n",
" 'variance': 0.249381268},\n",
" {'average': 192.5227238,\n",
" 'count': 3409439,\n",
" 'maximum': 65535,\n",
" 'minimum': 0,\n",
" 'name': 'PointSourceId',\n",
" 'position': 11,\n",
" 'stddev': 680.5012031,\n",
" 'variance': 463081.8874},\n",
" {'average': 310476.1839,\n",
" 'count': 3409439,\n",
" 'maximum': 310485.0477,\n",
" 'minimum': 310469.0825,\n",
" 'name': 'GpsTime',\n",
" 'position': 12,\n",
" 'stddev': 4.240892261,\n",
" 'variance': 17.98516717},\n",
" {'average': 0,\n",
" 'count': 3409439,\n",
" 'maximum': 0,\n",
" 'minimum': 0,\n",
" 'name': 'Synthetic',\n",
" 'position': 13,\n",
" 'stddev': 0,\n",
" 'variance': 0},\n",
" {'average': 0,\n",
" 'count': 3409439,\n",
" 'maximum': 0,\n",
" 'minimum': 0,\n",
" 'name': 'KeyPoint',\n",
" 'position': 14,\n",
" 'stddev': 0,\n",
" 'variance': 0},\n",
" {'average': 0,\n",
" 'count': 3409439,\n",
" 'maximum': 0,\n",
" 'minimum': 0,\n",
" 'name': 'Withheld',\n",
" 'position': 15,\n",
" 'stddev': 0,\n",
" 'variance': 0},\n",
" {'average': 0,\n",
" 'count': 3409439,\n",
" 'maximum': 0,\n",
" 'minimum': 0,\n",
" 'name': 'Overlap',\n",
" 'position': 16,\n",
" 'stddev': 0,\n",
" 'variance': 0}]"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"valid_pipe = pdal.Reader.las(filename=las_filename) | pdal.Filter.stats()\n",
"valid_pipe.execute()\n",
"\n",
"# Get top key from the nested metadata dictionary\n",
"metadata = valid_pipe.metadata\n",
"top_key = list(metadata.keys())[0]\n",
"print(top_key)\n",
"\n",
"# Get top 2 keys from the next level metadata\n",
"print(metadata[top_key].keys())"
"# Getting statistic from the metadata\n",
"valid_pipe.metadata[\"metadata\"][\"filters.stats\"][\"statistic\"]"
]
},
{
Expand All @@ -646,7 +776,7 @@
"source": [
"## Accessing Data\n",
"\n",
"The `array values` can be accessed from the executed pipeline using `valid_pipe.arrays`. "
"The data values can be accessed from the executed pipeline using `valid_pipe.arrays`. "
]
},
{
Expand Down

0 comments on commit 422a500

Please sign in to comment.