Use Laspy #17

kalbermattenm · 2015-12-04T18:04:17Z

This PR is an attempt to replace liblas with laspy...

Currently we are using a SITN fork in a collective.recipe.cmd section for buildout (waiting for https://github.com/grantbrown/laspy/issues/35...)

@rbovard, could you test this branch on your side ?

Be aware that you should delete the [buildout] section of your local buildout file (the thing where we did something like part -= liblas)

kalbermattenm · 2015-12-04T18:04:36Z

This is (really) not ready to be merged...

kalbermattenm · 2015-12-04T18:05:59Z

TODO:

Update README
Check that everything is working
Use the pypi package (and get rid of Numpy in setup.py)
Remove the angle condition because it is part of an other PR (Rounding errors on acos(alpha) #16)
Remove liblas wheel (https://github.com/sitn/las_extractor/tree/master/wheels)

kalbermattenm · 2015-12-04T18:50:47Z

There might be a memory issue in laspy, so I will have to make some "intensive" tests on the dev server before bringing this into production. (see https://github.com/grantbrown/laspy/issues/36#issuecomment-162047090)

rbovard · 2015-12-07T08:26:22Z

buildout.cfg

 recipe = collective.recipe.cmd
 on_install = true
 on_update = true
-cmds = ${buildout:directory}\buildout\bin\pip.exe install ${buildout:directory}\wheels\${vars:liblas}
+cmds = ${buildout:directory}\buildout\bin\pip.exe install https://github.com/sitn/laspy/archive/e2a1a3307d4c252f80855c3e80ec373fdfc7265b.zip


It won't work here...

Could you be a bit more explicit ? (stack or whatever...)

oh... I got it... just get rid of the .exe...

Wait... I am going to update the branch...

Thanks but you also need to use / instead of \ ;)

After that I have the same right issue as the last time:

running install_lib creating /usr/local/lib/python2.7/dist-packages/laspytest error: could not create '/usr/local/lib/python2.7/dist-packages/laspytest': Permission denied ---------------------------------------- Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-ewBo_0-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-wP6wzv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-ewBo_0-build While: Installing laspy.

Is not better to use v1.3.0 from pypi?

Yes it is.... But it has been released on Friday (during the night in Europe...)

I really do not get why during the installation it tries to install/reach something in your Python root folder... We are using a virtual environment, so it should touch these folders....

kalbermattenm · 2015-12-07T08:38:28Z

Branch updated.

kalbermattenm · 2015-12-07T12:59:33Z

Could you try it by using the pypi package ?

To do so:

remove laspy from the part section of buildout (line 18)
in setup.py just add laspy=1.3.0 after numpy

I would also recommend to empty the content of the buildout folder and run bootstrap again:

python bootstrap-buildout.py (don't allow site pacakges..)

rbovard · 2015-12-07T14:02:28Z

Thanks, install is fine now..

But yet there is a Python error:

mod_wsgi (pid=5181): Exception occurred processing WSGI script 'lidar/buildout/parts/modwsgi/wsgi'.
Traceback (most recent call last):
  File "lidar/buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "lidar/buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "lidar/buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "lidar/buildout/eggs/pyramid_tm-0.11-py2.7.egg/pyramid_tm/__init__.py", line 94, in tm_tween
    reraise(*exc_info)
  File "lidar/buildout/eggs/pyramid_tm-0.11-py2.7.egg/pyramid_tm/__init__.py", line 75, in tm_tween
    response = handler(request)
  File "lidar/buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "lidar/buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/config/views.py", line 355, in rendered_view
    result = view(context, request)
  File "lidar/buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/config/views.py", line 501, in _requestonly_view
    response = view(request)
  File "lidar/las_extractor/views/lidar_profile.py", line 123, in lidar_profile
    jsonOutput, zMin, zMax, checkEmpty, perfLogStr = pointCloudExtractorV2(geom.coordinates, bufferSizeMeter, outputDir, dataDir, jsonOutput, csvOut, classesList, classesNames, perfLogStr)
  File "lidar/las_extractor/util/point_cloud_profiler.py", line 143, in pointCloudExtractorV2
    xvalues = np.copy(data[:, 0])
IndexError: too many indices

kalbermattenm · 2015-12-07T14:17:57Z

Hmmm... I do not have this issue... Have you tried multiple times or only once to extract some data ?

We definitively should add some logging in this script. I do not know if I will have some time today...

I should also update this PR....

rbovard · 2015-12-07T14:33:16Z

Multiple times, on multiple places...

rbovard · 2015-12-07T14:34:29Z

I should also update this PR....

I can do it, add what I did.

kalbermattenm · 2015-12-07T14:45:17Z

Wait a bit... I am refactoring the whole thing...

kalbermattenm · 2015-12-07T15:03:53Z

@rbovard, I added some logging in the code... If you update your code (be aware that there are quite a few modifications...), then you in the vars section of your own buildout file you can add debug_log = True. This should result in the following logging stuff in your Apache error file:

2015-12-07 15:51:54,403 WARNI [las_extractor.views.lidar_profile][Dummy-2] Request: d7c4ba6c-165f-49d4-b50b-6526d4572295
2015-12-07 15:51:54,407 WARNI [las_extractor.util.point_cloud_profiler][Dummy-2] PG REQUEST TIME: 0:00:00.003000
2015-12-07 15:51:54,894 WARNI [las_extractor.util.point_cloud_profiler][Dummy-2] ITERATE OVER TILE AND POINTS TIME: 0:00
:00.486000
2015-12-07 15:51:54,894 WARNI [las_extractor.util.point_cloud_profiler][Dummy-2] Found 645 points
2015-12-07 15:51:54,914 WARNI [las_extractor.views.lidar_profile][Dummy-2] TOTAL TIME: 0:00:00.510000

I would be interessted by the "Found xxx points" thing...

You should also be aware of this discussion: https://github.com/grantbrown/laspy/issues/36#issuecomment-162047090. There is a memory leak somewhere and I can see that memory utilization is increasing with each request I make...

Currently I would not (and cannot) use that on a production server...

rbovard · 2015-12-07T15:31:08Z

Ok, thanks.

Code updated... but no logs in my Apache logs even if I use log.error :(

kalbermattenm · 2015-12-08T08:51:34Z

This is strange... Is that working for c2cgeoportal ? (there are plenty log.warnings and log.error in it...)

I do not really get why this is not working... ?

rbovard · 2015-12-08T15:04:17Z

I don't know why but yes it's working for c2cgeoportal and not for las_extractor...

I managed to output something:

PG REQUEST TIME: 0:00:00.021431
ITERATE OVER TILE AND POINTS TIME: 0:00:02.658463
Found 0 points

So there is no points... I can see that you don't check it before copy the coordinates into new variables ;)

I'll investigate why there is no data...

kalbermattenm · 2015-12-08T15:17:56Z

ok... that's better... but it should not open any tile where there are no points touching the profile, thus you should have something in it...

Maybe you could also check which tile is opened inside the for tile in tileList: loop by printing / outputting the variable tile.

rbovard · 2015-12-08T15:30:35Z

I have my tiles...

For example: tile = /var/sig/geodata/altimetrie/lidar/2013/1261-41-g_28_8.las

kalbermattenm · 2015-12-08T15:33:37Z

OK... and can you retrieve any coordinates ?

Just add some output after that (print x and y) (by the way, what are you using to create some output..?)

x = x.item()
y = y.item()

Another test would be to check if this tile really intersects the profile you have drawn (in QGIS for instance...)

rbovard · 2015-12-08T15:38:47Z

I have some x and y...

by the way, what are you using to create some output..?

I throw exceptions with raise(util.LaspyException(...))

kalbermattenm · 2015-12-08T15:51:14Z

Could you print out also these two variables... normalPointToLineDistance and bufferSizeMeter (or raise an exception inside the if normalPointToLineDistance <= bufferSizeMeter: condition...)

What I am trying to do is find why there is nothing in the table variable...

Maybe you could increment it by yourself until you find why...

rbovard · 2015-12-09T15:58:02Z

There was an issue about my las files (projection mismatch), sorry about all the noise...

Now the script returns an other error after a long time, but I don't know if it's related to my configuration or not.

mod_wsgi (pid=13826): Exception occurred processing WSGI script 'buildout/parts/modwsgi/wsgi'.
Traceback (most recent call last):
  File "buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "buildout/eggs/pyramid_tm-0.11-py2.7.egg/pyramid_tm/__init__.py", line 94, in tm_tween
    reraise(*exc_info)
  File "buildout/eggs/pyramid_tm-0.11-py2.7.egg/pyramid_tm/__init__.py", line 75, in tm_tween
    response = handler(request)
  File "buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/config/views.py", line 355, in rendered_view
    result = view(context, request)
  File "buildout/eggs/pyramid-1.5.7-py2.7.egg/pyramid/config/views.py", line 501, in _requestonly_view
    response = view(request)
  File "las_extractor/views/lidar_profile.py", line 134, in lidar_profile
    jsonOutput, zMin, zMax, checkEmpty = pointCloudExtractorV2(geom.coordinates, bufferSizeMeter, outputDir, dataDir, jsonOutput, csvOut, classesList, classesNames, debug_log)
  File "las_extractor/util/point_cloud_profiler.py", line 186, in pointCloudExtractorV2
    generate_json(profile, jsonOutput, csvOut, classesList, classesNames)
  File "las_extractor/util/point_cloud_profiler.py", line 52, in generate_json
    classifName = classesNames[row[4]]
KeyError: 4.0

It's quiete long to debug, because I need to wait about 5 minutes to get this error...

rbovard · 2015-12-09T16:00:11Z

setup.py

@@ -27,13 +27,10 @@
        'numpy',
        'pyyaml',
        'pip',
+        'laspy',


kalbermattenm · 2015-12-10T06:58:57Z

How big are your LAS files ? Because if they are too big, then it is quite time consuming to loop over the points... Your LAS file are 50m x 50m, which makes quite a lot of them, but which significantly improves the performances...

kalbermattenm · 2015-12-10T07:02:48Z

The line on which you get an error is trying to get values from this file
https://github.com/kalbermattenm/las_extractor/blob/laspy/config.yaml.in

It looks like your Lidar classification values are not parsed as integers. Could you try to modify https://github.com/kalbermattenm/las_extractor/blob/laspy/las_extractor/util/point_cloud_profiler.py#L52 into classifName = classesNames[int(row[4])]

rbovard · 2015-12-10T07:50:11Z

How big are your LAS files ? Because if they are too big, then it is quite time consuming to loop over the points...

I have 3'600+ tiles of ~1.4 Mo. What about you? Maybe I should do something to have less heavier tiles?

kalbermattenm · 2015-12-10T08:36:54Z

At SITN, we have tiles of about 500 Ko in size (but the biggest weight 5Mo) and about 300'000 tiles...

Hmmm... I am quite stuck... Could you analyze what takes so much time ?

rbovard · 2015-12-10T11:34:51Z

IT'S FINALLY WORKING!

Thanks a lot for your help.

It's still very slow, for this one I waited 1 min... I think I should use less heavier tiles.

kalbermattenm · 2015-12-10T12:01:08Z

Be aware of the memory leak !!

grantbrown · 2015-12-21T15:17:16Z

Just a heads up, I got some work done on the laspy memory leak in this commit at the end of last week. Once we can confirm that the problem is resolved, I'll push a new version to PyPi.

kalbermattenm · 2015-12-22T07:01:30Z

Thanks @grantbrown a lot for the information (and for having investigated this issue) !

@rbovard, I am going to update the library today ! It would be good if we could check that the whole thing has improved (next year...)

kalbermattenm added 3 commits December 4, 2015 18:55

Create a recipe to get laspy from SITN fork

f6166e5

Switch to laspy

37a9866

Add Numpy as a local dependency to make laspy happy...

1956176

rbovard reviewed Dec 7, 2015
View reviewed changes

Get rid of .exe

ce16f00

kalbermattenm added 3 commits December 7, 2015 15:56

Use laspy and remove old conf

6947537

Add logging

a3987f0

Remove print

91dc631

rbovard reviewed Dec 9, 2015
View reviewed changes

setup.py

@@ -27,13 +27,10 @@

'numpy',

'pyyaml',

'pip',

'laspy',

Copy link

rbovard Dec 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra ,

Use Laspy #17

Are you sure you want to change the base?

Use Laspy #17

Conversation

kalbermattenm commented Dec 4, 2015

kalbermattenm commented Dec 4, 2015

kalbermattenm commented Dec 4, 2015

kalbermattenm commented Dec 4, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kalbermattenm commented Dec 7, 2015

kalbermattenm commented Dec 7, 2015

rbovard commented Dec 7, 2015

kalbermattenm commented Dec 7, 2015

rbovard commented Dec 7, 2015

rbovard commented Dec 7, 2015

kalbermattenm commented Dec 7, 2015

kalbermattenm commented Dec 7, 2015

rbovard commented Dec 7, 2015

kalbermattenm commented Dec 8, 2015

rbovard commented Dec 8, 2015

kalbermattenm commented Dec 8, 2015

rbovard commented Dec 8, 2015

kalbermattenm commented Dec 8, 2015

rbovard commented Dec 8, 2015

kalbermattenm commented Dec 8, 2015

rbovard commented Dec 9, 2015

Choose a reason for hiding this comment

kalbermattenm commented Dec 10, 2015

kalbermattenm commented Dec 10, 2015

rbovard commented Dec 10, 2015

kalbermattenm commented Dec 10, 2015

rbovard commented Dec 10, 2015

kalbermattenm commented Dec 10, 2015

grantbrown commented Dec 21, 2015

kalbermattenm commented Dec 22, 2015