No result files, or they are empty when running python scripts/run.py or ./testbed --no-gui #326

compressionist · 2022-03-10T17:24:31Z

compressionist
Mar 10, 2022

Hi all,

I'm trying at least one of your examples (I'm running python scripts/run.py or a ./testbed --no-gui) to get at least some results, and while I see your open source running without warnings, there are no output files results or only empty files are generated.

~/instant-ngp$ python scripts/run.py --mode sdf --scene armadillo --save_mesh sdf_armadillo --save_snapshot sdf_armadillo
Scene armadillo
19:49:09 SUCCESS Loaded mesh "/home/ubuntu/instant-ngp/data/sdf/armadillo.obj" file with 1 shapes.
19:49:09 SUCCESS Built TriangleBvh: nodes=21845
19:49:09 SUCCESS Built OptiX GAS and shaders
19:49:09 SUCCESS Built TriangleOctree: depth=10 nodes=209341 dual_nodes=839938. Populating dual nodes...
19:49:10 SUCCESS Loaded mesh: triangles=99976 AABB=[min=[-0.695084,-0.825671,-0.632707], max=[0.695084,0.825671,0.632707]] after scaling=[min=[0.0865273,0.00744793,0.124301], max=[0.913473,0.992552,0.875699]]
19:49:10 INFO   Loading network config from: /home/ubuntu/instant-ngp/configs/sdf/base.json
19:49:10 INFO   GridEncoding: Nmin=16 b=1.38191 F=2 T=2^19 L=16
19:49:10 INFO   Model:     3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=4)]-->1
19:49:10 INFO    total_encoding_params=12196240 total_network_params=7168
Training: 100%|███████████████████████████████████████████████████████████▉| 99984/100000 [10:24<00:00, 160.06step/s, loss=0.000872]
Saving snapshot sdf_armadillo
Generating mesh via marching cubes and saving to sdf_armadillo. Resolution=[256,256,256]
19:59:34 INFO   #vertices=0 #triangles=0
~$ find -name sdf_armadillo*
./instant-ngp/logs/sdf_armadillo_no-gui.log
./instant-ngp/sdf_armadillo
~$ ls -lah ./instant-ngp/sdf_armadillo
-rw-rw-r-- 1 ubuntu ubuntu 0 Mar 9 19:59 ./instant-ngp/sdf_armadillo

By the way, you changed the parameters for running python scripts/run.py. So, please answer which parameters should be used instead of those specified for the testbed. I've pulled out of the python code the parameters to run about half of your examples, but I will have to modify run.py to run the other models. So, for example, how should I run run.py for nerf_synthetic?

~/instant-ngp$ python scripts/run.py --scene data/nerf_synthetic/lego/transforms_train.json
Traceback (most recent call last):
 File "scripts/run.py", line 79, in <module>
  raise ValueError(args.scene,"Must specify either a valid '--mode' or '--scene' argument.")
ValueError: ('data/nerf_synthetic/lego/transforms_train.json', "Must specify either a valid '--mode' or '--scene' argument.")

~/instant-ngp$ python scripts/run.py --mode nerf_synthetic --scene lego/transforms_train.json
usage: run.py [-h] [--scene SCENE] [--mode [{nerf,sdf,image,volume}]] [--network NETWORK] [--load_snapshot LOAD_SNAPSHOT]
       [--save_snapshot SAVE_SNAPSHOT] [--nerf_compatibility] [--test_transforms TEST_TRANSFORMS]
       [--screenshot_transforms SCREENSHOT_TRANSFORMS] [--screenshot_frames [SCREENSHOT_FRAMES [SCREENSHOT_FRAMES ...]]]
       [--screenshot_dir SCREENSHOT_DIR] [--screenshot_spp SCREENSHOT_SPP] [--save_mesh SAVE_MESH]
       [--marching_cubes_res MARCHING_CUBES_RES] [--width WIDTH] [--height HEIGHT] [--gui] [--train] [--n_steps N_STEPS]
       [--sharpen SHARPEN]
run.py: error: argument --mode: invalid choice: 'nerf_synthetic' (choose from 'nerf', 'sdf', 'image', 'volume')

Please do at least the above. Your project is quite interesting, but there are many problems with the documentation. I have already spent a few working days (I have been working with ML and open source for several years), but so far, I have not seen a single non-empty file as a result of running your examples.

compressionist · 2022-03-14T16:30:09Z

compressionist
Mar 14, 2022
Author

If possible, give us some tips on what to do to save the model after training. It would also be nice to save intermediate models and results. For example, I ran "./testbed --no-gui --mode nerf --scene data/nerf/closet2 >logs/output.log 2>&1 &" over the weekend, and now I see over four million iterations in my log and learning is still going on.

09:23:52 INFO     iteration=4579520 loss=0.000298189
09:23:52 INFO     iteration=4579536 loss=0.000286814
09:23:52 INFO     iteration=4579552 loss=0.000278757
09:23:53 INFO     iteration=4579568 loss=0.000287419
09:23:53 INFO     iteration=4579584 loss=0.000284334
09:23:54 INFO     iteration=4579600 loss=0.000289864
09:23:54 INFO     iteration=4579616 loss=0.000263825
09:23:54 INFO     iteration=4579632 loss=0.000269695
09:23:55 INFO     iteration=4579648 loss=0.000295802
09:23:55 INFO     iteration=4579664 loss=0.000288519
09:23:55 INFO     iteration=4579680 loss=0.000299271
09:23:56 INFO     iteration=4579696 loss=0.000281064

Still, there are no output files other than my log generated from your terminal output. Can I record intermediate or final results of running scripts/run.py or ./testbed with a GUI?

0 replies

Tom94 · 2022-03-14T16:37:52Z

Tom94
Mar 14, 2022
Maintainer

Hi there, a couple of notes:

./testbed is intended as an absolute barebones program -- it can't do anything other than report training progress without a GUI. Instead, you're supposed to use the Python bindings (as in the example run.py) to programmatically instrument the codebase.
Your initial command line is actually saving out a snapshot of the model parameters. The trouble is that you passed the same filename for the mesh as for the snapshot, which means that your snapshot is immediately overwritten by the (strangely empty) mesh. Try using --save_snapshot sdf_armadillo.msgpack.
As for why the mesh is empty (see the "0 vertices" console output), I can only speculate. It works fine on my end. Perhaps try updating via git pull master && git submodule update --recursive and recompiling.
Lastly, training typically takes on the order of 10000-30000 steps. You don't need to run it for as long as you do. I recommend passing --n_steps 15000 for SDF training.

Cheers!

0 replies

compressionist · 2022-03-15T01:30:02Z

compressionist
Mar 15, 2022
Author

Hi Thomas,
Thanks for your advice!

I did an update from git and recompiled the project as you suggested, the snapshot was created, but unfortunately, the mesh remained zero sizes or not created at all when I ran nerf fox. there are also errors when training with nerf fox.

$ python scripts/run.py --n_steps 15000 --mode sdf --scene armadillo --save_mesh sdf_armadillo.msgpack --save_snapshot sdf_armadillo
Scene  armadillo
17:30:56 SUCCESS  Loaded mesh "/home/ubuntu/instant-ngp/data/sdf/armadillo.obj" file with 1 shapes.
17:30:56 SUCCESS  Built TriangleBvh: nodes=21845
17:30:56 SUCCESS  Built OptiX GAS and shaders
17:30:57 SUCCESS  Built TriangleOctree: depth=10 nodes=209341 dual_nodes=839938. Populating dual nodes...
17:30:57 SUCCESS  Loaded mesh: triangles=99976 AABB=[min=[-0.695084,-0.825671,-0.632707], max=[0.695084,0.825671,0.632707]] after scaling=[min=[0.0865273,0.00744793,0.124301], max=[0.913473,0.992552,0.875699]]
17:30:57 INFO     Loading network config from: /home/ubuntu/instant-ngp/configs/sdf/base.json
17:30:57 INFO     GridEncoding:  Nmin=16 b=1.38191 F=2 T=2^19 L=16
17:30:57 INFO     Model:         3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=4)]-->1
17:30:57 INFO       total_encoding_params=12196240 total_network_params=7168
Training: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████▉| 14992/15000 [01:32<00:00, 162.81step/s, loss=0.00115]
Saving snapshot  sdf_armadillo
Generating mesh via marching cubes and saving to sdf_armadillo.msgpack. Resolution=[256,256,256]
17:32:29 INFO     #vertices=0 #triangles=0

$ python scripts/run.py --n_steps 15000 --mode nerf --scene data/nerf/fox --save_mesh nerf_fox.msgpack --save_snapshot nerf_fox_snapshot
...
18:05:31 SUCCESS  Loaded 50 images of size 1080x1920 after 0s
18:05:31 INFO       cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00721,1.41295]]
18:05:31 INFO     Loading network config from: /home/ubuntu/instant-ngp/configs/nerf/base.json
18:05:31 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
18:05:31 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
18:05:31 INFO     Color model:   3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
18:05:31 INFO       total_encoding_params=13074912 total_network_params=10240
Training: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████▉| 14992/15000 [02:00<00:00, 124.18step/s, loss=0.000371]
Saving snapshot  nerf_fox_snapshot
Generating mesh via marching cubes and saving to nerf_fox.msgpack. Resolution=[256,256,256]
18:07:33 INFO     #vertices=2939619 #triangles=5730400
Traceback (most recent call last):
  File "scripts/run.py", line 300, in <module>
    testbed.compute_and_save_marching_cubes_mesh(args.save_mesh, [res, res, res])
RuntimeError: /home/ubuntu/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/object.h:198 check failed output.m() == output_width()

0 replies

leventt · 2022-03-15T01:35:03Z

leventt
Mar 15, 2022

try these instead:

... --save_mesh mesh.obj --save_snapshot nerf_fox.msgpack ...

supported mesh formats are obj and ply and msgpack is the format for the network snapshot

0 replies

leventt · 2022-03-15T01:42:55Z

leventt
Mar 15, 2022

also just checking the error, try this too:

... --marching_cubes_res 128 ...

if that works there may be a bug related to cutlass mlp implementation and run.py script not having the same default resolution values perhaps (but speculating after skimming the code here)

0 replies

compressionist · 2022-03-15T02:03:31Z

compressionist
Mar 15, 2022
Author

Thanks, but unfortunately the same problem remains.

~/instant-ngp$ python scripts/run.py --n_steps 15000 --mode nerf --scene data/nerf/fox --marching_cubes_res 128 --save_mesh nerf_fox.obj --save_snapshot nerf_fox.msgpack
Scene  data/nerf/fox
18:57:04 INFO     Loading NeRF dataset from
18:57:04 INFO       data/nerf/fox/transforms.json
18:57:04 SUCCESS  Loaded 50 images of size 1080x1920 after 0s
18:57:04 INFO       cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00721,1.41295]]
18:57:04 INFO     Loading network config from: /home/ubuntu/instant-ngp/configs/nerf/base.json
18:57:04 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
18:57:04 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
18:57:04 INFO     Color model:   3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
18:57:04 INFO       total_encoding_params=13074912 total_network_params=10240
Training: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 14992/15000 [02:00<00:00, 124.57step/s, loss=0.0004]
Saving snapshot  nerf_fox.msgpack
Generating mesh via marching cubes and saving to nerf_fox.obj. Resolution=[128,128,128]
18:59:05 INFO     #vertices=466773 #triangles=908026
Traceback (most recent call last):
  File "scripts/run.py", line 300, in <module>
    testbed.compute_and_save_marching_cubes_mesh(args.save_mesh, [res, res, res])
RuntimeError: /home/ubuntu/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/object.h:198 check failed output.m() == output_width()

1 reply

leventt Mar 15, 2022

~~SDF mode seem to generate 0 verts or indices because of this line it looks like:~~ looks like I was wrong about this and vertices and indices didn't even exist before this line was reached:

instant-ngp/src/testbed_nerf.cu

Line 3002 in ad31b9d

if (m_testbed_mode == ETestbedMode::Nerf) {

and the error you are getting is because of the output padding vs un-padded output dimension check here:

instant-ngp/src/testbed_nerf.cu

Line 3010 in ad31b9d

GPUMatrix<float> color_matrix(mlp_out.data(), padded_output_width, n_verts);

and here respectively:

https://github.com/NVlabs/tiny-cuda-nn/blob/bd29d77c589e7593680a3aa508e35a0136df904b/include/tiny-cuda-nn/object.h#L198

I will try to implement a fix by having that output dimension check padded... But I have a feeling, I may be mistaken as the GUI version of ray-marching would not have worked otherwise and we know it works. Perhaps I will try checking if the CLI version of ray-marching works differently than the GUI version somehow.

If you have a chance to try the GUI version of the ray-marching, can you perhaps let me know if it gives you the same error?

compressionist · 2022-03-15T02:39:04Z

compressionist
Mar 15, 2022
Author

I changed ".obj" to ".ply" it looks like in this case, the mesh is being created for sdf amadrillo since run.py generated 324 bytes mesh file. I don't know if this is ok as I haven't set up a GUI on my remote server yet.

$ python scripts/run.py --load_snapshot sdf_armadillo_snapshot.msgpack --marching_cubes_res 128 --n_steps 30000 --mode sdf --scene armadillo --save_mesh sdf_armadillo.ply --save_snapshot sdf_armadillo_snapshot.msgpack
19:22:25 SUCCESS  Built TriangleOctree: depth=10 nodes=209341 dual_nodes=839938. Populating dual nodes...
19:22:26 SUCCESS  Loaded mesh: triangles=99976 AABB=[min=[-0.695084,-0.825671,-0.632707], max=[0.695084,0.825671,0.632707]] after scaling=[min=[0.0865273,0.00744793,0.124301], max=[0.913473,0.992552,0.875699]]
Loading snapshot  sdf_armadillo_snapshot.msgpack
19:22:26 INFO     Loading network config from: sdf_armadillo_snapshot.msgpack
19:22:26 INFO     GridEncoding:  Nmin=16 b=1.38191 F=2 T=2^19 L=16
19:22:26 INFO     Model:         3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=4)]-->1
19:22:26 INFO       total_encoding_params=12196240 total_network_params=7168
Training:   0%|                                                                                                                                   | 0/30000 [00:00<?, ?step/s]
Saving snapshot  sdf_armadillo_snapshot.msgpack
Generating mesh via marching cubes and saving to sdf_armadillo.ply. Resolution=[128,128,128]
19:22:26 INFO     #vertices=0 #triangles=0


$ ls -lah sdf*
-rw-rw-r-- 1 ubuntu ubuntu   0 Mar 14 19:20 sdf_armadillo.obj
-rw-rw-r-- 1 ubuntu ubuntu 324 Mar 14 19:22 sdf_armadillo.ply
-rw-rw-r-- 1 ubuntu ubuntu 24M Mar 14 19:22 sdf_armadillo_snapshot.msgpack

2 replies

leventt Mar 15, 2022

Yeah this makes sense as it looks like, SDF mode is not supported for that flag just yet and the supported nerf mode, has that un-padded vs padded output dimensions check. So yeah, I guess SDF mode isn't marching any rays in this case but only saving the initial memory allocation perhaps without the rest of the calculation.

I am about to finish setting up a reproducible case for the nerf mode. I can confirm my suspicion after building the python bindings soon. I will report back :)

leventt Mar 15, 2022

hmm, I can't seem to reproduce the error unfortunatelly:

python script--n_steps 10 --mode nerf --scene data/nerf/fox --marching_cubes_res 128 --save_mesh nerf_fox.obj --s
ave_snapshot nerf_fox.msgpack
D:\ngp>python scripts\run.py  --n_steps 1500 --mode nerf --scene data/nerf/fox --marching_cubes_res 128 --save_mesh nerf_fox.obj --save_snapshot nerf_fox.msgpack
20:14:58 INFO     Loading NeRF dataset from
20:14:59 INFO       data\nerf\fox\transforms.json
20:14:59 SUCCESS  Loaded 50 images of size 1080x1920 after 0s
20:14:59 INFO       cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00721,1.41295]]
20:14:59 INFO     Loading network config from: D:\ngp\configs\nerf\base.json
20:14:59 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
20:14:59 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
20:14:59 INFO     Color model:   3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
20:14:59 INFO       total_encoding_params=13074912 total_network_params=9728
Training:  99%|███████████████████████████████████████████████████▌| 1488/1500 [01:27<00:00, 16.96step/s, loss=0.00411]
Saving snapshot  nerf_fox.msgpack
Generating mesh via marching cubes and saving to nerf_fox.obj. Resolution=[128,128,128]
20:16:31 INFO     #vertices=250642 #triangles=446174

but I can't test fully fused MLP like you do on your GPU, can you perhaps change this line under your local dependencies folder for tiny-cuda-nn copy?
https://github.com/NVlabs/tiny-cuda-nn/blob/bd29d77c589e7593680a3aa508e35a0136df904b/include/tiny-cuda-nn/object.h#L198

You would want to change this:

CHECK_THROW(output.m() == output_width());

to this:

CHECK_THROW(output.m() == padded_output_width());

if that works I can put in a PR...

leventt · 2022-03-15T06:53:11Z

leventt
Mar 15, 2022

I made a PR:
#349

to address the issue for the SDF mode... I am not able to recreate the NeRF mode issue but if I can find a way to recreate somehow, I have some plans to propose a fix for that too.

0 replies

Tom94 · 2022-03-15T07:24:45Z

Tom94
Mar 15, 2022
Maintainer

Thank you for helping investigate! Both bugs (NeRF as well as SDF mesh export) should be fixed on latest master.

1 reply

leventt Mar 15, 2022

my pleasure \o/

compressionist · 2022-03-15T18:06:35Z

compressionist
Mar 15, 2022
Author

Thanks you @leventt and @Tom94!

It appears that after changing output_width() to padded_output_width() in ".../tiny-cuda-nn/object.h" NERF training working without issues. At least, the mesh and the snapshot files are generated.

$ ls -lah nerf_fox*
-rw-rw-r-- 1 ubuntu ubuntu 65M Mar 15 09:46 nerf_fox.msgpack
-rw-rw-r-- 1 ubuntu ubuntu 71M Mar 15 09:46 nerf_fox.obj

So please let us know when the master repository is ready after your fix.

1 reply

leventt Mar 15, 2022

yes @Tom94 already merged fixes to master for both of your issues. try pulling the changes... that change I recommended isn't the most appropriate fix so prefer the master

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No result files, or they are empty when running python scripts/run.py or ./testbed --no-gui #326

{{title}}

Replies: 10 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

No result files, or they are empty when running python scripts/run.py or ./testbed --no-gui #326

Replies: 10 comments · 5 replies

compressionist Mar 14, 2022 Author

Tom94 Mar 14, 2022 Maintainer

compressionist Mar 15, 2022 Author

compressionist Mar 15, 2022 Author

compressionist Mar 15, 2022 Author

Tom94 Mar 15, 2022 Maintainer

compressionist Mar 15, 2022 Author

Replies: 10 comments 5 replies

compressionist
Mar 14, 2022
Author

Tom94
Mar 14, 2022
Maintainer

compressionist
Mar 15, 2022
Author

compressionist
Mar 15, 2022
Author

compressionist
Mar 15, 2022
Author

Tom94
Mar 15, 2022
Maintainer

compressionist
Mar 15, 2022
Author