Clearing training data

[drperpen]

Hi,
I am training a 90 frames sequence shot. In order to free up the GPU memory after saving the snapshot for the current frame iteration, I am using testbed.clear_training_data(), but it doesn't seem to flush the GPU memory. Has anyone had success with this? Thanks!

    testbed.load_training_data(transforms_json)
RuntimeError: Could not allocate memory: S:\local_repos\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:112 cudaMalloc(&rawptr, n_bytes+DEBUG_GUARD_SIZE*2) failed with error out of memory

[drperpen]

I found a solution using:

pyngp.free_temporary_memory()

https://github.com/NVlabs/instant-ngp/blob/00dea16686a80100d264df9adeb26ae29137143b/src/python_api.cu#L224