Occasional dataloader-related build error

[goliaro]

From time to time, building FlexFlow (both locally and in CI) will result in the following error, which goes away by rerunning:

[ 92%] Building NVCC (Device) object python/CMakeFiles/flexflow_c.dir/flexflow_c_generated_flexflow_dataloader.cu.o
Error copying file (if different) from "/__w/FlexFlow/FlexFlow/build/python/CMakeFiles/flexflow_c.dir//flexflow_c_generated_flexflow_dataloader.cu.o.depend.tmp" to "/__w/FlexFlow/FlexFlow/build/python/CMakeFiles/flexflow_c.dir//flexflow_c_generated_flexflow_dataloader.cu.o.depend".
CMake Error at flexflow_c_generated_flexflow_dataloader.cu.o.Release.cmake:246 (message):
  Error generating
  /__w/FlexFlow/FlexFlow/build/python/CMakeFiles/flexflow_c.dir//./flexflow_c_generated_flexflow_dataloader.cu.o


make[2]: *** [python/CMakeFiles/flexflow_native_python.dir/build.make:77: python/CMakeFiles/flexflow_c.dir/flexflow_c_generated_flexflow_dataloader.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:1010: python/CMakeFiles/flexflow_native_python.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

[lockshaw]

Huh, interesting. I take it this isn't that common, as I've actually never seen it before. If it's not urgent I can take a look into it since I've been messing with the cmake code recently after I finish up with #562. I'll assign it to myself for now but feel free to reassign it if you want it addressed more promptly 🙂