Futures Executor Unpickling Error for Large Workloads

[RamenMode]

FuturesExecutor fails with pickle.UnpicklingError: could not find Mark. when trying to unpickle a file. To replicate,

1. pull fork with branch futures_test located at my fork
2. run kevin_future_test_files/future_perf.py

python3 kevin_future_test_files/future_perf.py

This runs several batch sizes for the reduction task applied to sum(1...10**8), each 4 times.
Error will occur after roughly 5-7 iterations (likely around middle of 2nd batch size)

Note: Tested this with smaller sizes like sum(1...10**6), no issues. Looked at debug logs and transaction logs, could not seem to find anything indicate disk/memory overflow and debug logs look relatively normal besides the fact that the workers stop sending progress updates to the manager. also tried with -d all option manually using vine_factory, no debug logs revealed any issues

[dthain]

@RamenMode have you found any more details about this problem?

[RamenMode]

@dthain unfortunately I have not. @BarrySlyDelgado and I have attempted to test it further, but it actually did a full run (and worked) on a different machine, but then failed again. Since it is very intermittent, we've been unable to find a reason. I will continue trying, but at this moment, I want to make sure the high-level executor functions functionality is working (currently implementing for function calls in addition to PythonTasks). Will hopefully have updates soon.