DaskVine Serverless

[BarrySlyDelgado]

Proposed changes

This enables serverless with DaskVine. A new class, FunctionCallDask, creates function calls instead of PythonTasks. These options set with the task_mode option.

Post-change actions

Put an 'x' in the boxes that describe post-change actions that you have done.
The more 'x' ticked, the faster your changes are accepted by maintainers.

• make test Run local tests prior to pushing.
• make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
• make lint Run lint on source code prior to pushing.
• Manual Update Did you update the manual to reflect your changes, if appropriate? This action should be done after your changes are approved but not merged.
• Type Labels Select github labels for the type of this change: bug, enhancement, etc.
• Product Labels Select github labels for the product affected: TaskVine, Makeflow, etc.
• PR RTM Mark your PR as ready to merge.

Additional comments

I have tested it on a small scale locally and distributed. There are still some things to fix.

[dthain]

Please merge or rebase on master.

Also note change needed to #3589 .

Also talk to @tphung3 about the bug he just found in scheduling!

[dthain]

Taking a quick look, I see that an output file is declared in submit_finalize. Is there a corresponding remove_file in the destructor?

[BarrySlyDelgado]

No, there is no remove_file at the moment.

[dthain]

I'm asking you if it needs one. :)

[BarrySlyDelgado]

It can be added to the destructor however, the DaskVine executor will need only to delete the FunctionCall tasks and remove the files once all the tasks that may consume its output are completed.

[dthain]

@BarrySlyDelgado please rebase or merge on master to pick up the memory leak fix from @JinZhou5042 .

[dthain]

@BarrySlyDelgado is this the code being used to run HEP applications? If so, rebase on master and let's get this merged.

[dthain]

Hello?

[BarrySlyDelgado]

Yes, this is currently used for testing function calls for the HEP applications.

[dthain]

I'm ready to merge this once you have pulled in the necessary changes from master.