[Feature Request] allow opt-out of workflow isolation

[neelance]

The bug #1432 just caused a lot of pain for us. It made our workers consume a lot of memory over time (and go OOM repeatedly) and made them go really slow over time, since every (implicit) call to new Promise was affected. It took me several weeks of investigating, giving up and then trying again on a different day to track down this root cause. This is because the issue only happened in production and most metrics that I could get from production did not clearly point to the issue. Even diffs on heap snapshots did not clearly point to AsyncLocalStorage because it contained a huge amount of other heap differences and AsyncLocalStorage did not catch my attention. I only got lucky that a CPU profile seemed odd and I noticed that the internals of Promise took much too long.

Our code was using new AsyncLocalStorage as a constant at the toplevel of a module. Normally such a constant only gets initialised once. It is quite unintuitive that with workflow isolation one has to consider that this code gets executed multiple times and thus reason through performance implications. #1432 plans to hide these implications, but it also mentions that maxCachedWorkflow might still be a performance issue.

This is not the first issue that we have due to the workflow isolation feature and I'm sorry, but I doubt that it will be the last. In theory workflow isolation seems like a good idea, but in practice it adds complexity that causes very uncommon issues. I want to point out that for example the Go SDK does not try to apply workflow isolation. Instead it expects the developer to write proper workflow code that respects deterministic execution requirements. I would rather teach my team to handle a transparent nondeterminism-error due to bad workflow code than to have issues like the one above. Therefore I would like to request the option to opt-out of workflow isolation (and instead opt-in to simplicity). I still appreciate the effort you've put into workflow isolation and its performance optimisations like reusable VMs.