[FEATURE] Shared mode/User Isolation option for job clusters

[aseemanand]

There is currently a permission issue for shared mode/user isolation for job clusters while using Brickflow. Per Databricks, local_disk0/.ephemeral_nfs is a directory used internally by the system, and users typically don't have direct access to it in shared cluster mode.

Important note:

Spark cannot directly interact with workspace files on compute configured with shared access mode, the purpose of this cluster is being secured, the action the import is trying to do is not allowed.

Reference: https://docs.databricks.com/en/files/workspace-interact.html#read-data-workspace-files

Cloud Information

• AWS
• Azure
• GCP
• Other

Describe the solution you'd like
Job cluster successfully working with Shared mode/User isolation mode in Brickflow.

Describe alternatives you've considered
We gave SELECT on all referenced tables and views in Single User mode to avoid Databricks limitation. This incurs tech debt as it means any new table/view in the upstream source can break our data pipeline .

Additional context
None

[stikkireddy]

Related to #31. Will take a look at this.

[aseemanand]

This is resolved. Thanks @stikkireddy @asingamaneni

#97 (comment)