[feature] Enable Support for Srverless Compute

[maxim-mityutko]

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[codecov]

Codecov Report

Attention: Patch coverage is 72.97297% with 10 lines in your changes missing coverage. Please review.

Project coverage is 91.86%. Comparing base (35d0824) to head (2543874).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
brickflow/engine/workflow.py	65.00%	7 Missing ⚠️
brickflow/codegen/databricks_bundle.py	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #165      +/-   ##
==========================================
+ Coverage   90.33%   91.86%   +1.53%     
==========================================
  Files          22       22              
  Lines        3384     4316     +932     
==========================================
+ Hits         3057     3965     +908     
- Misses        327      351      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maxim-mityutko]

TODO: new bundle tests for serverless
TODO: deploy classic and serverless workflows
TODO: validate backwards compatibility