[Feature] Add entry that allows for defining root path for DABs

[nc-sika]

When working with Databricks Asset Bundles (DABs), we've consistently encountered challenges in maintaining a proper sys.path that allows for seamless module imports across various development environments, including:

1. Local development (VSCode with Databricks Connect)
2. Databricks UI
3. Job Clusters

To address this, we've implemented a workaround in nearly all of our notebooks. The following code block is typically used at the beginning of each script script, that acts as a entry point, e.g. notebooks:

if 'dbutils' in globals():
# We're in a Databricks notebook
    project_root = Path('../../../').resolve()
else:
    from databricks.sdk import WorkspaceClient
    w = WorkspaceClient()
    dbutils = w.dbutils
    # We're not in a Databricks notebook (e.g., running with Databricks Connect)
    current_file_path = Path(file).resolve()
    project_root = current_file_path.parents[3]
if str(project_root) not in sys.path:
    sys.path.insert(0, str(project_root))

This problem is also discussed in the Databricks Community thread: How to set Python rootpath when deploying with DABs

To streamline development and improve consistency across environments, we propose that Databricks Asset Bundles (DABs) include a feature to specify the Python root path directly. This enhancement would eliminate the need for the current workarounds.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.